Kaiyuan Eric Chen ✈️RSS🇦🇺

Graph-as-Policy (GaP) is a new variant of Agentic Robotics from @NVIDIA and @UCBerkeley that builds computation graphs (like ROS) to ensure modularity, manage complexity, and facilitate interpretability. 🧵Open code and paper: graph-robots.github.io/gap/ #Robotics #Automation #AgenticAI #EmbodiedAI

I will be presenting RoboVista at #RSS2026 today! This joint work with @GoogleDeepMind assess how effectively current VLMs process diverse robotic applications (esp. field robotics!) . Fun fact: most of all annotators already have PhD degrees. Come talk to me at poster session!

Thanks to the great team ! including: @UksangYoo @ruwantw from @BoschGlobal @guanzhi_wang @Adithya_Murali_ @balakumar_ @yukez @DrJimFan #spencer_huang from @nvidia @Shuangyu0_0Xie @letian_fu @uynitsuj @talking_kim @sandeep_b24 @HaoruXue from and @Ken_Goldberg from @UCBerkeley

The paper, more demo videos, 8 benchmarks, the skill library, and the code are all open. 🔓 🌐 Site: graph-robots.github.io/gap/ 📄 Paper: arxiv.org/abs/2607.05369

Graph-as-Policy (GaP) is a new variant of Agentic Robotics from @NVIDIA and @UCBerkeley that builds computation graphs (like ROS) to ensure modularity, manage complexity, and facilitate interpretability. 🧵Open code and paper: graph-robots.github.io/gap/ #Robotics #Automation #AgenticAI #EmbodiedAI

The Results: 📦 Under object pose variation: In benchmarks where model-free VLA policies dropped to 20% success rate, GaP achieved 93%-99% success rates. 🧼 Automated fine- tuning: For a bimanual crate-washing task, GaP matched performance of hand-engineered code.

How GaP works: 🗣️ Task Description: Describe a task (e.g., "make popcorn - repeat"), workcell, object and pose range, and provide a set of model-based and model-free skills. 🧠 Multi-Agent Orchestration: A multi-agent harness, built over coding tools like #Claude and #Gemini, decomposes a special prompt and assembles an interpretable computation graph from an open skill library 🔄 Self-Learning in Sim: The graph is rehearsed in simulation, using contact feedback to diagnose its own failures, and rewrites its own structure until performance plateaus. 🦾 Sim-to-Real: The optimized graph is then exported to run on the physical robot. GaP can provide the interpretability and reliability of classical engineering, but with coding agents doing the authoring and fine tuning. 🤯

📍 𝗖𝗮𝗻 𝗔𝗜 𝗡𝗮𝘃𝗶𝗴𝗮𝘁𝗲 𝗠𝗮𝗽𝘀 𝗟𝗶𝗸𝗲 𝗛𝘂𝗺𝗮𝗻𝘀 𝗗𝗼? 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗶𝗻𝗴 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵! 🗺️🤖 𝘙𝘦𝘢𝘥𝘪𝘯𝘨 𝘮𝘢𝘱𝘴, like Google Maps and Theme Park Maps, is second nature for humans. It is a highly challenging task that requires visual understanding, spatial reasoning, and long-horizon planning. We're curious - 𝗖𝗮𝗻 𝗟𝗮𝗿𝗴𝗲 𝗩𝗶𝘀𝗶𝗼𝗻-𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗩𝗟𝗠𝘀) 𝗱𝗼 𝗶𝘁 𝘁𝗼𝗼? 🤔 We’re excited to share 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵, the first-ever dataset and benchmark specifically designed for evaluating how well LVLMs perform on pixel-based map navigation tasks! 🚀 🔑 𝗪𝗵𝘆 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵 𝗶𝘀 𝗮 𝗚𝗮𝗺𝗲-𝗖𝗵𝗮𝗻𝗴𝗲𝗿: • 📌 1600+ Complex Pathfinding Queries from 100 uniquely challenging map scenarios (urban areas, theme parks, universities, malls, and more). • 📌 Introduces Map Space Scene Graph (MSSG): a novel data structure for mapping visual landmarks and spatial relationships to structured navigation tasks. • 📌 Evaluates state-of-the-art LVLMs like GPT-4o, Llama-3.2, and Qwen-2-VL under zero-shot and Chain-of-Thought (CoT) reasoning methods, revealing key insights into their spatial reasoning and navigation abilities. 🚩 𝗞𝗲𝘆 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: • Despite their impressive capabilities, current LVLMs struggle significantly with spatial reasoning and structured decision-making. • CoT prompting boosts spatial reasoning performance but sometimes introduces redundant details. 👀 𝗖𝗵𝗲𝗰𝗸 𝗼𝘂𝘁 𝗼𝘂𝗿 𝗳𝗶𝗻𝗱𝗶𝗻𝗴𝘀, 𝗱𝗮𝘁𝗮𝘀𝗲𝘁, 𝗮𝗻𝗱 𝗰𝗼𝗱𝗲 𝗵𝗲𝗿𝗲: 🔗 Arxiv: lnkd.in/gBv-sFJ3 Huge thanks to our incredible collaborators for making this happen, from @TAMU, @UCBerkeley, @mbzuai, @UMich, and @UCRiverside! 🎉 Let’s continue to bridge the gap between human intuition and AI navigation! 🗺️💡

Releasing a guide for scaling LLM embedding generation by 9x compared to autoscaling cloud services: ⚡️9x more GPUs ➡️ 9x speedup 💰61% cost reduction with spot The secret? Going multi-region with @skypilot_org to overcome single-region capacity limits. blog.skypilot.co/large-scale-em…

New model: Spin up Gemma 3 in your cluster/cloud $ sky launch gemma3.yaml docs.skypilot.co/en/latest/exam…

Try our live demo for the RAG on legal documents: rag.skypilot.co:8000

💡We built an open-source RAG with DeepSeek-R1! Here's what we learned: 📄 Don’t use DeepSeek R1 for retrieval; Use specialized embeddings — Qwen’s embedding model is amazing! @Alibaba_Qwen 🤖 Do use R1 for response generation @deepseek_ai 🔧 Use AI infra, vLLM @vllm_project & SkyPilot, to boost performance by 5x & scale up by 100x! Check out our complete code and learnings: blog.skypilot.co/deepseek-rag/

🚀We built a million-scale semantic image search with VectorDB and OpenAI’s CLIP. 🖼️ This is achieved by scaling out image embedding generation using SkyPilot — cutting time from ⏲️120 hours to 1 hour and costs from 💸$231 to $46! See how: blog.skypilot.co/large-scale-ve… Great thanks to the fantastic tools from: @trychroma @huggingface @OpenAI

SkyPilot now supports Janus-Pro by DeepSeek! • Text to image generation ✍️🖼️ • Image Q&A 📸🤖 On any of your infra! github.com/skypilot-org/s…

@zoodotdev @apgoldberg1

Transform your hardware design With Zoo Dev's Text-to-CAD, turn simple text prompts into detailed designs instantly.