Brandon Ong

To understand what it takes to build a humanoid robot with model-based control, we finetuned @physical_int 's (PI) Pi05 model for our custom use case and environment. We incurred ~$10K in hardware costs, compared to the typical ~$20K set up (DROID/ALOHA). Here are the lessons and challenges we faced building the first working prototype (shown in the video) in 3 months. Part 1: Hardware, Software, Model Selection, Custom Embodiment, Inference, Embedded Hardware, Hierarchical Planner Part 2: Model Evaluation, Data Collection, Model Training, Simulation and Teleoperation We hope sharing our experience accelerates the learning of others who are in a similar starting point.

We're in Boston this week for #RoboticsSummit, and NYC next week. Who should I meet to exchange notes and explore overlaps on vertical robotics? We're building robots for datacenters and are actively thinking about early commercialization angles, continuous post-training pipelines fed by fleet data and scalable verification for RL.

👀

The next frontier will be autonomous datacenters ran by AI and robots.

We'll be in SF next week for Data Center Expo. Do I know anyone deploying robots in industrial or similar settings? We're building robots for datacenters and are working together with our first enterprise partner. Would love to share notes and explore overlaps. Real-world RL, engineering deployment pipelines for continuously improving models, and early commercialization are top of mind.

mandatory day zero shot

@YorkYang5050 super cool! just DM-ed you!

Zero-shot. Plug-and-play. Dexterous manipulation. Running continuously in production. We keep pushing robotics out of the lab and into the real world — bringing the technology to everyone. 🌎

Digital twins for datacenters are still hard to build in 2026 As agents control systems more autonomously, the need for high-level observability and visualization will only increase We need to know quickly how agents are running to make sure they do not go off the rails

Who’s working on continuous improvement model posttraining tools in the open source? (For AI robotics models) Sounds like the kind of thing that would benefit from co-developing with the community. Would love to meet others building their own.

Congrats on the new MolmoAct 2 release by @allen_ai! A few features that stood out for those considering this for real-world deployments: 1. YAM embodiment unlock + 720h teleoperated dataset 720 hours of bimanual YAM data is a meaningful contribution. The YAM embodiment is a simple bimanual arm setup for dexterous tasks, very similar to the dual PiperX and Trossen WidowX arms. Anyone building on @physical_int 's Pi05 or similar models with a YAM-type robot now has significantly more data to fine-tune from, which should reduce the fine-tuning samples needed for a custom task, assuming the target task and environment fall within the dataset's distribution. The dataset spans household, factory, and coffee-shop settings with high object and scene variation. @cortexairobot was the data vendor. Hoping the appendix detailing the quality control protocol gets released. 2. Depth reasoning as a reproducible recipe, but only with layer-level access MolmoAct2-Think shows one way to inject depth information into the action model. Before producing an action, the model predicts a compact discrete depth representation that conditions the action expert through per-layer KV conditioning. The mechanism requires surgical access to the VLM's intermediate attention states at every layer, something only possible with fully open architectures. 3. Swappable VLM backbones for converting VLM -> VLM-ER The released training recipe effectively decouples the perception backbone from the action head. You can pick a VLM optimized for your task domain rather than accepting a generic vision encoder. Hypothetical example: for warehouse sorting where success hinges on reading tiny, cluttered, blurry SKU labels, start from a VLM fine-tuned for OCR (e.g., a custom Qwen-VL or InternVL variant) instead of a generalist web-scale VLM. Apply the MolmoAct2-ER training recipe to that backbone to produce an "OCR-VL-ER" variant, then attach a flow-matching Action Expert. The result is a bespoke VLA that inherits your perception fine-tuning, optimized for label-reading manipulation rather than generic open-world scenes. This assumes catastrophic forgetting is minimized and the backbone retains most of the baseline capabilities it had before fine-tuning. With this recipe, you can swap in domain-specific backbones (medical imaging, industrial inspection, high-res OCR) and convert them into action models entirely from open components.

Simulations are core to robotics research, but spinning up custom scenes is still tedious and has a steep learning curve We built mujoco workbench (mwb): cli + agent skills for codex/claude code to scaffold and debug sim scenes from natural language

"A robotics-shaped take-off curve." Cracking distribution is as hard as the technical challenges to commercialize early, especially when customers need to see a plausible trajectory, hardware budgets are tight, and data collection needs funding. "Channel construction is the most underestimated lever in the stack." Creating the data flywheel that powers continuous improvement and eventual task reliability is what closes the gap to real-world deployments that actually create value. "Deployment-system engineering matters as much as model architecture." Similar to the systems around early LLM applications and AI voice agents, a model's capabilities are only as good as the scaffolds around it. This is where engineering depth and iteration speed compound into capability. The teams that earn the right to a first deployment reliable enough to kickstart the data flywheel will crack adoption in a new vertical.

@LukasForTech The skills and CLI focus on scene creation and not the physics engine for now. So if you mean creating scenes from the standard MuJoCo library for your UAV testing then yes, it is supported.

@bytedunks This would streamline my UAV prototyping. Does it support aerodynamic force modeling, or is it limited to rigid-body kinematics?

Sims are an essential piece of any researcher's experimental loop, both for training data and for evaluation. However, spinning up quick scenes still requires a learning curve and reading extensive documentation. This is true even for the small but critical fraction of training data needed for diversity, or for quick, directionally correct evaluation checks. In response, we built MuJoCo Workbench (MWB), a CLI and set of agent skills to prototype custom scenes with coding agents like Codex and Claude Code. Repo is in the next post. It's an attempt to make building diverse scenes a delightful experience, and to maximize what coding agents can do in researchers' hands to accelerate their experimental loops. How it works: 1. Install the bundled agent skills. 2. Describe what you want, and the agent scaffolds a working sim for you. No MuJoCo experience is required to get started. The skills teach the agent the mwb CLI, the scene layout conventions, and the debug tools, so it can iterate on behavior without you needing to know the plumbing. We're working on extending MWB with a built-in integration for real-time inference on open-source VLAs/VAMs like Pi05. If you recognize the problems mentioned here and want to learn more, reach out.

@brianchew @kstonekuan lead maintainer is the legendary claude whisperer @kstonekuan

@bytedunks @kstonekuan That demo is SICK

congrats! stoked to have you building embodied autonomy here in SG

GitHub repo: github.com/Hebbian-Roboti…

@FanaHOVA Curious to try the apps built with this setup!

@suraj_sharma14 Applied with @kstonekuan !

Robots that learn. Live. In 72 hours. Embodied AI Hackathon @ SPC: → May 15-17 • San Francisco • In-person → Tracks: Manipulation • Locomotion • Aerial • Best Overall → Hardware + compute provided • OpenAI • AWS • Hugging Face Applications close: May 12. If you've ever wanted to ship AI that moves in the real world. This is your weekend. Register 👇 embodied-ai-hackathon-spc.replit.app @southpkcommons