David McAllister

Excited to share Flow Matching Policy Gradients: expressive RL policies trained from rewards using flow matching. It’s an easy, drop-in replacement for Gaussian PPO on control tasks.

Children learn from play. Can robots do the same? We propose 𝐏𝐥𝐚𝐲𝐟𝐮𝐥 𝐀𝐠𝐞𝐧𝐭𝐢𝐜 𝐑𝐨𝐛𝐨𝐭 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠, a paradigm that gives embodied coding agents a play stage before downstream tasks arrive, and instantiate it with 𝐑𝐀𝐓𝐬 (Robotics Agent Teams), where robots discover reusable skills through curious play. Co-led with @jiaxin_ge_

@hoanhle_ Definitely, let me know when you’re around!

@davidrmcall congrats! would love to see you guys' lab when i visit bay area. 🫣

A huge team effort! Fully open data, infrastructure and eval for the community to build on

@davidrmcall Congrats @davidrmcall I’m a huge fan of yours

(autonomous 20x) 😉

Announcing @xdofai: We’ve raised $70 million to build the core robotic infrastructure ecosystem for robot foundation models. My cofounders Fred (@YideShentu), Nemo (@itsnemojin) and I have been pursuing the dream of general purpose robots for our entire lives. After work at Covariant, Meta and Tesla, it became clear to us that general purpose robots are coming, and we are building XDOF to help make them a reality. For the last two years, we’ve been working behind the scenes to support major labs and companies deploying robots. In us, they have a partner with full-stack expertise, from hardware to operations to policy training. As our first public contribution to the space, we are open-sourcing ABC-130K, the largest open source teleoperation dataset, in collaboration with our partners from UC Berkeley, Carnegie Mellon, MIT and Amazon FAR. Thank you to our customers, partners, collaborators and investors for your trust and conviction in us. Together, we can accelerate the future of robotics!

@JasonJZLiu Congrats!

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors. We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies. FACTR 2 consists of: 1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors. 2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training. w/ @StevenOh_ @_tonytao_ 🧵(1/N)

@redstone_hong @arthurallshire

@_sam_sinha_ @BerntBornich @1x_tech Congrats!

I am SO excited to be sharing that I am joining @BerntBornich and @1x_tech to lead the new 1X World Model Lab aimed at building the next frontier of embodied AI! The core guiding principle of the lab is: scale up along every damn axis!! 🚀 Robotics data is NOT a second-class citizen - it is too important of a problem to be left to fine tuning! Your model needs to see your most important tokens from step 0 We need to think about robotics through the first principles of AI: how do we best utilize the vast amounts of web-scale media and how do we create a data-flywheel to collect millions of hours of rich robot interactions. There is no other moat in AI outside of data and @1x_tech has done an INCREDIBLE job scaling manufacturing, production and hardware to build humanoid robots that can create a unique data-flywheel in unstructured environments. Scaling data collection for highly dexterous on-policy robot data will be the only way for creating a moat in AI. @JackMonas and team have made great progress in building World Models, and now the goal is to supercharge this effort by starting a hyper-focused scale and data-pilled lab. Before scaling compute / data / models, we are currently RAPIDLY scaling our team and hiring across the 4 core pillars of AI: model + data, data infra, ML infra and evals. Looking for folks that are excited about the 0->1 problem and share the same principles as us. There’s a single application for everyone in the lab - if you’re a good at engineering and ML, we will find a place for you in the team ❤️ AGI won’t be solved by fine-tuning… Let’s build the next frontier of AI together 🚀 My DMs are always open!!

Robot (@arthurallshire)

@_sam_sinha_ @LumaLabsAI Congrats Sam !

After 4 incredible years, today marks my last day at @LumaLabsAI. I’m so proud to have helped build the company from 5 to 250 people. Luma started as a humble attempt at bringing NeRFs to smartphones in 2022, and is currently an extremely successful startup capable of training frontier models that can truly compete with the best generative models in the world. I have so much love and respect for the whole team but I want to give a special thanks to @baaadas - for trusting me 3 years ago and deciding to join Luma and help us be a serious generative AI company, and then for building a culture that allowed me to do my best work ❤️ Finally, I’m so proud of our most ambitious and impressive model release: Uni-1, an industry defining unified multimodal model that will shape the future of the company, and the field for years to come. Working relentlessly on the project with some of my favourite people was the highlight of my career thus far. This project wouldn’t be possible without @shenbokui for his leadership and the rest of the Omni team for the incredible execution and dedication to the mission 🚀 Will take a week off to visit 🇨🇦 but have some exciting updates to share soon after 👀

@ronskoro Thanks! We didn’t try with distilled students though that sounds promising. Our experiments initialized from pretrained SD3 checkpoints, would you consider that late training?

@davidrmcall really cool work. the uniform copy beating backprop-through-chain (rows I vs J) leans on flow jacobians being roughly psd via ot straightness. did you try it on distilled few-step students or late-training checkpoints where flows get more rotational?

We developed a simple, sample-efficient online RL technique for post-training image generation models. We see it as a possible steerable alternative to CFG, driven by any scalar reward, including human preference.

@ayaanzhaque Congrats Ayaan!

Thinking mode demo! Images 2.0 now thinks before generating images, collects information with various tools, and plans out each image! Give it a try by clicking “thinking” in ChatGPT!

Check out our blog post at mcallisterdavid.com/fdfo-blog for a walkthrough of our design decisions. w/ fantastic collaborators: Miika Aittala, Tero Karras, Janne Hellsten, @akanazawa Timo Aila and Samuli Laine

@KushtimusPrime Congrats!

Our new work, STITCH 2.0, can perform consecutive running sutures to close a sample wound with the daVinci robot.

@vonekels Congrats!

When people share a space, their movements become intertwined. Embodied agents need to understand these social dynamics to interact effectively. Introducing MAGNet 🧲, a unified autoregressive diffusion forcing model for multi-agent motion generation that captures these interactions. MAGNet is flexible: predict the future, fill in missing motion, or have people react to each other, all while naturally scaling to N>2 people and generating ultra-long motion sequences.

𝗢𝗻𝗲 𝗺𝗲𝗺𝗼𝗿𝘆 𝗰𝗮𝗻’𝘁 𝗿𝘂𝗹𝗲 𝘁𝗵𝗲𝗺 𝗮𝗹𝗹. We present 𝗟𝗼𝗚𝗲𝗥, a new 𝗵𝘆𝗯𝗿𝗶𝗱 𝗺𝗲𝗺𝗼𝗿𝘆 architecture for long-context geometric reconstruction. LoGeR enables stable reconstruction over up to 𝟭𝟬𝗸 𝗳𝗿𝗮𝗺𝗲𝘀 / 𝗸𝗶𝗹𝗼𝗺𝗲𝘁𝗲𝗿 𝘀𝗰𝗮𝗹𝗲, with 𝗹𝗶𝗻𝗲𝗮𝗿-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 in sequence length, 𝗳𝘂𝗹𝗹𝘆 𝗳𝗲𝗲𝗱𝗳𝗼𝗿𝘄𝗮𝗿𝗱 inference, and 𝗻𝗼 𝗽𝗼𝘀𝘁-𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻. Yet it matches or surpasses strong optimization-based pipelines. (1/5) @GoogleDeepMind @Berkeley_AI