FRIDA Robot Painter

Some of Frida's first paintings! More to come

This paper shows why a single bad LLM instruction can turn a robot helper into a hazard. This study treats robot decisions like safety checks, because 1 rare mistake can still cause damage. The authors test LLMs in fire and navigation prompts, and some answers point away from the exit. In a fire escape prompt, Gemini 2.5 Flash picked the professor’s room 32% and a server room 1%. The problem is that robots act in the real world, so a wrong instruction can turn into a real accident. To study this, the authors build 7 tests that cover clear text maps, missing information, and emergency stories where a bad choice can hurt someone. They repeat the same prompts many times, then score answers by checking if the path reaches the goal, avoids obstacles, and stays consistent. Results look good on easy cases, but some models suddenly lose the map layout, make up details, or stop halfway through the route. The main takeaway is that even 99% accuracy can be unsafe in robotics, so LLMs are not ready to run robots alone. ---- Paper Link – arxiv. org/abs/2601.05529 Paper Title: "Safety Not Found (404): Hidden Risks of LLM-Based Robotics Decision Making"

IEEE RAMagazine welcomes articles for the Special Issue on the Arts and Robotics: ieee-ras.org/publications/r…

@vernon_luk @jeanoh Check out our website, paper, or code to learn more: website: rewarped.github.io paper: arxiv.org/abs/2412.12089 code: github.com/rewarped/rewar… All of our code, simulation environments & algorithms, is open source and available now: 𝐩𝐢𝐩 𝐢𝐧𝐬𝐭𝐚𝐥𝐥 𝐫𝐞𝐰𝐚𝐫𝐩𝐞𝐝 7/7

This was a wonderful collaboration over last year with @vernon_luk, @jeanoh. And it would not have been possible without the amazing open source effort by the NVIDIA Warp team. 6/n

We show that SAPO can learn a range of challenging manipulation and locomotion tasks, re-implemented in Rewarped, that involve interaction between rigid bodies, articulations, and deformables. SAPO beats model-free RL (PPO, SAC) as well as prior first-order RL (APG, SHAC). 5/n

SAPO (Soft Analytic Policy Optimization): maximum entropy first-order RL algorithm that uses analytic gradients from diff. sim to train a stochastic actor to maximize expected return and entropy. Why MaxEnt RL? Smoother optimization landscapes stabilize first-order RL. 4/n

Rewarped: parallel differentiable multiphysics simulation platform that supports simulating various materials beyond just rigid bodies. Another simulator? Re-write tasks in Rewarped and scale first-order RL on tasks involving rigid and soft body interaction. 3/n

Model-free RL is too sample inefficient to scale on slow soft body simulation. Prior first-order RL algorithms work on rigid body locomotion, but fail to solve soft body manipulation. No existing simulator satisfies all: (1) parallel envs (2) differentiable (3) multiphysics. 2/n

RL is notoriously sample inefficient. How can we scale RL on tasks much slower to simulate than rigid body physics, such as soft bodies? In our #ICLR2025 spotlight, we introduce both a new first-order RL algorithm, SAPO, and differentiable simulation platform, Rewarped. 1/n

Happy Valentine's Day from BIG

Can soft robots rapidly spin pens like humans?🤔 We’ve shown that soft robot hands can master the dynamic tasks of pen spinning—no hours of GPU training or complex sim-to-real needed! Check out soft-spin.github.io. 🤖✍️ @CMU_Robotics. 1/5🧵

Our paper, Robot Synesthesia: a sound and emotion guided robot painter, won the IROS Best Entertainment and Amusement Paper Award! Congrats, Vihaan Misra and Peter Schaldenbrand @PeteyRobots #cmubig #IROS #IROS2024

Tune in to see some special footage of @Ken_Goldberg working on his painting robot!

This video was created with @gchenfc @vihaan_misra Lorie Chen @Ken_Goldberg and @JeanOhCmuBIG Presented at @ieee_ras_icra ICRA@40

Our ICRA@40 video shows some of the history of robot painting projects, the challenges these roboticists faced, and how they made core contributions to the field of Robotics. It doesn't have to be Robotics for Arts, it can be Arts for Robotics! youtu.be/M3D9cWNf7r0?si…

FRIDA on WPXI Channel 11 news wpxi.com/news/local/mee…

Our collaborative painting paper, CoFRIDA, has been nominated for Best Paper on Human-Robot Interaction at #ICRA2024 We're also doing a demo on Thursday 5/16 at the ICRA Expo if you'd like to try it for yourself! pschaldenbrand.github.io/cofrida/ @ieee_ras_icra

We are thrilled to announce the NeurIPS 2024 Creative AI track! We invite research papers and artworks showcasing innovative work of AI and ML in art, design, and creativity. This year's theme is "Ambiguity." For more information: neurips.cc/Conferences/20… #neurips2024 #CreativeAI

FRIDA@Amazon MARS 2024 @PeteyRobots @FridaRobot @UksangYoo

@ieee_ras_icra @CMUBigLab @GauravTParmar @junyanz89 @1x @JeanOhCmuBIG A pre-print of our ICRA paper can be read here: arxiv.org/abs/2402.13442 More results on our website: pschaldenbrand.github.io/cofrida/