Tommy Ly

I joined the @IntrinsicAI AI for Industry challenge to learn robotics from scratch The goal is to plug a fiber cable into a 6mm port using only wrist-mounted cameras + force feedback. Setup: 6-DOF arm, Robotiq gripper, 3 RGB cameras on the wrist. No overhead view, no markers, no depth sensors. I didn't make the top 30, but here's everything I learned trying 🧵

@Control_wiz follow-up question, would you say that locomotion and SLAM are solved? We just need to finetune on hard terrain via simulation but the field is pretty much done done

@lymytom20 I haven't followed more recent work, but Drone racing and RL locomotion have been established since ~2018. Locomotion on relatively structured terrain is a fairly mature problem, and MPC variants have also shown strong results. Backflips are cool demos, but nothing more.

Always do MPC first. A lot of useful problem can be mapped to an MPC. RL is inefficient.

The Great Space Corridor

Sleep well, NY.

“The Savior of the Broken” Jalen Brunson

@Control_wiz youtube.com/watch?v=C4NQNe… yup pretty much what Mac Schwager found

VLA based models can only interpolate over the support of its training/pre-training distribution. To become a generalist, it would need training coverage dense enough to span an enormous (possibly unbounded) task space, which is not scalable. A generalist needs to learn on the job and plan accordingly.

@yunta_tsai thank you for pushing humanity forward

I am grateful to work under the greatest entrepreneur of our lifetime. If you had asked me 10 years ago if I would be given such a chance, I would have told you what kind of crack you were smoking. There were many stories written by people who never worked for him. But we work for him not because he is powerful or rich, but because he is inspiring. When immigrants like us, bringing nothing but a small suitcase to the US, can achieve something great, it is truly an American dream.

don't betray your destiny. deep down, you know what you want to become

newsletter.rootsofprogress.org/p/the-grand-pr… "They face a field of unknown unknowns, and no one can calculate the probability or predict the timing of their success. They stand alone against critics who say they’re idiotic or deluded, and they manifest unshakeable confidence that they will succeed, despite having no map of the path to their destination"

@ErenChenAI @TairanHe99 Can you share the link? Also I wanna ask how did he complete his PhD in < 3 years at CMU. That's such an insane pace!

Everything Was Already Determined at the Moment of the Big Bang This idea came up in @TairanHe99 Tai Ran’s podcast with OpenAI Research Engineer Jiayi Weng. If the universe is fundamentally deterministic, then every decision, every success, every mistake, and even this tweet may have been inevitable from the very beginning. One of the most thought-provoking conversations I’ve watched recently. Highly recommended.

@ErenChenAI This is the paper - agile-but-safe.github.io

Autonomous obstacle avoidance on a quadruped robot from Prof. Changliu Liu’s group at CMU.

@Control_wiz you can be the next Gwern but for robotics

Without doxxing myself, I’m a third-year grad student in control theory/robotics. I have about 2.5 years of ML and controls experience through co-ops and internships. Never claimed to be anyone special, just sharing my thoughts. Happy to be wrong.

@Control_wiz american dynamism = defense tech physical ai = robotics world model = Pearlian causal model high agency = motivated taste = vision the media tends to invent new term for old vocab

I’ve been in robotics for about 7 years. For most of it, especially in controls, it was a pretty niche field. Now it seems like “physical AI” is the thing to do. I’m glad robotics is getting all this attention, and we need more people to tackle these problems. But I also think it's overhyped and misleading. A lot of the spotlight is on curated demos or robots dancing/doing flips. They’re cool, but they make the field seem much further along than it really is. The hard problems, robust manipulation, sim2real, long horizon autonomy, reliability and more, don’t make for a cool 30 sec demo.

@JonathanKimXD @Control_wiz afaik, nobody knows who's this person gwern.net

@Control_wiz Not to be that guy, but can the public get your background so we can weigh your words

One of my goals with this account is to share my honest thoughts on where I think the frontier of robotics actually is. I might sound like a pessimist, but I think the field deserves realism. I’m happy robotics is getting all this attention, but I also have to admit that a lot of what we see in both industry and academia is exaggerated. Curated demos make the field seem much more advanced than it really is. I’ll share my thoughts, for what they’re worth.

@Control_wiz I guess Stephen Boyd Linear Dynamical System class is the OG world model

Wait till they find out that the OG world model is Ax + Bu

@Control_wiz youtube.com/watch?v=kYfNvm… It's Steve Jobs

"This world is built by people no smarter than you." I read this here and idk who it's from, but I can't stop thinking about.

In the world of robotics, machine learning is one small part of the equation. It also involves working with materials that may change over lifecycles to deliver persistent performance. For example, oxidation happens at different rates depending on humidity and thermal gradients, which in turn change material properties such as stiffness or elasticity. You know someone is knee-deep in robotics if asking them oxidation questions generates the same facial expression as asking normalization questions to a machine learning engineer.

I have so much fun writing this position with some of the most amaaazing people in robotics! Have a look at it here: paper.motoniq.ai #AI #MachineLearning #Robotics

imagine having 3 titans sitting for an oral exam..

My college degree recommendations for fresh high school graduates in the age of AI to be prepared for the next frontier: - Applied Physics (math and physics) - Applied Materials (physical agentic) - Agriculture (food production in space) - Aerospace (frontier transportation) - Civil Engineering (infra + life support from Earth and beyond) - Electronic Engineering (scaling compute and communications to the solar system) - Manufacturing (where things get made) - Mechanical Engineering (packaging, manufacturability, tolerances and cycling) - Medicine (personal drugs in space)