Yiting Chen (@YitingChen07) - Twitter Profili | Zamantika Mersobahis Locabet

Sabitlenmiş Tweet

Yiting Chen@YitingChen07·8 Eki

“You can’t make progress until you are able to measure it. Robotics still doesn’t have such a rallying call. No one agrees on anything.” I 💯 agree with the recent post from @DrJimFan. To break this impasse, we are excited to announce ManipulationNet (manipulation-net.org), a community-driven global infrastructure supported by @NIST, that enables benchmarking real-world robot manipulation research at scale with any robot at any time and anywhere on standardized task setups. ManipulationNet’s key features include: ✅delivers standardized hardware kits globally to support reproducible task setups ✅provides online task instructions in real-time, which could involve visual/language prompts, task-specific instructions, etc.. ✅collects authentic manipulation performance for comparable results on global leaderboards 🌍Join the ManipulationNet, let us aim to "connect the dots" of the up-to-date progresses and challenges to construct a network of real-world robot abilities and skills, a network of research roadmaps, and a network of research questions. Want to learn more about ManipulationNet? Check out the links below: 🔗Project website: manipulation-net.org 🔗Founding Committee and Developer Team: manipulation-net.org/committee.html 🔗Paper: manipulation-net.org/MNet_preprint.… (1/7)

English

9

22

109

26.7K

Yiting Chen retweetledi

Chris Paxton@chris_j_paxton·29 Oca

New blog post: how do we quantify progress in robotics? Benchmarks have made a huge difference in other areas of AI, but we don't have an equivalent to imagenet or SWE-Bench for robotics. Why not and what are people doing about it? Simulations, distributed evaluation, and real-world evaluations are all options, but come with serious costs. This makes it hard to compare robotics results and makes it hard to determine which progress is real.

English

14

11

143

18.8K

Yiting Chen@YitingChen07·30 Oca

Thank you @Ken_Goldberg for this amazing task! Looking forward to seeing how different systems tackle deformable linear object manipulation in the real world.

Ken Goldberg@Ken_Goldberg

Help measure progress in robotics research: a specific & verifiable physical benchmark for cable routing, part of a broader ManipulationNet effort by Kaiyu Hang (Rice U) and Kenny Kimble (NIST) and others from MIT, Berkeley, CMU, Columbia, UT, Tsinghua... manipulation-net.org/tasks/cable_ro…

English

0

2

132

Yiting Chen retweetledi

Yu Xiang@YuXiang_IRVL·26 Kas

Excited to share that we contributed our SceneReplica work into ManipulationNet as a task for Grasping in Clutter! ManipulationNet is a community effort for benchmarking in robotics. Evaluate your grasping algorithms or policies here: manipulation-net.org/tasks/grasping…

Yu Xiang@YuXiang_IRVL

If you are interested in benchmarking in robotics, we introduce a real-world pick-and-place benchmark by creating replicable scenes! irvlutd.github.io/SceneReplica/

English

1

18

128

18.9K

Yiting Chen@YitingChen07·5 Kas

Besides charts, we need real-world benchmarks to actually tell the manipulation performance. Check more on ManipulationNet: manipulation-net.org and more tasks will be released soon.

Generalist@GeneralistAI

GEN-0 models exhibit strong scaling laws, in which more pretraining data and compute consistently (and predictably) improve downstream post-training performance of the model across many tasks. 🔬📈🧠

English

0

1

165

Yiting Chen@YitingChen07·14 Eki

@chris_j_paxton Or we can say “locomotion is a simplified form of manipulation”

English

0

1

247

Chris Paxton@chris_j_paxton·13 Eki

Why is manipulation harder than navigation/locomotion

Bryson Jones@brysonkjones

@chris_j_paxton Wrote about this a bit recently: open.substack.com/pub/brysonkjon…

English

5

2

61

12.1K

Yiting Chen@YitingChen07·13 Eki

This is the correct paradigm for benchmarking manipulation and now it arrives! We always welcome new task proposals; our infrastructure can support them all. More details at manipulation-net.org

Chris Paxton@chris_j_paxton

As much as you can do system-level evaluation on the cheap this is the only way i think. - mail out standard object sets - have a client which records task performance and streams it

English

0

5

937

Yiting Chen@YitingChen07·13 Eki

@VectorWang2 Quantitative evaluation for embodied reasoning is not easy, at least we can start with those simple setups!

English

0

93

Vector Wang@VectorWang2·12 Eki

Believe me, though you can maybe build these with just an SO101 arm, these are harder than most of the home tasks in Vision & Language (and the physics behind) reasoning. A great way to quantitatively evaluate the "VL" ability in VLA models I would say.

English

2

6

37

5.6K

Yiting Chen@YitingChen07·10 Eki

@QianzhongChen Thanks! We will release more tasks in the future, stay tuned!

English

0

51

Qianzhong Chen@QianzhongChen·10 Eki

Great work for making fair robotics policy benchmarking!

Yiting Chen@YitingChen07

“You can’t make progress until you are able to measure it. Robotics still doesn’t have such a rallying call. No one agrees on anything.” I 💯 agree with the recent post from @DrJimFan. To break this impasse, we are excited to announce ManipulationNet (manipulation-net.org), a community-driven global infrastructure supported by @NIST, that enables benchmarking real-world robot manipulation research at scale with any robot at any time and anywhere on standardized task setups. ManipulationNet’s key features include: ✅delivers standardized hardware kits globally to support reproducible task setups ✅provides online task instructions in real-time, which could involve visual/language prompts, task-specific instructions, etc.. ✅collects authentic manipulation performance for comparable results on global leaderboards 🌍Join the ManipulationNet, let us aim to "connect the dots" of the up-to-date progresses and challenges to construct a network of real-world robot abilities and skills, a network of research roadmaps, and a network of research questions. Want to learn more about ManipulationNet? Check out the links below: 🔗Project website: manipulation-net.org 🔗Founding Committee and Developer Team: manipulation-net.org/committee.html 🔗Paper: manipulation-net.org/MNet_preprint.… (1/7)

English

1

19

5.5K

Yiting Chen@YitingChen07·10 Eki

@YouJiacheng @kaiwynd @NIST will update the webpage soon in the future, stay tuned!

English

0

1

28

You Jiacheng@YouJiacheng·10 Eki

@kaiwynd Hmmm where can I find NIST ATB M1? I can only find NIST ATB 1,2,3,4 but not M1...

English

2

0

336

Kaifeng Zhang@kaiwynd·9 Eki

The pursuit of a unified benchmark for robot manipulation has been long-standing — and with ManipulationNet, it’s finally taking a leap forward! manipulation-net.org ManipulationNet is a fully real-world, verifiable, and scalable benchmark for robotic manipulation. Anyone can evaluate and submit results via the mnet-client. Evaluations are centralized, and rankings are global. Born from the NIST Assembly Tasks, now expanding to embodied reasoning, peg-in-hole, and beyond! #robotics #AI #benchmark

Yiting Chen@YitingChen07

“You can’t make progress until you are able to measure it. Robotics still doesn’t have such a rallying call. No one agrees on anything.” I 💯 agree with the recent post from @DrJimFan. To break this impasse, we are excited to announce ManipulationNet (manipulation-net.org), a community-driven global infrastructure supported by @NIST, that enables benchmarking real-world robot manipulation research at scale with any robot at any time and anywhere on standardized task setups. ManipulationNet’s key features include: ✅delivers standardized hardware kits globally to support reproducible task setups ✅provides online task instructions in real-time, which could involve visual/language prompts, task-specific instructions, etc.. ✅collects authentic manipulation performance for comparable results on global leaderboards 🌍Join the ManipulationNet, let us aim to "connect the dots" of the up-to-date progresses and challenges to construct a network of real-world robot abilities and skills, a network of research roadmaps, and a network of research questions. Want to learn more about ManipulationNet? Check out the links below: 🔗Project website: manipulation-net.org 🔗Founding Committee and Developer Team: manipulation-net.org/committee.html 🔗Paper: manipulation-net.org/MNet_preprint.… (1/7)

English

3

45

9K

Yiting Chen@YitingChen07·9 Eki

@xiatao_sun @DrJimFan Thanks Xiatao! We hope this new infrastructure could bring us one step closer to this goal. Stay tuned!

English

0

1

75

Xiatao Sun@xiatao_sun·9 Eki

@YitingChen07 @DrJimFan This is exactly what the field needs. Standardized hardware kits + distributed benchmarking could finally give us apples-to-apples comparisons across labs. Excited to see how this develops!

English

1

0

97

Yiting Chen@YitingChen07·8 Eki

“You can’t make progress until you are able to measure it. Robotics still doesn’t have such a rallying call. No one agrees on anything.” I 💯 agree with the recent post from @DrJimFan. To break this impasse, we are excited to announce ManipulationNet (manipulation-net.org), a community-driven global infrastructure supported by @NIST, that enables benchmarking real-world robot manipulation research at scale with any robot at any time and anywhere on standardized task setups. ManipulationNet’s key features include: ✅delivers standardized hardware kits globally to support reproducible task setups ✅provides online task instructions in real-time, which could involve visual/language prompts, task-specific instructions, etc.. ✅collects authentic manipulation performance for comparable results on global leaderboards 🌍Join the ManipulationNet, let us aim to "connect the dots" of the up-to-date progresses and challenges to construct a network of real-world robot abilities and skills, a network of research roadmaps, and a network of research questions. Want to learn more about ManipulationNet? Check out the links below: 🔗Project website: manipulation-net.org 🔗Founding Committee and Developer Team: manipulation-net.org/committee.html 🔗Paper: manipulation-net.org/MNet_preprint.… (1/7)

English

9

22

109

26.7K

Yiting Chen@YitingChen07·9 Eki

@tarikkelestemur Absolutely! That’s our goal. Insights from the previous effort helped us build a better infrastructure this time. Stay tuned!

English

0

1

35

Tarik Kelestemur@tarikkelestemur·9 Eki

This is really cool. The previous NIST benchmarks were great but were not by majority of the people in the community. I hope this one brings us one step closer to have a common benchmark. I’d love if every lab/startup would report results on these tasks.

Yiting Chen@YitingChen07

“You can’t make progress until you are able to measure it. Robotics still doesn’t have such a rallying call. No one agrees on anything.” I 💯 agree with the recent post from @DrJimFan. To break this impasse, we are excited to announce ManipulationNet (manipulation-net.org), a community-driven global infrastructure supported by @NIST, that enables benchmarking real-world robot manipulation research at scale with any robot at any time and anywhere on standardized task setups. ManipulationNet’s key features include: ✅delivers standardized hardware kits globally to support reproducible task setups ✅provides online task instructions in real-time, which could involve visual/language prompts, task-specific instructions, etc.. ✅collects authentic manipulation performance for comparable results on global leaderboards 🌍Join the ManipulationNet, let us aim to "connect the dots" of the up-to-date progresses and challenges to construct a network of real-world robot abilities and skills, a network of research roadmaps, and a network of research questions. Want to learn more about ManipulationNet? Check out the links below: 🔗Project website: manipulation-net.org 🔗Founding Committee and Developer Team: manipulation-net.org/committee.html 🔗Paper: manipulation-net.org/MNet_preprint.… (1/7)

English

2

0

379

Yiting Chen@YitingChen07·9 Eki

"One infrastructure to host a wide range of real-world benchmarking tasks" Now starts from the classic peg-in-hole assembly and block arrangement, to a general network of manipulation skills and abilities!

Kaifeng Zhang@kaiwynd

The pursuit of a unified benchmark for robot manipulation has been long-standing — and with ManipulationNet, it’s finally taking a leap forward! manipulation-net.org ManipulationNet is a fully real-world, verifiable, and scalable benchmark for robotic manipulation. Anyone can evaluate and submit results via the mnet-client. Evaluations are centralized, and rankings are global. Born from the NIST Assembly Tasks, now expanding to embodied reasoning, peg-in-hole, and beyond! #robotics #AI #benchmark

English

0

4

489

Yiting Chen@YitingChen07·8 Eki

A huge thank you to all our committee members: Kaiyu Hang, Kenneth Kimble, Edward H. Adelson, Tamim Asfour, @DanicaKragic, Xiang Li, @YunzhuLiYZ , Aaron Prather, Nancy Pollard, Maximo A. Roa-Garzon, Robert Seney, and @yukez, and our developer team Podshara Chanrungmaneekul, Shihefeng Wang, and @kaiwynd , for contributing to this ambitious project! (7/7)

English

0

3

314

Yiting Chen@YitingChen07·8 Eki

ManipulationNet is born as a scalable infrastructure to support a wide range of real-world benchmarking tasks! Whether the tasks require interactive commands or are pre-configured for automated execution, they can all be benchmarked at scale within this paradigm. (6/7)

English

0

1

242

Yiting Chen@YitingChen07·8 Eki

The mnet-server is a central manager for all client connections, which is responsible for handling all task-relevant activities. It supports task instructions delivery, performance logs collection in real-time for distributed benchmarking. In general, the mnet-server is responsible for ✅identify the team’s information and its registered benchmarking task; ✅delivers task-specific information and instructions in real-time for interactive benchmarking tasks; ✅collects performance metadata, for example, execution logs, to help describe the manipulation process; ✅ receive and store the manipulation performance video for further central auditing. After the verification of the submitted performance by the committee, ManipulationNet posts the result on the global leaderboards for comparable research. (5/7)

English

0

1

245

Yiting Chen@YitingChen07·8 Eki

The mnet-client is a middle layer between the robotic system and the mnet-server (will be introduced in the following post) to support distributed manipulation benchmarking on standardized task setups. The robotic system communicates with the mnet-client through ROS services and topics, and the mnet-client directly connects to the mnet-server. In general, the mnet-client is responsible for: ✅collect authentic manipulation performance on standardized task setups and upload it to the server for comparable research; ✅deliver task instructions from the server to the robotic system in real-time, this could involve language/visual prompts, task-specific instructions, and more; ✅report task execution and human intervention logs from the robotic system to the server in real-time to better describe the manipulation performance. (4/7)

English

0

1

323

Yiting Chen@YitingChen07·8 Eki

ManipulationNet provides a re-imagined paradigm, which relies on the integrated interaction between distributed hardware (for reproducible task setup in the real world) and software (for performance collection and task instructions delivery) to achieve the goal of distributed manipulation benchmarking at scale. Upon registration, the hardware will be mailed by ManipulationNet to ensure the quality and comparability. The software, mnet-client (github.com/ManipulationNe…), serves as a middle layer between the manipulation system and the mnet-server to support the benchmarking process. (3/7)

English

0

1

387

Yiting Chen

Keşfet