Yu Fang

If you're curious about the background that inspires a lot of our group's research on skill learning and video understanding, check out this great piece by UNC Research. It covers some of my journey from being a basketball player to an AI researcher. research.unc.edu/story/reading-…

✍️Authors: Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding Thanks to my co-authors! @JiaqiLiu835914 @YYang9923 @dingmyu @unccs [6/6]

🤖Real-world Experiments We study different aspects of language grounding. Each scene is designed with three possible tasks: one well-learned in-domain task with sufficient demonstrations, and two under-observed tasks defined by counterfactual instructions. Across object recognition, spatial reasoning, goal targeting, out-of-distribution generalization, and long-horizon reasoning, our proposed CAG consistently improves the performance of pi0.5 and reduces counterfactual failures. [5/6]

Do Vision-Language-Action Models truly follow your language instructions? We present When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. They promise to ground language instructions in robot control, yet in practice, often fail to follow language faithfully. 📄 Paper: arxiv.org/abs/2602.17659 🌐 Project: vla-va.github.io 💡 Highlights Vision shortcuts and counterfactual failures. When given instructions that lack strong scene-specific supervision, they default to well-learned scene-specific behaviors regardless of language intent. Counterfactual benchmark. We introduce LIBERO-CF, the first counterfactual benchmark for evaluating language following in VLAs. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs. Our solution. We propose Counterfactual Action Guidance (CAG), a simple plug-and-play dual-branch inference scheme that strengthens language conditioning without changing pretrained VLA architectures or weights. Experiments. CAG is effective across multiple dimensions of language grounding, consistently improving both language grounding and task success on under-observed tasks. #VLA #Robotics #Vision #Language

An article from Moonlake about why they’re building what they’re building. (I’m an advisor)

my first PI project: we added memory! this is a step function capabilities unlock: 15 minute long multi-step tasks in novel environments, controlled by text prompting having run many of the evals, I legit think this is the GPT 2 moment for robotics

Introducing OHRA (One Hand to Rule Them All) — a canonical representation that unifies diverse dexterous robot hands into a shared space, enabling cross-hand policy transfer and up to 81.9% zero-shot generalization to unseen morphologies 🌐zhenyuwei2003.github.io/OHRA arxiv 2602.16712

Train Beyond Language. We bet on the visual world as the critical next step alongside and beyond language modeling. So, we studied building foundation models from scratch with vision. We share our exploration: visual representations, data, world modeling, architecture, and scaling behavior! [1/9]

Introducing FOFPred — a language-driven future optical flow prediction framework that enables improved robot control and video generation. Instead of reacting to motion, FOFPred predicts how motion will evolve — conditioned on natural language. 🌐 Project: fofpred.github.io 📄 Paper: arxiv.org/abs/2601.10781 💻 Code: github.com/SalesforceAIRe… 🤗 Model: huggingface.co/Salesforce/FOF… 🕹️ Demo: fofpred.salesforceresearch.ai 🧵[1/3]

What if we can simulate an *interactive 3D world*, from a single image, in the wild, in real time? Introducing PointWorld-1B: a large pre-trained 3D world model that predicts env dynamics given RGB-D capture and robot actions. 🌐 point-world.github.io from @Stanford @nvidia

Great collaboration with Kanchana Ranasinghe @kahnchana, Le Xue @Le_Xue01, Honglu Zhou @zhou_honglu, Juntao Tan, Ran Xu @stanleyran, Shelby Heinecke @shelbyh_ai, Caiming Xiong @CaimingXiong, Silvio Savarese @silviocinguetta, Daniel Szafir, Mingyu Ding @dingmyu, Michael S. Ryoo @ryoo_michael, Juan Carlos Niebles @jcniebles! @SFResearch @unccs

🎬Qualitative Results Check out our project page for more results!

🤖Robotic VLA Benefits from Joint Learning with Motion Image Diffusion We introduce joint learning with motion image diffusion that enhances VLA models with motion reasoning capabilities. 📄Paper: arxiv.org/abs/2512.18007 🌐Project: vla-motion.github.io Key Highlights 🧠Our method seamlessly augments VLA models with motion reasoning capabilities, while preserving their real-time inference efficiency. 🔎We present motion image diffusion using a DiT, providing dense pixel-level dynamic supervision that complements sparse action supervision. We show that the optical-flow-based motion images are the most effective representation for joint action-motion learning. 🎯We enhance π-series VLA models to achieve 97.5% average success on LIBERO and 58.0% on RoboTwin. #VLA #Robotics #Motion