Xiaochuan Li

Agentic test-time scaling (TTS) is effective -- until you find its inherent limits. 💡We show that classic TTS methods offered limited practical gains due to two fundamental limitations: the context ceiling and the verification gap. 🧵 Check the website: general-agentbench.github.io

Routines in Claude Code? Route (of models) in Claude Code!

What makes us humans general agents? 🤔 It’s probably not that we’ve mastered every app or UI. New tools appear constantly, yet we become productive quickly. What really transfers are our cognitive abilities: how we perceive, reason, and use memory. 🧠 Introducing CocoaBench, an evaluation framework for general agents with compositional cognitive abilities. 👉 Features 1⃣Complex, realistic tasks🧩 • Human-crafted tasks that are long-horizon, understandable, and span diverse scenarios and domains • Assume only a small set of general tools (browser, terminal, file system; no per-task APIs) • Challenging for existing agent systems. ChatGPT Agent reaches only 44% success rate. • Check out our example tasks on the website (link in reply) and see if you can solve them. They’re fun 🙂 2⃣Covering diverse cognitive abilities 🧠 CocoaBench covers different choices in the following dimensions of cognitive architecture: • Perception: how the agent gathers and preprocesses information from websites, terminal outputs, files, and images • Reasoning : planning, deductive/inductive/abductive inference, and both symbolic & visual reasoning skills • Memory : managing working memory in the long horizon, saving and loading procedural memory (skills), etc. 3⃣CocoaAgent framework🛠️ • Seamless integration with the AIO Sandbox for isolated browser / terminal / file operations • Easy to plug in any models or your own custom agent • Flexible evaluation functions This is just our first release, and we’re actively expanding CocoaBench with more tasks, analyses, and agents. Excited to see what the community builds on top of it! Project website + more details in the thread👇

@nlpxuhui @MSFTResearch @MaartenSap 🥳

Thrilled to be a 2026 @MSFTResearch PhD Fellow! Huge thanks to my advisor @MaartenSap , collaborators, and friends ❤️❤️ Missions ahead 🫡

Thanks for sharing our work! If you are interested in the “upper bound of test time scaling”, welcome to read and give feedback!

As long-horizon software engineering tasks grow in complexity, a single agent can no longer finish the tasks alone — effective multi-agent collaboration becomes necessary. This leads to a natural question: how can multiple agents be coordinated to asynchronously collaborate over a shared artifact in an effective way? We answer this question in our new preprint: Effective Strategies for Asynchronous Software Engineering Agents! We suggest that to coordinate multiple software engineering agents, branch-and-merge is the key coordination mechanism, and that human SWE primitives like git worktree, git commit, and git merge are all you need to support it. (1/n)

Creating user simulators is a key to evaluating and training models for user-facing agentic applications. But are stronger LLMs better user simulators? TL;DR: not really. We ran the largest sim2real study for AI agents to date: 31 LLM simulators vs. 451 real humans across 165 tasks. Here's what we found (co-lead with @sunweiwei12).

🌹This resource is invaluable to anyone (including me) who is beginning to explore the model's architecture.

@huybery Best wishes Hui Bro!

bye qwen, me too.

🚀Excited to share that we bridge the connection of Clawbot & Simworld! 🧩We are motivated to move beyond isolated toy tasks and into a shared physical world with routines, interactions, and coordination. 🚧Lightweight setup: plug in your own agent easily!

Midtraining is a new part of many training pipelines, but when does it help and can it backfire? 🤔 In our new preprint, we use controlled experiments to pin this down. TL;DR; midtraining helps the most when it “bridges” pretraining and posttraining, and mitigates forgetting after posttraining. Timing is also very important. 🧵

[8/8] Paper: arxiv.org/pdf/2602.18998 Code: github.com/cxcscmu/Genera… Website: general-agentbench.github.io Thanks to all who contributed to this work! @TimMing10252742 @AbhijayPaladugu PranavSetlur AndyTang @haok1402 ShuaiShao RongJin @XiongChenyan

[7/8] Takeaway: “More compute at test time” is insufficient. The context ceiling and verification gap serve as bottlenecks for effective improvement. We hope the community can develop better context management methods and improve model self-cognition to enable truly TTS.

Agentic test-time scaling (TTS) is effective -- until you find its inherent limits. 💡We show that classic TTS methods offered limited practical gains due to two fundamental limitations: the context ceiling and the verification gap. 🧵 Check the website: general-agentbench.github.io

1/⚠️ Parallel test-time scaling (e.g., pass@k) usually wastes compute - models often repeat the same dominant failure❌ How should we effectively generate creative solutions? While typical methods such as increasing temperature 🌡️ usually fail, we put forward Mode‑Conditioning (ModC) - a simple yet powerful training and test-time framework that allocates compute across diverse reasoning modes🎨We show that ModC largely improves pass@k across SFT, distillation, and RL settings. With ModC, we get 4-8x efficiency gains in math reasoning using the same training data!