Sharon Li@SharonYixuanLi
Scaling outcome-based RL won't solve long-horizon agentic tasks. Credit assignment is the bottleneck, and turn-level reward is inevitable.
The question is how to get it. Process labels are expensive; we all know that.
Excited to release TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents. No trained critic, no process labels, no Monte Carlo continuations, no strong LLM judge. 🧵
📄 arxiv.org/abs/2607.13988
🤔 The core problem:
Outcome rewards work well when a task contains a short reasoning trace and an easily verified answer. But agents may perform tens or hundreds of tool interactions before responding.
A single final reward tells us whether the agent succeeded, but not which tool calls uncovered decisive evidence, which were redundant, or which sent the agent down the wrong path. This makes RL increasingly sparse, noisy, and inefficient as trajectories grow longer.
⚙️ Our approach: TRACE
TRACE assigns credit at individual tool-call boundaries.
For each trajectory prefix, a frozen reference model measures how much more predictable the gold answer has become. TRACE converts this into a state value and uses temporal-difference changes to reward each interaction:
→ Positive credit when a tool call moves the agent closer to the answer
→ Near-zero credit when it adds redundant information
→ Negative credit when it moves the agent away from the answer
Probability to predict the final answer remains the global training objective; turn-level rewards simply reveal which actions contributed to it.
🔍 Why the formulation matters:
The turn-level rewards telescope across the trajectory. Repeating searches or reopening the same evidence cannot artificially accumulate additional one-step credit.
TRACE can also propagate delayed progress backward: if a search surfaces a promising page and a later `open` reveals the decisive evidence, both interactions can receive appropriately localized credit.
📈 The results:
Using pure RL (without cold-start SFT, agentic mid-training, or live-web training data), TRACE substantially improves long-horizon search:
→ Qwen3-4B: 7.2 → 35.6 on BrowseComp-Plus
→ Qwen3-30B-A3B: 8.4 → 42.6 which is on-par with the Tongyi-DeepResearch-30B-A3B.
Under controlled comparisons with the same models, data, tools, and outcome rewards, TRACE also outperforms GRPO, GSPO, and GiGPO across both model scales.
📄 Paper: arxiv.org/abs/2607.13988
Led by the amazing @LeitianT, and huge thanks to our collaborators at Microsoft Research: Baolin Peng, Wenlin Yao, Tao Ge, Hao Cheng, Mike Hang Wang, and @JianfengGao0217.