Training reasoning agents has long been stuck between two tricky options:
1- Reinforcement learning with verifiable rewards (RLVR), which is low-cost but also tricky due to sparse rewards.
2- On-Policy Distillation (OPD) from larger models, which provides granular feedback on responses but is costly because it requires a large teacher.
A third option is on-policy self-distillation (OPSD), which solves the costs of OPD but results in low-quality training due to information leakage from the teacher to the student.
RLVR with Self-Distillation (RLSD), a new technique by researchers at JD.com, addresses this problem by making small changes to the self-distillation process. It uses the sparse signal from the verfiable reward to determine the direction of the update (i.e., whether to reinforce or penalize a behavior). And it uses the signal from the self-distillation to determine the magnitude of the update (i.e., how much relative credit or blame a specific step deserves).
The result: RLSD at 200 training steps already beats RLVR with GRPO trained for 400 steps while avoiding the costs of OPD and the poor training quality of OPSD.
VentureBeat@VentureBeat
How to build custom reasoning agents with a fraction of the compute venturebeat.com/ai/how-to-buil…
English
