Amazing and timely paper! Congrats to the authors! 👏👏👏 This opens up exciting prospects for sample-efficient continual learning in LLMs. But, more importantly, this is a perfect complement to training-efficient gradient-free optimization methods (GEPA, AlphaEvolve, etc) 🤞 With this result, we could use GEPA on critique feedback to find Pareto-optimal in-context changes to improve performance in an empirical way while creating useful sample trajectories at the same time. Together with context distillation (here called self distillation), we can move these learnings into the model parameters without catastrophic forgetting 🎊 And, if you want to have some RL-like reward shaping, you can just multiply the distillation loss by a per-sample or per-token advantage, too 😊 I've been thinking about GEPA+context distillation before, and the big open research question for me was catastrophic forgetting. I was hoping that like RL, it's more on-policy, and other works have shown that RL suffers less from catastrophic forgetting. This work already answers this question in the affirmative. Really exciting! This would be quite interesting and easy to try in e.g. Tinker probably 😊
