For a rigorous treatment of these topics, see Moritz Hardt's book on the science of machine learning benchmarks
mlbenchmarks.org
English
Ricardo Olmedo
126 posts
Ricardo Olmedo
@rdolmedo_
PhD student @MPI_IS, working with Moritz Hardt and Bernhard Schölkopf | Currently visiting @Stanford
Stanford Katılım Ocak 2014
321 Takip Edilen700 Takipçiler
When properly adjusting for task adaptation, model rankings even transfer across benchmarks.
This surfaces an important axis of model capability: being adaptable to downstream tasks of interest.
arxiv.org/abs/2507.05195
English
That being said,
🗣️model comparisons are scientifically uninformative unless we control for test task adaptation 🗣️
Absolute benchmark performance is nearly meaningless. Rate of progress on newly proposed benchmarks should be taken with a grain of salt.
arxiv.org/abs/2407.07890
English
Counterintuitively, the more attention the community places on a benchmark, the more we can trust who is at the top.
If everyone is trying their best (and has comparably hefty wallets), the only path to the top is genuine technical progress.
English
Model comparisons are confounded by benchmark-specific adaptation.
But‼️when frontier labs fiercely compete on the hottest benchmark of the year, and billions are on the line, everyone benchmaxes.
Skill and ingenuity are then the only differentiators.
English
Claude 3 Opus scored 4% on SWE-bench at release.
Shockingly, a Pythia-scale model trained **only on pre-1931 data**, with a bit of fine-tuning, outperforms the April 2024 SOTA.
Clearly, Opus is the better model. Why should we care about benchmarks, then? 👇🧵
Ricardo Olmedo@rdolmedo_
We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues. After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library. 🧵👇
English
noooooooo
stop code maxxing on my vintage LLM
Ricardo Olmedo@rdolmedo_
We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues. After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library. 🧵👇
English
@Zakarth Post-training is all about aligning models with downstream tasks. Here the question is, what capabilities does pre-training on the internet give you that are not recoverable with a little post-training?
English
@rdolmedo_ Ok but if you’re fine tuning it to do a specific task that defeats the point.
English
We fine-tuned Alec Radford’s 1930 vintage LLM to solve SWE-bench issues.
After just ‼️250‼️ training examples, the model solves its first issue, a simple patch to the xarray library.
🧵👇
English
@BlackHC However, the teacher model is Qwen Coder 3B active (the old one). With a stronger teacher, one should get even better results.
English
@BlackHC The seed dataset used for fine-tuning is SWE-smith, from the SWE-bench folks
swesmith.com
English
@wanon77789 Haha amazing, I haven't tried it with Claude Code, I would recommend the mini-swe-agent scaffolding since that's what it was fine-tuned on
English
@max_tretikov Both the 1930 and web models have the same pre-training FLOPs.
English
@rdolmedo_ Hold on, wait. Didn't microsoft try to do "textbooks are all you need" a few years ago and find that curation doesn't replace scale? How is this possible?
English
@willdepue I find it surprising that 250 training trajectories are sufficient to produce this level of multi-turn agentic reasoning, especially given that the model was not mid-trained on trillions of reasoning tokens.
English
@willdepue More impressive than the patch itself is that the model initially fails at applying the patch (turn 12), but runs tests, realizes its mistake, and finally fixes it (turn 44).
ricardodominguez.github.io/blogs/pydata__…
English
QAM
Curious about how pre-training data affects post-SFT performance?
Check out Section 4.1 of
arxiv.org/pdf/2407.07890
English