Tomasz Sternal
31 posts
[3] Memory-Efficient LLMs Training with Dynamic Sparsity: From Stability to Practical Scaling
English
[2] When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
English
Thanks to my wonderful collaborators, our three papers got accepted to #ICML2026 🎉
Huge thank you to the team and see you in Seoul 🇰🇷!
English
Tomasz Sternal retweetledi
I think it's incredible that, while obviously not as good as modern LLMs, this LLM is able to do in-context learning and write basic Python code. Really highlights the intelligence of LLMs.
David Duvenaud@DavidDuvenaud
Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below! with @AlecRad and @status_effects 🧵
English
Tomasz Sternal retweetledi
Process Reward Agents for Steering Knowledge-Intensive Reasoning
paper: huggingface.co/papers/2604.09…
English
It was a fantastic learning experience. Thank you so much to the amazing team for this collaboration! @de_Jiung @KStyppa @Michael_D_Moor @thoefler @ETH_en
English
Our new preprint is out!
We introduce Process Reward Agents (PRA) - a new framework in which the reasoning capabilities of a frozen reasoning model are decoupled from the Reward Agent, steering the reasoning process at test time.
Michael Moor@Michael_D_Moor
Preprint: arxiv.org/abs/2604.09482 Page: process-reward-agents.github.io Code: github.com/eth-medical-ai… Big thanks to a stellar team of co-authors @de_Jiung @TomaszSternal @KStyppa @thoefler! @ETH_en 1/
English
Tomasz Sternal retweetledi
AI is increasingly changing how we do mathematics.
Erdős Problem #650, open for over 60 years, was solved a few weeks ago through a collaboration between human mathematicians, an informal reasoning model (GPT 5.4 Pro @OpenAI) and a formal one (Aristotle @HarmonicMath). 🧵
English
@kingofknowwhere Love this take! Matches my experience - never stops surprising me when people with 5 ML papers at ICML/NeurIPS can’t explain what precision and recall are.
English
@KShevchenkoReal @KShevchenkoReal, can you please quote where you took the 90% estimate from? I thought this particular route through the Polish-Belarusian border is closer to 3% of the total freight movement.
English
Poland refuses to reopen its border with Belarus, cutting China off from a €25B/year trade artery. Some 90% of China-EU rail freight moves through Poland, now frozen amid Zapad-2025 drills and a Russian drone incursion. Beijing asked Warsaw to restore the route (vital for platforms like Temu & Shein) but after 3 hours of talks, FM Sikorski said no. With sea routes slower and air transport up to 30% more expensive, Europe’s e-commerce supply chains risk serious disruption. #ChinaEconomy #EUEconomy
Photo: ResearchGate
English
the solutions to your problems are hidden throughout history, you just need to find them.
English
Tomasz Sternal retweetledi
@iScienceLuvr @jxmnop played with SFT maybe has some experience with RL as well?
x.com/jxmnop/status/…
Jack Morris@jxmnop
OpenAI hasn’t open-sourced a base model since GPT-2 in 2019. they recently released GPT-OSS, which is reasoning-only... or is it? turns out that underneath the surface, there is still a strong base model. so we extracted it. introducing gpt-oss-20b-base 🧵
English
Has anyone successfully done RL post-training of GPT-oss with meaningful performance gains?
What libraries even support it? I guess technically TRL/axolotl, maybe Unsloth... but there are no good examples of doing it...
English
@Rainmaker1973 The picture is extremely misleading. Plastic bottles are not made of polyurethane but from polyethylene terephthalate. Pestalotiopsis microspora has nothing to do with plastic bottles.
English
Scientists found a fungus in the Amazon called Pestalotiopsis microspora that literally snacks on plastic.
Pestalotiopsis microspora is not your typical fungus. It can survive entirely on polyurethane, one of the most common (and most persistent) types of plastic — and it does so even in oxygen-free environments, like buried landfills.
[Khan, Sehroon, et al. “Biodegradation of Polyester Polyurethane by Aspergillus tubingensis.” Environmental Pollution, vol. 225, Mar. 2017, pp. 469–480]
English
Tomasz Sternal retweetledi
To all undergrads interested in learning about AI: be wary of taking “Intro to AI” as your first AI course. In many programs, the class you actually want first is “Intro to Machine Learning”.
AI technology has exploded in the past 15 years thanks to deep neural networks. Yet at many schools, the “Intro to AI” curriculum has barely changed from what it was in 2010, and spends often only a few lectures on machine learning. Unfortunately, revamping “Intro to AI” is controversial at many universities, and inertia tends to dominate.
Don’t decide which course to take based on the name alone. Instead, check the syllabus. Ideally, the course covers linear regression, gradient descent, backpropagation, and reinforcement learning. Each university is different and some “Intro to AI” courses will cover all these topics, but most don’t.
If you plan to pursue AI as a career, I think it makes sense to take “Intro to AI” later for a broader perspective on intelligence. But if your goal is an intro to the technology powering modern chatbots, image recognition/generation tools, and coding assistants, the class you probably want first is “Intro to Machine Learning”.
English
@Hesamation @ShunyuYao12 Sounds exciting, however, the main claim that "RL finally generalises" seems overly optimistic. This arxiv.org/pdf/2504.13837 sounds much more realistic: the RL won't get us beyond the reasoning patterns the base model has already learned from the training data.
English
this is one of the best blog posts of 2025 by the openai researcher @ShunyuYao12.
"we're at AI's halftime,"
it's a playbook of what will matter the most in AI research and the startup ecosystem, and how to prepare best for it.
for decades, AI research focused on algorithms and new models to beat the benchmarks.
but something important has changed the game: "RL finally generalizes."
the working “recipe”: massive language pretraining (priors) + scale + reasoning-as-action inside an RL loop.
the result of this benchmark climbing. the game shifts: from solving problems to defining the right problems. evaluation becomes center stage.
the core benchmark now is the "utility problem". benchmarks don't really translate well to real-world tasks.
so this is the second-half playbook: invent evaluation setups tied to real utility; then apply the recipe to win under those new rules.
in RL the key trio is environment, algorithms, and priors. we've spent so much time on the best algos but algos overfit to the environment they are born in.
for the “second half,” evaluation = environment design: build setups closer to reality (human-in-the-loop, non-IID, sequential/with memory) to drive real utility, not just benchmark wins.
English
Very cool post on the limitations of Sliding Window Attention! Together with the earlier post on Attention Sinks, it gives a great overview of both the strengths and weaknesses of SWA.
Guangxuan Xiao@Guangxuan_Xiao
It's a common belief that L SWA layers (size W) yield an L×W receptive field. My post shows why the effective range is limited to O(W), regardless of depth. The reasons are information dilution and the exponential barrier from residual connections: guangxuanx.com/blog/stacking-…
English
A statement is true if and only if denying it would force you to give up something else you already believe is true.
English