Kempner Institute at Harvard University

What is the #KempnerInstitute? Watch a short video feature to dive deeper into our mission and meet the researchers behind it: bit.ly/49ODO2K #AI #ML #NeuroAI #neuroscience @Harvard

Authors: Tessa Han (@tessa1157), Sebastian Bordt (@sbordt), Hanlin Zhang (@_hanlin_zhang_), and Sham Kakade (@ShamKakade6).

NEW from the #KempnerInstitute: Kempner researchers find that weight decay during pretraining improves a model's ability to continue to learn. Read the #DeeperLearning blog post: bit.ly/4eYwkQH

If you're at #ICML2026, don't miss the workshop on AdaptFM: Resource-Adaptive Foundation Model Inference! A team that includes the #KempnerInstitute's @JonathanGeuter and @elmelis will be presenting their work! Saturday 8am-5pm KST! icml.cc/virtual/2026/7… #AI

Also tagging: @maryiletey, @CPehlevan

Day 4 of #ICML2026 brings a full slate of #KempnerInstitute workshop presentations, with work on learning dynamics, mechanistic interpretability, weight-space symmetries, and more. Check out Friday's lineup! #AI #NeuroAI

Next talk tomo #ICML2026: How far does Taylor's Thm take us in deep learning? @alex_damian_ and I take the local quadratic approx seriously, asking how well it actually holds during LLM pretraining. Surprisingly, on 150M models, it tracks the true loss deep into training ↓

🎤 Our #ICML2026 Oral is happening TODAY!! Why do diffusion models trained on non-overlapping data draw near-identical images from the same seed? 👇 It's largely a linear effect — the shared Gaussian statistics of the data — and we make it precise with random matrix theory. Come see Jacob ZV present it, or dive into the full thread below!👇 🎤 Oral: Today 4:00 PM KST · Ballroom 201–203 📌 Poster: Today 5:00 PM KST · Hall A

How does neural feature geometry evolve during training? Modeling feature spaces as geometric graphs, we show that nonlinear activations drive transformations resembling discrete Ricci flow - revealing how class structure emerges and suggesting geometry-informed training principles. Led by Moritz Hehl. Details here: arxiv.org/abs/2509.22362

A poster session you won't want to miss this afternoon! And stay tuned for some exciting new dLLM results that we have in the works :)

Presenting our work at the #ICML26 Oral session: Prescriptive Scaling Reveals the Evolution of Language Model Capabilities We study a practical question for language models: given a pre-training compute budget, what downstream performance can we reliably expect after post-training? 📍 #ICML Oral Hall C 🕧 Thu, Jul 9, 4:30 PM–4:45 PM Local Time

Excited to be a part of this amazing community!

Day 3 of #ICML2026 features #KempnerInstitute research on neural feature geometry, protein language models, scientific data annotation, time series reasoning and more. See Thursday's presentations! #AI

We are thrilled to announce the appointments of two new associate faculty members: Nada Amin and Sitan Chen — pioneering scientists who joined the #KempnerInstitute on July 1! Read more: bit.ly/4vmnQHZ @sitanch @hseas #AI #ML

Transformers are sequence models, but much of in-context learning theory considers prompts of independent examples / tokens. We asked: what changes when ICL prompts are actually sequential? I’ll be presenting this at #ICML2026 HiLD Workshop!

Wow~ Thrilled that our paper "A Random Matrix Theory Perspective on the Consistency of Diffusion Models" received an Outstanding Paper Honorable Mention at #ICML2026! 🏆 The mystery: Diffusion models trained on disjoint data splits create near-identical images from the same seed. The surprise: this is largely a linear effect -- the shared Gaussian statistics of the data already predict much of the output. 🧵👇 With wonderful friends and collaborators Jacob Zavatone-Veth & @CPehlevan. Grateful to @KempnerInst ! 💙

It was a pleasure to speak at #icml2026. Thank you!

Learn more about the broad science-of-pretraining effort that #KempnerInstitute Co-Director @ShamKakade6 discussed in yesterday's #ICML2026 invited talk: bit.ly/4eNVmSp @alexmeterez @CPehlevan @alex_damian_

#KempnerInstitute Investigator @du_yilun led a team that developed a new robotic control system allowing legged robots to adapt movements to new tasks w/o retraining... learn more in this article from @hseas! 👇

A few more backstage notes 🎬 ▸ The project began last summer, when I was learning RMT from @CPehlevan & Jacob — originally to understand ridge regression for predicting visual neurons. Then I realized ridge regression and the diffusion denoiser share so much structure that many results port over directly. Chasing multiple problems at once can reveal hidden connections! ▸ I finished the main RMT calculation the night before July 4th last year — the result was so clean I got too excited to sleep, and ended up watching the fireworks completely sleep-deprived 😂 (minor thanks to the Boston heat wave). ▸ This paper got a borderline score at ICLR'26, then a desk reject over a post-rebuttal info leak 😅 The current version only added minor evidence & cleanups — same result. ▸ Its intellectual foundation — the hidden linear structure (w/ @johnjvastola) — was rejected twice from main conferences since 2023. We got too frustrated and sent it to TMLR, where it finally found a home. Same ideas, once hard to publish, now getting an award. So don't give up!!💪

Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)