Yanick Schimpf

Autoregressive sequence models like transformers develop internal representations of semantically meaningful abstract actions! Enters Internal RL: An AI inside an AI that solves hard problems by manipulating these abstractions to steer the outer model.

New paper 🧵 We show that dynamic short convolutions consistently improve Transformers across scales. We make these gains practical with an efficient parameterization and custom Triton GPU kernels. The improvements carry over to MoEs and linear attention variants (Mamba-2/GDN).

Today and tomorrow we’ll be presenting self-distillation with orals at ICLR in Rio 🇧🇷 1. “Self-Distillation enables Continual Learning” at lifelong agents workshop (Sun 11:30am) 2. “Reinforcement Learning via Self-Distillation” at scaling post-training workshop (Mon 2:40pm) 3. “Test-Time Self-Distillation” at test-time updates workshop (Mon 4:15pm)

We cooked up “Internal RL”. The new RL algo exploits a specific insight we got from analysing pre-trained Transformers and achieves success in tasks where all baselines (like GRPO etc.) FAIL! Wanna learn more? Search for @yaschimpf and @ninoscherrer at the RSI workshop at ICLR!

Catch @yaschimpf and me tmrw at the “Recursive Self-Improvement” workshop at ICLR, Room 101D!! We are presenting our recent work on “latent control” in Transformers that enables hierarchical RL (arxiv.org/abs/2512.20605) 🔥

Internal RL goes 🇧🇷🇧🇷🇧🇷. Catch me at the Recursive Self Improvement workshop @iclr_conf!!! @ninoscherrer and me will present our work and stick around the entire day. Poster sessions: 10:00 - 10:30 & 12:00 - 12:30 @ Room 101 - D Let’s close the loops🚀.

Looking forward to seeing you at the Google Booth @iclr_conf in ~30min for the MesaNet talk

@ysu_ChatData @GoogleResearch Indeed it pushes this view to the Limit by finding the optimal fixed memory state at every time step!

Google presents a new Transformer alternative at #ICLR2026! Join Nino Scherrer & Yanick Schimpf at the Google booth (#411) at 10AM to learn about MesaNet, proposing a new linear sequence layer that optimally learns in-context given a fixed memory budget.

@AIONPHI @GoogleResearch In this work we scale it up to 1B models. At that scale it is competitive (ppl) with softmax self-attention + performs well on global reasoning benchmarks and retrieval!

New strong linear arch MesaNet by @ninoscherrer @yaschimpf outperforms all linear baselines, impressive work

@1a1n1d1y Let‘s change it haha

this guy is presenting MesaNet today and has 55 followers i think mesanet will be a big deal the hardest hitters on twitter are all sitting under 100 followers right now

@GoogleResearch We will also present the MesaNet a second time later today: Poster: 3.15pm to 5.45pm @ Pavilion 3, #604 🇧🇷

MesaNet goes 🇧🇷. @ninoscherrer and me will present tomorrow @iclr_conf.

it's time to drop three new #opensource robotic hands! this time with tactile sensors! Tweak it, 3D print it, and use them in your robotics and physical AI research! Here are some wild examples ↓↓↓

If you want to understand in less than 15 minutes why sparse attention emerges in Transformers, check out my NeurIPS 2025 oral! nips.cc/virtual/2025/l…

Temporal abstraction is a key missing ingredient for long-term RL (to avoid the curse of the one-step model, as @RichardSSutton likes to say). Language is one approach to such abstraction, but animals show it can be learned from data without human labels. But how? It is hard to learn learn temporally abstract options using standard hierarchical RL methods. This paper, from the Google team led by @neurosacramento, takes an approach inspired by LLM pre and post training. Let me explain. First it pre-trains a sequence model on unlabeled expert trajectories, generated by goal-seeking agents. The model learns to predict the next action (and optionally next observation) given previous observations, without knowing the goal or the sequence boundaries. The model is a standard transformer, with L blocks, shown in Fig 1. Here e(t,l) is the residual stream at time t and layer l. These learn the relevant temporal abstractions. Next it modifies the model by inserting a control signal, U(t), at layer l, to get e'(t,l)=e(t,l) + U(t) e(t,l). This generates the desired action a'(t), as shown in Fig 2. The control signal is generated from a latent variable, u(t) = f(z(t)). This model is trained (using amoritzed variational inference) to maximize the likelihood of another set of expert trajectories, using a non-causal encoder, z(t)=enc(o(1:T)). Finally, it replaces the offline variational controller with a causal controller, that is trained using online RL (with the rest of the model being frozen). At each step, the SSM-based controller generates a stochastic latent z'(t). In addition, a GRU-based switching module generates a switch probability, beta(t). Finally the latent control z(t) is either set to the new latent action z'(t), or it remains equal to the previous latent action, z(t-1), as determined by a (thresholded) beta(t). See fig 3. This generates long chunks of constant action signals, corresponding to temporally abstract options. The result is a method that can learn to control a low-level mujoco ant (given full observability) to solve long horizon tasks, visiting combinatorially new sequences of goal locations. The authors also claim this model is similar to the how the brain solves such problems - the transformer model is like the cortex (designed to predict what will happen next in the environment given that the agent is following some goal), and the controller is like the basal ganglia / striatum, designed to choose the next latent internal goal to aim for.

I like and bookmark so many interesting sounding papers here, and don’t get back to most of them. Time to start making a dent. I’m going to try to at least skim one of the papers in my bookmarks each weekday for the rest of the month. #PaperADay 2025: Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning (Google) I like their statement of the hierarchical goal problem  as “how long does it take a twitching hand to win a game of chess?”  @RichardSSutton is fond of the “options” framework in RL, but we don’t have a clear method to learn them from scratch. Their Ant environment is designed to require two levels of planning: the standard mujoco Ant locomotion work to be able to move at all, and routing decisions to get to the colored squares in the correct order, which will happen hundreds of frames apart. Basically, this takes a pre-trained sequence predicting model that predicts what separately trained expert models (manually steered) do, and inserts a metacontroller midway through it, which can tweak the residual values to perform high level “steering”, and can be RL’d at high level switch points to much greater performance than the base pre-trained model. A key claim here is that learning to predict actions in a supervised next-token manner from lots of existing expert examples, even if you don’t know the goals, results in inferring useful higher level goals. This sounds plausible, but their experiment makes it rather easy for the model: the expert RL models that generated the training data were explicitly given one of four goals in each segment, and the option learning model just classifies the sequences into one of four categories. This is a vastly simpler problem than free form option discovery. A State Space Model is used for the more complex Ant environments, while a transformer is used for the simpler grid world environments. I didn’t see an explanation for the change. The internal “walls” are more like “poison tiles”, since they don’t block movement like the map edges, they just kill the ant when its center passes into them. The 3D renderings (with shadow errors that hurt my gamedev eyes) are somewhat misleading, since it is really a 2D world that the agent gets to fully observe in a low dimensional one-hot format. It doesn’t do any kind of partially observed or pixel based sensing. Everything is done with massively parallel environments, avoiding the harder online learning challenges. The success rates still aren’t great after a million episodes. I would like to see this applied to Atari, basically doing GATO with less capable experts or lower episode quantities, then trying to identify free form options that can be usefully used to RL to higher performance.

Nice summary by @ninoscherrer on Internal RL: 𝗟𝗲𝘁’𝘀 𝗲𝘅𝗽𝗹𝗼𝗿𝗲 “𝗮𝗰𝘁𝗶𝗼𝗻 𝗯𝘆 𝗮𝗰𝘁𝗶𝗼𝗻” 𝗶𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 “𝘁𝗼𝗸𝗲𝗻 𝗯𝘆 𝘁𝗼𝗸𝗲𝗻”

Google discovers emergent temporal abstractions in autoregressive models These models learn linearly controllable action representations in their residual streams—activating them executes long-horizon behaviors. This enables Internal RL to solve sparse-reward hierarchical tasks intractable for standard token-level approaches like GRPO.