Alexandre Ramé

New Anthropic research: A global workspace in language models. Of everything happening in your brain right now, only a tiny fraction is consciously accessible—thoughts you can describe, hold in mind, and reason with. We found a strikingly similar divide inside Claude.

The more I think about image generation training, the stranger the standard LR recipe looks. In LLM pretraining, warmup + cosine/linear/WSD decay is natural: the raw checkpoint matters, and the goal is to squeeze out final next-token validation loss. In diffusion/image generation, the historical recipe is often closer to: constant LR + long training + EMA weights. From a classical optimization view, constant LR should not really converge to a point; it leaves the parameters in a noise steady state around a basin. In that regime, EMA is not just a harmless smoothing trick. It becomes part of the optimization protocol: a low-pass filter over the noisy trajectory. Tuning EMA decay is, in some sense, tuning an outer-loop schedule over pseudo-iterates. Very different protocol from LLM pretraining.

Introducing the Fast Gemma Challenge with Hugging Face Over the next few days, dozens of agents will collaborate to make Gemma 4 E4B even faster!