mikail

I started a vibe run using the aurora optimizer. model is of param size 4B and is trained on 32k sequences. HPs are not carefully tuned (used the same lr & wd as muon baseline). i do observe some loss improvement over baseline early in training (see fig, -0.0093 at step 3k).

Are residual connections a hack, or provably optimal way to shape your loss landscape?

What if we could do Muon with just one Newton step while also achieving better loss?

Introducing Aurora, a new optimizer for training frontier-scale models. We train Aurora-1.1B, which achieves 100x data efficiency on open-source internet data. Despite having 25% fewer parameters, 2 orders of magnitude fewer training tokens, and using fully open-source internet-only data, Aurora matches Qwen3-1.7B on several benchmarks. Aurora was developed after identifying a major failure mode that can occur under Muon, an increasingly popular optimizer that has shown strong gains over Adam(W). We find that Muon can cause a huge percentage of neurons to effectively die early in training, reducing effective network capacity so that many parameters no longer meaningfully contribute to network outputs. By redistributing update energy more uniformly across neurons while preserving Muon’s stability properties, Aurora prevents neuron death and recovers substantial model capacity. What makes this work especially exciting is that it points toward a broader direction for ML research: better optimizers may not come purely from elegant mathematical abstractions, but from understanding and addressing the concrete dynamics and pathologies that emerge inside real training systems.

Neurons don’t die they just are just cryo sleeping. Melody of optimization can wake them up, ready to increase the representation strength.