Fabian Schaipp

The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting! 1/2

Adding caution to the update step also doesn't help. makes it worse actually. does anyone actually use cautious update or cautious weight decay in practice ?

From our analysis, the achievable error satisfies: ε ∼ max { BS / T, T^(−1/3), 1 / (T² BS)^(1/6) } These terms correspond to limiting terms due to 1) iterations 2) optimization efficiency 3) stochastic noise Balancing them gives the scaling law.

Super excited that TabICLv2 is out 🎉 🚀Beats RealTabPFN-2.5 with no tuning and purely synthetic pre-training data. 👉Introduces QASSMax for long-context generalization, early target embedding, repeated feature grouping, Muon, etc., and a much diversified synthetic data prior.

Learning rate matters more than your LoRA variant. In this study they sweep LR hard across LoRA variants (DoRA, Init[AB], PiSSA, MiLoRA) and find: > If you tune LR properly, they all converge to approx the same peak perf. > Rank still matters and can flip which variant looks best depending on dataset. > Optimal learning rate is a function of how steeply curved the loss is: >> more curved → smaller steps (lower LR) >> less curve → larger steps (higher LR)

To keep training stable at high sparsity, we increased dense layers, applied momentum-based expert load balancing, and used z-loss to control logit scale during training. The loss curve stayed smooth throughout the run.