Fabian Schaipp

The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting! 1/2

Super excited that TabICLv2 is out 🎉 🚀Beats RealTabPFN-2.5 with no tuning and purely synthetic pre-training data. 👉Introduces QASSMax for long-context generalization, early target embedding, repeated feature grouping, Muon, etc., and a much diversified synthetic data prior.

Learning rate matters more than your LoRA variant. In this study they sweep LR hard across LoRA variants (DoRA, Init[AB], PiSSA, MiLoRA) and find: > If you tune LR properly, they all converge to approx the same peak perf. > Rank still matters and can flip which variant looks best depending on dataset. > Optimal learning rate is a function of how steeply curved the loss is: >> more curved → smaller steps (lower LR) >> less curve → larger steps (higher LR)

To keep training stable at high sparsity, we increased dense layers, applied momentum-based expert load balancing, and used z-loss to control logit scale during training. The loss curve stayed smooth throughout the run.

it's christmas again?? lfg

korea is far from frontier, yes. but we’re not done yet. let’s see what happen, lmao.