Sabitlenmiş Tweet
"Sharpness of the loss surface tells us about generalization... except when it doesn't (like in transformers)."
But what does sharpness mean? You might say
"it's how much the function changes within a little ball".
Ok, so in the simple picture shown below, at which of the two points is the loss surface sharper?
To find out
* why this is a tricky question, and
* why our answer to it does allow sharpness to tell us about generalization, even in transformers(!!),
come to our #ICML2025 #spotlight poster E-2001 on Wednesday at 11:00am-1:30pm PDT
Work by Marvin F. da Silva and Felix Dangel
@dalfcs @VectorInst
sageev@osageev
[1/🧵] ✨ Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It ✨ Super excited to announce our paper on factoring out parameter symmetries to better predict generalization in transformers (accepted as #ICML25 spotlight! 🎉) Amazing work by Marvin da Silva (@marvinfsilva) and Felix Dangel (@f_dangel). Symmetries hide sharpness — Riemannian geometry reveals it👇
English
