Alexander Theus @ ICML

Our Workshop on Weight-Space Symmetries @icmlconf @weightsymmetry is starting now! You can reach Room 403 by going up from Hall A. Program: weightsymmetry.com/schedule

👇 Check out our new preprint: FPRM (Fixed-Point Reasoning Model). • No ACT head: loop to a fixed point, with convergence as the halt signal. • No hierarchy: a single loop beats HRM/TRM variants! • SOTA among 7M models: Sudoku-Extreme 94.2%, Maze-Hard 87.0%, ARC-AGI-1 47.5%.

This TRM variant makes a transformer block a contractive map, so that looping becomes a fixed-point process. They leverage this by approximating gradients with Neumann series (Truncated BPTT). Very cool work! 🔗arxiv.org/abs/2606.18206

#ICLR2026 Into mode connectivity, model merging, or permutation invariance? We show how optimization dynamics shape the loss landscape of merged weights. Come check it out! 📅 23/04 10:30AM – 13:00PM 📍 Pavilion 3 P3-1809 w/ @TheusResearch @DamienTeney @orvieto_antonio

📢 Submissions are OPEN for the Weight Space Symmetry Workshop @icmlconf! ⏰ Deadline extended → April 30 (23:59 AOE) Consider submitting any work related to weight symmetries: optimization, model merging, weight space learning, and so on! #ICML2026 #weightsymmetry2026

📢Excited to announce the Workshop on Weight-Space Symmetries @icmlconf! We welcome 4-page submissions analysing symmetries, their effects on training and model structure, and practical methods to utilize them. Submission Deadline: April 24 (23:59 AoE) #ICML2026

Excited to announce that our paper has been accepted as an Oral at NeurIPS 2025! 🥳

10/ 📄 Paper: arxiv.org/abs/2506.22712 By: @Theus__A , Alessandro Cabodi, @SAnagnostidis , @orvieto_antonio , @unregularized , and @val_boeva 🙏 Huge thanks to my amazing co-authors for this collaboration! #Transformers #LMC #MachineLearning #DeepLearning

9/ 🔑 Takeaway: Transformers can be linearly connected — but only if you exploit richer network symmetries. We show that general symmetry alignment (not just permutations) unlocks low-loss paths across ViTs and GPT-2.

1/ 🚨 New paper alert! 🚨 We explore a key question in deep learning: Can independently trained Transformers be linearly connected in weight space — without a loss barrier? Yes — if you uncover their rich symmetries. 📄 arXiv: arxiv.org/abs/2506.22712