Antonio Montano ☼

Beyond De-Skilling: Intelligence Explosion and the End of Skill as a Stable Category 4m4.it/posts/beyond-d…

“Deep learning is alchemy” may be the most repeated criticism in AI. It also misses the mark. Alchemy failed to deliver results. Deep learning, by contrast, has produced transformative technologies. And fields like medicine are only partially understood without being deemed alchemical. So calling AI “alchemy” captures part of the problem, but not all of it. Modern AI is not simply undisciplined experimentation. It contains significant amounts of rigor. But we still struggle to answer basic questions: • Do models understand? • Why do they generalize? • When will they fail? The deeper issue is that rigor takes different forms—and in AI, those forms are unevenly developed. My new paper distinguishes three: • Conceptual rigor: coherent terminology and paradigms • Epistemic rigor: reliable scientific understanding • Operational rigor: reliable performance and deployment This framework helps explain both the extraordinary progress of modern AI and the uncertainty surrounding it. Conceptual rigor asks whether the field knows what it's talking about. • What exactly is intelligence? • What qualifies as AGI? • What does it mean for a system to be aligned? Consider the debate over whether current models are intelligent. One person points to their breadth of performance. Another points to weak planning. Another emphasizes sample inefficiency. Another asks whether it has a grounded model of the world. They appear to disagree about one property. Often, they are evaluating four. This is why conceptual clarity matters in practice. Questions about intelligence, understanding, AGI, and alignment do not remain confined to philosophy: they shape how things are measured, optimized, and built. Epistemic rigor asks whether empirical success has become scientific understanding. The paper focuses on three criteria: • Can findings be reproduced? • Can behavior be predicted in advance? • Can success and failure be explained? AI experiments are unusually reproducible in principle: code, data, and models can be copied. But conclusions may still depend heavily on random seeds, hyperparameters, implementation choices, benchmark selection, and compute budgets. Reproducing a number is not always the same as reproducing the conclusion drawn from it. Prediction is harder. Scaling laws can forecast some training outcomes. Infinite-width theory can lead to more tractable settings. Classical learning theory explains important pieces. But we still lack broad principles telling us when a model will generalize, fail under distribution shift, or remain robust under adversarial perturbations. Explanation is harder still. Neural networks are mathematically specified, yet their learned features resist human interpretation. A behavior may arise from training data, optimization dynamics, internal representations, or interactions among all of them. The system is transparent in code but opaque in meaning. Operational rigor is where modern AI is strongest: benchmarks, evaluations, monitoring, red-teaming, and deployment controls. The field has become highly effective at improving systems without first obtaining a scientific theory of them. Benchmarks turn capabilities into measurable targets. Post-training shapes behavior. Tools and scaffolding compensate for model weaknesses. Operational rigor can therefore partially substitute for scientific understanding. That imbalance defines the deep-learning era: • Capabilities rise rapidly. • Explanations lag behind. • Benchmarks become optimization targets. • New systems generate new phenomena faster than theory can absorb them. AI is advancing while continually changing the object that science must explain. For AI to mature as both a science and a technology, it will require all three forms of rigor: • Clearer concepts to define our goals. • Stronger science to predict and explain system behavior. • Better engineering to make systems genuinely reliable. The future of AI depends not simply on demanding “more rigor,” but on identifying which kind is missing—and understanding how the imbalance shapes what we can build, know, and control.

[LG] MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers R Garcia, J Liu, R Junkins, S Eyuboglu… [Stanford University] (2026) arxiv.org/abs/2607.10034

[LG] Domain-Aware Scaling Laws Uncover Data Synergy K Hamidieh, L Mackey, D Alvarez-Melis [MIT & Microsoft Research] (2026) arxiv.org/abs/2607.11052

'Approximations and Learning for Continuous State and Action MDPs under Average Cost Criteria', by Ali D. Kara, Serdar Yüksel. jmlr.org/papers/v27/24-… #quantization #quantized #discretization

[LG] Reference-Based Distillation Detection in LLMs R Rawat, S Chen, A Anand, M Duan… [UC Berkeley] (2026) arxiv.org/abs/2607.09692

Several articles & resources I’ve been reading lately to learn about how LLM architectures have evolved: * finbarr.ca/five-years-of-… 5 years of GPT-class model progress, only goes to 2023 but that puts more focus on early models where details are precisely known * magazine.sebastianraschka.com/p/from-gpt-2-t… great companion post to the first one as its diagrams and code examples are absurdly helpful. @rasbt has so many articles that could be on this list but I picked this one because of the visuals in particular * gregorygundersen.com/blog/2025/10/0… denser than other posts. my attention can glaze over some of the deeper math formulas but as a synthesis of key LLM concepts have evolved this is excellent * cameronrwolfe.substack.com/p/the-history-… both accessible in the basics "what is a token?" and detailed, this one goes into other notable models from early LLM days (post image is from this article)

The first experimental evidence of recursive self-improvement (RSI). Autoresearching the autoresearch agent for eight days. The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)

An exact information theory of generalization phase transitions in Bayesian diffusion models Henry Hunt, Mason Kamb, Surya Ganguli arxiv.org/abs/2607.08041 [𝚌𝚜.𝙻𝙶 𝚌𝚘𝚗𝚍-𝚖𝚊𝚝.𝚍𝚒𝚜-𝚗𝚗]

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories by @behrouz_ali @Farn8sh_h @AdelJavanmard @mirrokni — a step[nap] towards RSI. k-a.in/LLM-sleep.html

AI Adoption, Productivity, and the Missing Middle – Random Bits of Knowledge 4m4.it/longforms/ai-a…

📜 Full Bayesian Reinforcement Learning via LF-IBIS by Stefano Masini, Cecilia Viscardi, Michela Baccini 🔗 arxiv.org/abs/2607.01741

[LG] How are linear representations learned? Exact solutions to the dynamics of abstraction W W. Yang, A M. Saxe, P E. Latham [University College London] (2026) arxiv.org/abs/2607.08843

[LG] Prompt-Driven Exploration S Jiang, J Marangola, D Zhang, R Kowdeed… [MIT] (2026) arxiv.org/abs/2607.08837

[LG] Learning More from Less: Reinforcement Learning from Hindsight I Xu, S Jiang, J Marangola, N Dashora… [MIT & Stanford University] (2026) arxiv.org/abs/2607.09042

No Selves, No Consciousness ojs.aaai.org/index.php/AAAI… ✍️ @MiTiBennett via @RealAAAI 👉 The paper argues that true consciousness requires a multi-layered, hierarchical structure of "selves" developed through adaptive learning 💡 It posits that current large language models lack this necessary organizational architecture, distinguishing mere functional simulation from authentic conscious experience @Corix_JC @SonuMonika @TheAIObserverX @maponi @JagersbergKnut @ahier @CEO_AISOMA @theomitsa @dinisguarda @TarakRindani @FrRonconiii @Nicochan33 @SusanHayes_ @JoannMoretti @FernandaKellner @pchamard @PVynckier @MaryRich78 @mikeflache @amalmerzouk @NathaliaLeHen @Eli_Krumova @chidambara09 @RLDI_Lamy @WillyRayNick @DanielleLargier @sminaev2015

📜 A law of robustness for two-layer neural networks with arbitrary weights by Yitzchak Shmalo 🔗 arxiv.org/abs/2607.07778

Depth-recurrent models share parameters across depth - promising for latent reasoning in continuous space - but prior work lacked compute-matched baselines and ignored a real bottleneck: the hidden state is too small to carry many steps of latent reasoning. Dreamer (Aleph Alpha + TU Munich) unifies three attention directions in one recurrent layer: standard self-attention over the sequence, depth attention over the same token's previous depths, and expert attention - MoE reformulated as attention. Depth attention widens the effective hidden state; expert attention widens the effective layer. Everything is compute-, parameter-, and memory-matched against strong MoE baselines. Results at roughly 1B and 2B scale: Dreamer needs 2-8x fewer training tokens for the same accuracy, and the depth-16 recurrent model beats the depth-32 layered baseline across all reasoning benchmarks (GSM8K, MATH, MMLU math and more) - about half the parameters, FLOPs, and memory. The analysis is the fun part: experts specialize by depth but half of them get reused across many depths, and deeper layers learn to look back at what earlier depths computed. Made a short visual breakdown - one diagram per trick. Swipe through. 👇 --- paper - arxiv.org/abs/2601.21582 full summary pdf - gist.github.com/vukrosic/014a8… 🔬 Every Sunday I run a hands-on live AI research with 1 on 1 help: skool.com/become-ai-rese…

Brilliant paper "Mathematical capacity, which is the trained ability to verify, interpret, and challenge mathematical reasoning, is not a byproduct of theorem production but a form of infrastructure, built over generations by institutions that cannot be reconstituted on demand." Jevon's paradox for mathematicians is real we need to be doubling down on funding for math arxiv.org/abs/2607.06377

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong arxiv.org/abs/2607.07508 [𝚌𝚜.𝙻𝙶 𝚌𝚜.𝙰𝙸]