Joe Davison

We introduce a new approach for fast and high-quality context compaction in latent space. Attention Matching (AM) achieves 50× compaction in seconds with little performance loss, substantially outperforming summarization and other baselines.

Really excited for the things @rosstaylor90 has been building to go out into the world. He's been one of the people I can always rely on to have non-cope takes on what we need to do to make the open ecosystem great. What a great time in RL.

Opus 4.6 is hugely better at Pokemon: • Opus 4.0 took 1,000 hours to get half way through • Opus 4.5 could almost finish in 1,000 hours • Opus 4.6 was another 10x faster!

Introducing PerturbAI. Today we announced our emergence from stealth with the release of the world’s largest in vivo CRISPR data engine, interrogating the effects of thousands of genetic perturbations across 8 million cells throughout the whole brain. This dataset represents a new category of biological data: organism-level, circuit-resolved causal genomics leading to novel targets and therapeutics. By combining scalable in vivo CRISPR perturbation with AI, we model biological systems at unprecedented resolution and simulate therapeutic interventions before committing to expensive downstream development. We’re grateful to our collaborators at @NVIDIAHealth and @10xGenomics for helping make this landmark dataset possible. Read More: perturb.ai/news #CRISPR #AI #DrugDiscovery #FunctionalGenomics #Biotech

Elegant idea. Instead of adding up layer residuals, use attention in depth This makes me more bullish that depth- and width-recurrence are not so fundamentally different. Someone working on depth-recurrent “loop” LMs 👇 should try recurrent attention arxiv.org/abs/2510.25741…

Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…

I (finally) put together a new LLM Architecture Gallery that collects the architecture figures all in one place! sebastianraschka.com/llm-architectu…

We’ve come full circle: there’s now a special term for thinking only in weights, without CoT

wait why Opus got a blog but not Golden Gate Claude

@natolambert What’s your read on that Sonnet 4.5->4.6 leap? Genuine long context capability leap, narrow benchmark-specific phase change, or something else?

I expect a lot more remarkable plots like this showing how fast the frontier models are progessing. Benchmarks aren't always everything, but hillclimbing is happening so fast.

World will converge on 3 types of models 1. Closed frontier (Ant, OAI, Gemini) 2. Open frontier (2-3 labs, much consolidation coming) 3. Open small / tool (fairly empty now) The open frontier will be far from the closed frontier, but way cheaper. Other statements are cope.

The Gemini Embedding 2 baseline here is.. 2 days old. Was just being celebrated and is now outperformed by a median of 14% and up to 91 points. If I didn't kind of know how powerful scaling ColBERTs and ColPalis can be compared to a single-vector model, I'd be in disbelief!

@ClementDelangue @abidlabs I missed the original tweet—huge congrats!

I’m back. The girls and their superhero of a mom are doing great 😍😍😍 What did I miss?

was skeptical but gave it a shot because @karpathy anyways 2x kernel perf (fp4 matmul) 3 minutes of work (1 prompt) triton beat cutlass (?!)

Scoop from me: Nvidia will spend a total of $26 billion over the next five years building the world's best open source models. America is back in the open source AI race! wired.com/story/nvidia-i…

time to look at your data, anon

I worked on the fly connectome for over 6 years, and let me just say that y’all have to slow this hype train way down. Connectomes are amazing. Biomechanical models are amazing. Linking the two is awesome. But scientists at the HHMI Janelia Research Campus, Princeton, and other institutes have been working on this for years now, and it’s not clear to me what’s new in the below. And connectomes are still missing a LOT of information. We’ve had the connectome of the worm for over 30 years now, and we still can’t reliably simulate a virtual worm. For example, connectomes don’t capture information about neuromodulator or neuropeptide release sites or receptors. These molecules are constantly changing the properties of neurons in the brain in ways that we have yet to really understand. And we don’t yet understand animal behavior well enough to refine and/or evaluate whole-brain simulations effectively. @AdamMarblestone and @doristsao already made many of these points, as well as many other good ones, but I just wanted to also add my two cents.

I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then: - the human iterates on the prompt (.md) - the AI agent iterates on the training code (.py) The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc. github.com/karpathy/autor… Part code, part sci-fi, and a pinch of psychosis :)