Joe Davison

Excited to launch the accompanying free RLHF Course for my book. To kick it off, I've released: - Welcome video - Lecture 1: Overview of RLHF & Post-training - Lecture 2: IFT, Reward Models, Rejection Sampling - Lecture 3: RL Math - Lecture 4: RL Implementation I'm going to add question & answer videos throughout the lecture to go deeper on topics that need it, and potentially cover some topics that are too recent and in flux to go in print. I expect 10-15 videos in total over the next few months. At the same time, development around the code for the book is picking up. It's a great time to build the foundation for post-training methods. YT playlist and course landing page below.

Introducing Kernels on the Hugging Face Hub ✨ What if shipping a GPU kernel was as easy as pushing a model? - Pre-compiled for your exact GPU, PyTorch & OS - Multiple kernel versions coexist in one process - torch.compile compatible - 1.7x–2.5x speedups over PyTorch baselines

We introduce a new approach for fast and high-quality context compaction in latent space. Attention Matching (AM) achieves 50× compaction in seconds with little performance loss, substantially outperforming summarization and other baselines.

Really excited for the things @rosstaylor90 has been building to go out into the world. He's been one of the people I can always rely on to have non-cope takes on what we need to do to make the open ecosystem great. What a great time in RL.

Opus 4.6 is hugely better at Pokemon: • Opus 4.0 took 1,000 hours to get half way through • Opus 4.5 could almost finish in 1,000 hours • Opus 4.6 was another 10x faster!

Introducing PerturbAI. Today we announced our emergence from stealth with the release of the world’s largest in vivo CRISPR data engine, interrogating the effects of thousands of genetic perturbations across 8 million cells throughout the whole brain. This dataset represents a new category of biological data: organism-level, circuit-resolved causal genomics leading to novel targets and therapeutics. By combining scalable in vivo CRISPR perturbation with AI, we model biological systems at unprecedented resolution and simulate therapeutic interventions before committing to expensive downstream development. We’re grateful to our collaborators at @NVIDIAHealth and @10xGenomics for helping make this landmark dataset possible. Read More: perturb.ai/news #CRISPR #AI #DrugDiscovery #FunctionalGenomics #Biotech

Elegant idea. Instead of adding up layer residuals, use attention in depth This makes me more bullish that depth- and width-recurrence are not so fundamentally different. Someone working on depth-recurrent “loop” LMs 👇 should try recurrent attention arxiv.org/abs/2510.25741…

Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…

I (finally) put together a new LLM Architecture Gallery that collects the architecture figures all in one place! sebastianraschka.com/llm-architectu…

We’ve come full circle: there’s now a special term for thinking only in weights, without CoT

wait why Opus got a blog but not Golden Gate Claude

@natolambert What’s your read on that Sonnet 4.5->4.6 leap? Genuine long context capability leap, narrow benchmark-specific phase change, or something else?

I expect a lot more remarkable plots like this showing how fast the frontier models are progessing. Benchmarks aren't always everything, but hillclimbing is happening so fast.

World will converge on 3 types of models 1. Closed frontier (Ant, OAI, Gemini) 2. Open frontier (2-3 labs, much consolidation coming) 3. Open small / tool (fairly empty now) The open frontier will be far from the closed frontier, but way cheaper. Other statements are cope.

The Gemini Embedding 2 baseline here is.. 2 days old. Was just being celebrated and is now outperformed by a median of 14% and up to 91 points. If I didn't kind of know how powerful scaling ColBERTs and ColPalis can be compared to a single-vector model, I'd be in disbelief!

@ClementDelangue @abidlabs I missed the original tweet—huge congrats!

I’m back. The girls and their superhero of a mom are doing great 😍😍😍 What did I miss?

was skeptical but gave it a shot because @karpathy anyways 2x kernel perf (fp4 matmul) 3 minutes of work (1 prompt) triton beat cutlass (?!)

Scoop from me: Nvidia will spend a total of $26 billion over the next five years building the world's best open source models. America is back in the open source AI race! wired.com/story/nvidia-i…

time to look at your data, anon