spacy

Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition. We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it. We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model. For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both. Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions. The full blogpost, experiments, and discussion are in the thread below.

Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition. We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it. We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model. For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both. Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions. The full blogpost, experiments, and discussion are in the thread below.

Our results suggest that similar to tuning model architectures and training recipes, well-designed harnesses can reduce the cost of curating more data and generating longer rollouts while increasing the coverage of tasks solvable through post-training. One might be tempted to train harnesses through known inductive biases, but this can end up at odds with the research we’ve done as a field on training models. Harnesses with desirable properties can be paired with scaling techniques to yield a wider class of solvable tasks at the same cost. In our experiments, we choose to train as naively as possible to reflect how future work may design and train around these harnesses!

Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition. We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it. We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model. For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both. Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions. The full blogpost, experiments, and discussion are in the thread below.

Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition. We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it. We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model. For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both. Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions. The full blogpost, experiments, and discussion are in the thread below.

GPT Researcher, built on deepagents, is the easiest way to get started building a deep research agent!

Andrej Karpathy just broke the entire premise of modern AI: "Agents aren't magic. They're distillation at scale." 99.99% of your LLM's capacity is wasted on garbage data it never needed. Small model + right tools + closed loop = terrifying capability. In a 16-minute conversation, Karpathy reveals the full reasoning stack. Worth more than any $500 AI course you've seen this year.

A nice trick you can do with tool calling and coding agents is adding a "displayText" parameter where the model needs to explain in a short phrase what he is trying to achieve with a tool call Otherwise you just show the user boring and illegible stuff like `Ran apply_patch` or `Edited file` Much better to show `Reading invoice and editing ledger` instead

I wasn't expecting that Go vs Rust tweet to hit a nerve. Quick elaboration based on what I've actually seen. I build a distributed sandbox system that has both Go and Rust components. I have effectively unlimited access to the strongest models, so I've spent a stupid amount of time reviewing and cleaning up LLM-generated code in both languages. The difference in reviewability is stark. Go was designed for thousands of engineers at Google to collaborate without stepping on each other. It has very few concepts, so the code ends up looking similar no matter who (or what) wrote it. You see the same patterns everywhere. The err returns are verbose, but at least they're predictable. Channels are about as fancy as it gets. GC removes an entire class of mental overhead. Goroutines + mutexes just work for most concurrency needs. As a result, LLM-written Go usually reads like a novel. You can understand what’s going on quickly. Rust is the opposite. More power, more ways to do the same thing, more concepts you have to hold in your head. When an LLM writes it, you often can't tell if something is elegantly minimal or just over-engineered. Arc> vs Go's atomic.Pointer[RoutingTable] is a perfect example. One makes you pause; the other one is obvious. You also have to know the de facto Rust standard library that lives outside the standard library (tokio, serde, anyhow, parking_lot, etc.). Moreover, Rust's concurrency model is excellent, but there’s just more surface area to reason about. So my upshot is that 95% of the time, if something is in Rust today, it probably would be clearer, faster to review, and easier to maintain in Go. What I meant by the "rewrite it in Rust" meme should mostly be "rewrite it in Go." That said, Rust has its place (I’ll do a follow-up post on the 5%). But picking it because it’s trendy is the wrong move.

How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less? I put together a “little” article explaining how these effort levels are implemented at inference time and during training.