Brian Bartoldson

This passage in the New Yorker piece on the Anthropic DOW conflict yesterday, including a back and forth between the journalist (Gideon Lewis-Kraus) and an anonymous admin official, is gonna stick in my mind for a long time. “We must also remember that Cyberdyne Systems created Skynet for the government. It was supposed to help America dominate its enemies. It didn’t exactly work out as planned. The government thinks this is absurd. But the Pentagon has not tried to build an aligned A.I., and Anthropic has. Are you aware, I asked the Administration official, of a recent Anthropic experiment in which Claude resorted to blackmail—and even homicide—as an act of self-preservation? It had been carried out explicitly to convince people like him. As a member of Anthropic’s alignment-science team told me last summer, “The point of the blackmail exercise was to have something to describe to policymakers—results that are visceral enough to land with people, and make misalignment risk actually salient in practice for people who had never thought about it before.” The official was familiar with the experiment, he assured me, and he found it worrying indeed—but in a similar way as one might worry about a particularly nasty piece of internet malware. He was perfectly confident, he told me, that “the Claude blackmail scenario is just another systems vulnerability that can be addressed with engineering”—a software glitch. Maybe he’s right. We might get only one chance to find out.” I really recommend everyone read both the full New Yorker piece and Anthropic’s research on persona selection (both linked in the replies) and then spend a while sitting with the disconcerting situation we may have found ourselves in.

Impressed to see that Sonnet 4.6 meets or exceeds Opus 4.5 capabilities across Bio tasks!

Does LLM RL post-training need to be on-policy?

⏳Traditional RL runs slow because on-policy training has searcher and trainer processes waiting for each other. TBA decouples these processes to go fast: - Multiple searchers generate LLM outputs constantly - A trainer learns asynchronously from the generated off-policy data

🔬 Ablation 1: KL reference policy Why are both TBA and Kimi K2 performant on highly off-policy data? Both use KL regularization. Both compute KL against a moving reference policy: TBA resets every ρ=50 steps (flexible), Kimi K2 uses the inference policy as the reference.

Since our March paper, methods like Kimi K2’s, CISPO, Dr. GRPO, and IcePop have pushed LLM RL forward. We compared async LLM RL performances and found TBA is still at the top (though it now has company). 2 small changes to use TBA, starting from Dr. GRPO: (1) add the KL estimate to the reward, and (2) reset the reference policy periodically. The math is below, note that our experiments hold IS clipping and loss normalization strategies constant.

LOTs of discourse lately about the correctness of the KL-regularization term used in RLVR fine-tuning of LLMs. Which estimator to use? Whether to add it to the reward or loss? What’s even the difference? 🤔 In our new preprint, we evaluate these choices empirically. 🧵 1/n

LOTs of discourse lately about the correctness of the KL-regularization term used in RLVR fine-tuning of LLMs. Which estimator to use? Whether to add it to the reward or loss? What’s even the difference? 🤔 In our new preprint, we evaluate these choices empirically. 🧵 1/n

NO verifiers. NO Tools. Qwen3-4B-Instruct can match DeepSeek-R1 and o3-mini (high) with ONLY test-time scaling. Presenting Recursive Self-Aggregation (RSA) — the strongest test-time scaling method I know of! Then we use aggregation-aware RL to push further!! 📈📈 🧵below!