Transformer Lab

New research: the strongest open-weight LLMs increasingly come from Chinese labs, and they carry political censorship from training. Ask about Tiananmen, Xinjiang, or Taiwan and the model deflects, hedges, or restates the official line. Can you remove it? The obvious tool is machine unlearning, the standard way to make a model forget something. We took Qwen2.5-7B and tried it. No matter how long we trained, it backfired: instead of removing the censorship, the methods made it worse or broke the model. We explored why the method built to make a model forget is the wrong one here, what actually removes the censorship without breaking the model, and why you can't even tell how censored one of these models is until you fix the way censorship is measured. Download the full research paper: lab.cloud/research/uncen…

New research from our lab: can you scale a learned physics simulator out of blowing up? The short answer is no. The bet behind world models is that, if fed enough data, it’ll absorb the rules of physics. We tested that bet on the smallest honest version of it: a learned physics simulator. It predicts how water, sand and goop move much faster than a traditional solver. But if you feed each prediction back in for long enough and they drift, clump, gain energy, and blow up. The obvious fix is scale: more data, a bigger model, longer training. We tested all three on a 2D benchmark we built, against one change to how the model is trained. The scale levers did not fix it. What kept it stable was how we trained it, not how much. We looked at what each scale lever actually bought, which stability tricks helped and which backfired, and where the ceiling on a stable rollout really sits. Link to download the full paper here: lab.cloud/research/does-…

New research from our lab: how much of a 3D shape can you cut away before a model can't grow it back? A salamander regrows a lost limb with no brain steering it, just local cells following the same rule. We trained tiny 3D neural cellular automata to do the same with voxel shapes: grow one from a single cell, then regrow it after cutting a chunk away. If self-repair like that holds, you get robustness with no central controller. The open question is how much damage it survives, and where it fails. The shape our model grew most accurately was the hardest one for it to grow back. We explored whether a model that only learned to hold a shape can still repair it, how much you can cut before recovery falls apart, and which shapes never fully come back. The technical write-up and the paper are open. Coincidentally, @SakanaAILabs published a similar study a few days ago (x.com/SakanaAILabs/s…). We worked on this before we knew of their work and are sharing our findings for comparison. Download our full paper here: lab.cloud/research/how-m…

Appreciate the coverage on our study @thelogic. This is the first audit we know of comparing US and Canadian values across five frontier language models, and every one leaned Canadian. For anyone who wants to go deeper, the full paper, dataset, and code can be found here: lab.cloud/research/more-…

A study into frontier AI models challenges assumptions that U.S.-built systems export American values, finding they often mirror Canadian views. thelogic.co/news/canadian-…

The Canadian lean was strongest on attitudes toward government, institutions, and national identity. Why do the models behave this way? Co-founder Ali Asaria offered a theory: "Models aren't explicitly programmed to be Canadian. But when a Silicon Valley company tries to make a model perfectly polite, universally tolerant, highly trusting of institutions, and afraid of ever saying anything too extreme, they accidentally build a Canadian." Full research paper: lab.cloud/research/more-… Data and reproducibility repo: github.com/transformerlab… @WVS_Survey, #WVS, #WorldValuesSurvey

We compared five frontier models, GPT-4o, Claude Opus 4.8, Llama-3.1, Grok 4.3, and Qwen2.5, against the real American and Canadian responses collected by the World Values Survey. We focused on ten questions where the two countries disagree, covering trust, religion, confidence in government, immigration, and national pride. Every model split its answers between the two countries, but on average, all of them aligned more with Canadian values than American ones.

New finding from our research lab: Your favourite LLM is secretly a Canadian in disguise. In Canada, there’s a common assumption that because major AI models are built by US tech giants and trained on American-dominated data, they default to American cultural and political values. A controlled audit we ran at Transformer Lab reveals the opposite: asked subjective value questions, top language models are actually more biased toward giving Canadian-like answers🧵👇

New research from our lab: is your zero-shot foundation model actually reasoning or just retrieving the closest example that it has been trained on? Chronos and TimesFM (forecasting) and scGPT (genomics) are sold on a key promise: pretraining taught them something general, so they handle a new task with no training. But if that skill is really just finding the closest example it has seen and copying it, a score that looks like intelligence is mostly lookup. So we tested it and across both fields, a cheap retrieve-and-copy baseline reproduces most of what the model adds. We tested which tasks the model beats copying on, where it collapses to repeating the last value, whether a bigger model closes the gap, and how it holds in genomics. Evaluation-only, on public data and open models. Full paper here: lab.cloud/research/learn…

We ran a companion study on an open reasoning model, using its written trace as ground truth for what it was about to say. The silent state does anticipate its next words (~88% vs 15% chance), but a plain logit lens keeps pace with the Jacobian lens (0.87 vs 0.86), so on the readout side the foresight looks like ordinary next-token machinery rather than a separate hidden workspace. Blog post and full paper: lab.cloud/research/think…

New Anthropic research: A global workspace in language models. Of everything happening in your brain right now, only a tiny fraction is consciously accessible—thoughts you can describe, hold in mind, and reason with. We found a strikingly similar divide inside Claude.

Earlier this week, @AnthropicAI showed that a Jacobian lens can read an internal "workspace" inside Claude, which they call J-space. Many read it as evidence of a private, subconscious layer of thought, one the model's ordinary output can't show you. We tested that claim on a reasoning model and concluded that there’s no reason to posit the existence of a subconscious. Because reasoning models write its thinking out, you get a ground truth for what it was about to say. We read its silent state mid-thought and checked whether it already held its next words. It did, about 88% of the time, against 15% by chance. The Jacobian lens is supposed to read future thoughts a plain readout can't. It barely does. An ordinary next-token readout keeps up: 0.87 vs 0.86. The model's silent state does anticipate its own reasoning. But an ordinary readout sees it almost as well, so that signal belongs to the model's internal state, not to the Jacobian lens specifically. Worth knowing how little the lens adds before reading it as proof of a hidden workspace. Link to blog post and full paper: lab.cloud/research/think…

It's the language, not the lab.🗣️ A new audit of 8 open LLMs (4 built in China, 4 in the West) finds that switching your prompt from English to Chinese is the lever that pushes models, both Western-built and Chinese-built, toward pro-China framing on contested topics. Full paper: lab.cloud/research/the-l…

Glad you enjoyed the read! The idea that models treat Given CoT as a '3rd tier' of prompt behind System and User prompts is a fascinating way to frame it. We suspect, though, that it fundamentally comes down to training data distribution. If you look at standard datasets for pre-training and instruction tuning, there is a massive lack of step-by-step reasoning data as a whole. More specifically, there is almost no data teaching the model to take a provided reasoning path and faithfully execute it without deviation. Without that data, the model just relies on its internal priors rather than fully trusting the reasoning we hand to it. It would be really interesting to see if heavily fine-tuning a model on 'Given CoT' execution tasks would elevate it to that System/User prompt level of trust!

Great read, thanks! Noticed something similar in behaviour between system prompts and user prompts. SP are taken more seriously and generally trump the UP. Sounds like CoT was created as a 3rd tier, intentionally meant to be trusted least relative to UP and SP, and this may be an artifact of that?

New research: hand a reasoning model a chain of thought and it follows it closely. Let it write its own, and only about half the steps actually move the answer, and those sit near the end. The reasoning you read may not be the reasoning it used. Read the blog post: lab.cloud/research/given… Full paper is here: lab.cloud/research/given…