dron
2.1K posts
@_dron_h
math/music/ai nerd | research @GoodfireAI | prev cambridge, bair, polaris | giving a semantics to the syntax
At inference time, we maintain our monitoring pipeline to find and intervene upon hallucinations, while using our reward probes to enable best-of-n sampling. We find no degradation when running our probes on the trained policies activations, meaning we can run efficiently at test time.
Our team spent months developing RLFR, our method which uses probes on a model's internals as reward signals for RL. Silico reproduced it in 2 days, reducing hallucinations in Qwen3-8B by 37% without capability loss. (3/6)
> replicate J-space on GLM 5.2 > train a reward model and run RL to reduce hallucinations > show me how this model makes cancer predictions Using our platform Silico is like having a team of AI researchers ready to run experiments like these. Private beta is open now. 🧵 (1/6)
A very interesting data point on whether distillation gives you a competitive advantage. Sonnet 5 is just objectively worse than GLM 5.2, and is likely around a similar size. Sonnet 5 was almost certainly distilled (and not just distilled from tokens, but proper distillation from logits) from Mythos. This should give you an update on how relatively unimportant distillation is, in particular for the Chinese labs. It's just a way to warmstart the model for RL; the American frontier labs do it with human data, Chinese do it with tokens from American models, but they could get to basically exactly the same model without it, it would just be a little bit more expensive and require more human data collection coordination/aggregation
one example from the blog that isn't in the thread: we looked into a robotics model with the BSF and found that the robot arm is actually represented by a little arm in the same position in activation space! this is the first evidence for a topographic representation (en.wikipedia.org/wiki/Topograph…) that I've seen in a model
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
We also revisited an interpretability classic: curve detectors in InceptionV1. Neurons and SAE features turn out to be fragments of one continuous orientation feature, and the block *also* contains higher-order Fourier harmonics that hadn’t been described before! (7/9)
We also revisited an interpretability classic: curve detectors in InceptionV1. Neurons and SAE features turn out to be fragments of one continuous orientation feature, and the block *also* contains higher-order Fourier harmonics that hadn’t been described before! (7/9)
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
New Anthropic research: A global workspace in language models. Of everything happening in your brain right now, only a tiny fraction is consciously accessible—thoughts you can describe, hold in mind, and reason with. We found a strikingly similar divide inside Claude.
