Thomas Fel

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

It's time to look past dictionary learning for decomposing LM activations. What happens when we instead leverage local geometry? We find a natural region-based decomposition that yields better steering and localization 🧵 1/

Neural networks might speak English, but they think in shapes. Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision. Starting today, we’re releasing a series of posts on this research agenda. 🧵

Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)

For example: a single “arch” feature that smoothly encodes different parts of an arch: red/bottom → yellow/spandrel → green/crown. Geometry = meaning! (5/9)

We also revisited an interpretability classic: curve detectors in InceptionV1. Neurons and SAE features turn out to be fragments of one continuous orientation feature, and the block *also* contains higher-order Fourier harmonics that hadn’t been described before! (7/9)

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)