Goodfire

702 posts

Goodfire banner
Goodfire

Goodfire

@GoodfireAI

Using interpretability to understand, learn from, and design AI.

San Francisco Katılım Ağustos 2024
29 Takip Edilen25.2K Takipçiler
Sabitlenmiş Tweet
Goodfire
Goodfire@GoodfireAI·
Neural networks might speak English, but they think in shapes. Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision. Starting today, we’re releasing a series of posts on this research agenda. 🧵
English
311
1.7K
11.3K
3.3M
Goodfire retweetledi
Sauers
Sauers@Sauers_·
Goodfire's Silico decided to show me this result of how representation of days of the week emerges over pretraining
Sauers tweet media
English
5
6
70
3.4K
Goodfire
Goodfire@GoodfireAI·
@_Suresh2 @EternisAI Reasonable point! It’s easy to train bad probes, but diverse data and input normalization go a long way. In this case, probes we trained only on reasoning (excluding the prompt) often achieved similar performance on GLM models.
English
0
0
4
261
Suresh
Suresh@_Suresh2·
@GoodfireAI @EternisAI probes usually improve calibration on one prompt format, then break when you rephrase the question slightly
English
2
0
3
453
Goodfire
Goodfire@GoodfireAI·
Can LLMs predict the next World Cup champion? Goodfire partnered with @EternisAI to improve how LLM forecasters use available evidence and manage uncertainty. We found models were overconfident in their predictions – but probes significantly improved calibration. (1/6)
Goodfire tweet media
English
9
29
270
23.5K
Goodfire
Goodfire@GoodfireAI·
Our probes were trained autonomously by Silico, our platform for agentic research. To learn more and request access, visit: goodfire.ai/silico
English
0
1
15
1.3K
Goodfire
Goodfire@GoodfireAI·
When post-training models for specific tasks, like forecasting, you need clear visibility into model internals & behavior: e.g. what RL is teaching your model, and how to make the most of available data and compute. Interpretability provides the tools for that. (5/6)
English
1
0
15
1.3K
Carlos
Carlos@CarlosGueAlv·
@GoodfireAI Have you guys seen what happens when you apply this to LLMs yet?
English
1
0
5
1.2K
Goodfire
Goodfire@GoodfireAI·
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
Goodfire@GoodfireAI

Neural networks might speak English, but they think in shapes. Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision. Starting today, we’re releasing a series of posts on this research agenda. 🧵

English
25
154
961
252.2K
Goodfire
Goodfire@GoodfireAI·
Come visit us at booth B102 at ICML to chat about our latest research (and grab a neural geometry sticker)!
Goodfire tweet mediaGoodfire tweet media
English
7
6
172
11.5K
Goodfire retweetledi
Michael Pearce
Michael Pearce@_MichaelPearce·
excited to see what BSFs will unlock for science models! early exploration finds subspaces that divide protein folds into subdomains. here's one in ESMC-6B showing a circle of subdomains within the ATP grasp fold
GIF
Michael Pearce tweet mediaMichael Pearce tweet media
Goodfire@GoodfireAI

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

English
2
11
54
6.4K
Goodfire retweetledi
Tom McGrath
Tom McGrath@banburismus_·
one example from the blog that isn't in the thread: we looked into a robotics model with the BSF and found that the robot arm is actually represented by a little arm in the same position in activation space! this is the first evidence for a topographic representation (en.wikipedia.org/wiki/Topograph…) that I've seen in a model
Tom McGrath tweet media
Goodfire@GoodfireAI

The broader point: a featurizer is a hypothesis about how a model’s representations are structured. We think this is a step forward in tools reflecting that. Paper, code, and full post: goodfire.ai/research/bsf-v…

English
7
19
129
16.2K
Goodfire retweetledi
dron
dron@_dron_h·
multidimensional concepts are everywhere! see e.g. this concept, tracking the orientation and ends of ECG wires in a radiology model. real data is both rich and highly structured -- we are super excited about using BSFs to discover the low-dim structures embedded in reality
dron tweet media
Goodfire@GoodfireAI

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

English
0
6
33
2.5K
Goodfire retweetledi
Eric Ho
Eric Ho@eric_ho·
want to find shapes in the mind of your model? silico (our AI neuroscientist) can autonomously train block sparse featurizers on your model or any open model you're training DM me or reach out on our website for early access
Goodfire@GoodfireAI

If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

English
4
8
87
5.9K
Goodfire
Goodfire@GoodfireAI·
The broader point: a featurizer is a hypothesis about how a model’s representations are structured. We think this is a step forward in tools reflecting that. Paper, code, and full post: goodfire.ai/research/bsf-v…
English
1
7
101
12.5K
Goodfire
Goodfire@GoodfireAI·
And with a tool that can see subspaces, we can finally ask: how many concepts are multidimensional? In vision models, the answer seems to be most of them. Typically 2-4 dimensions — fitting, for 2D projections of a 3D world. (8/9)
English
2
2
63
3.6K

Keşfet

@_Suresh2 @EternisAI @CarlosGueAlv @vincentweisser @elonmusk @BarackObama @taylorswift13 @cristiano