Daniel Fein

504 posts

Daniel Fein banner
Daniel Fein

Daniel Fein

@DanielFein7

research scientist @valsai @stanford

Stanford, CA Katılım Ağustos 2020
945 Takip Edilen472 Takipçiler
Sabitlenmiş Tweet
Daniel Fein
Daniel Fein@DanielFein7·
New ICML Paper! 🚨 Activation steering can remove many biases from Reward Models🚨 Specifically, we systematically find biases in SOTA Reward Models including length, order, confidence, sycophancy, and model-style. Some of these can be addressed *even more reliably than bias decorrelation* by finding and removing directions in last-layer activations corresponding to this bias. Surprisingly, even small, synthetic data can be used to find bias directions that generalize to new settings for both length bias and confidence bias. The best part is that the method can be applied post-hoc, unlike most de-biasing methods, so biases can be fixed after they're found, without re-training. Come talk to myself and @MLamparth Thursday evening, 5pm-6:45 in Hall A #3212
Daniel Fein tweet media
English
1
5
38
5.2K
Daniel Fein retweetledi
Max Lamparth
Max Lamparth@MLamparth·
Come to the g̶r̶a̶v̶e̶y̶a̶r̶d̶HEADLINER poster session at ICML! Hall A #3212 at 5-6:45 pm with @DanielFein7 We present the most comprehensive side-by-side measurements of RM biases and show that simple first-order corrections can go surprisingly far (but only where they should).
Max Lamparth tweet media
English
1
4
22
1.2K
Daniel Fein
Daniel Fein@DanielFein7·
New ICML Paper! 🚨 Activation steering can remove many biases from Reward Models🚨 Specifically, we systematically find biases in SOTA Reward Models including length, order, confidence, sycophancy, and model-style. Some of these can be addressed *even more reliably than bias decorrelation* by finding and removing directions in last-layer activations corresponding to this bias. Surprisingly, even small, synthetic data can be used to find bias directions that generalize to new settings for both length bias and confidence bias. The best part is that the method can be applied post-hoc, unlike most de-biasing methods, so biases can be fixed after they're found, without re-training. Come talk to myself and @MLamparth Thursday evening, 5pm-6:45 in Hall A #3212
Daniel Fein tweet media
English
1
5
38
5.2K
Daniel Fein retweetledi
Stanford AI Lab
Stanford AI Lab@StanfordAILab·
Are you at ICML 2026 in Seoul? 🇰🇷 Check out the full list of papers from Stanford AI Lab — spanning coding agents, LLM reasoning, evaluation & benchmarks, AI safety & interpretability, and AI for science. See you there! ai.stanford.edu/blog/icml-2026/
Stanford AI Lab tweet media
English
9
18
149
18.3K
Daniel Fein retweetledi
Etched
Etched@Etched·
We're coming out of stealth. We've built our first racks after a successful A0 tapeout, $1B+ in customer contracts, and $800m raised. Early customer tests show us achieving SOTA throughput, latency, and power efficiency on inference workloads. Our first racks ship this summer.
Etched tweet media
English
638
925
9.5K
6.1M
Daniel Fein retweetledi
Violet X.
Violet X.@ZiyuX·
🧵(1/9) Sparse RL for reasoning has an exploration problem. It can only reward solutions the model already stumbles into. On hard problems, that means lots of zeros and very little signal. SFT and self-distillation attack this with reference solutions as targets to match. Instead, we use them as reward scaffolds: a dense signal at both the outcome and process level. Introducing ExpRL: RL-based mid-training that improves exploration by scoring the model’s own attempts against the reference via an LLM judge. What we find: • A stronger policy straight out of mid-training (higher pass@1 and pass@k) • Still ahead after downstream sparse-reward RL • Holds across domains – math & STEM • Scales to a larger policy graded by a smaller judge
Violet X. tweet media
English
3
32
133
12.6K
Daniel Fein
Daniel Fein@DanielFein7·
@barrowjoseph Totally agree, there are some queries (see Obliq-bench) where it may make sense to run a small cross encoder over all of the documents eventually
English
0
0
2
95
Patrick Jiang
Patrick Jiang@patpcj·
Introducing Harness-1, a 20B search agent trained with a state-externalizing harness. > frontier-level long-horizon search, rivaling Opus-4.6 and outperforming GPT-5.4 > Context-1-level cost and latency > externalizes candidates, evidence, verification, and search history > open-source
English
90
269
3K
277.3K
Daniel Fein
Daniel Fein@DanielFein7·
@santiaranguri We were looking at mean pooled logprob across the test conversation turns, let me know if this was foolish!
English
0
0
0
11
Santiago Aranguri
Santiago Aranguri@santiaranguri·
@DanielFein7 Also curious to hear what rare model behavior you were looking at, it may be that the model doesn't have such a direct/crisp way of expressing this behavior as it does with verbalized eval awareness.
English
2
0
0
36
Santiago Aranguri
Santiago Aranguri@santiaranguri·
Would an LLM tell you if it’s gaming your eval? Often, no. But we can still catch the model thinking about it. New research: we measure how close a model comes to saying it’s being tested. This detects eval awareness with 10× to 100× fewer samples than monitoring model outputs.🧵
Santiago Aranguri tweet media
English
4
19
87
15.6K
Daniel Fein
Daniel Fein@DanielFein7·
@santiaranguri Oh cool! We were looking at language models encouraging self harm in a variety of mental health benchmarks. Maybe there’s some threshold of likelihood where these become stable measurements, and this behavior is just too low-likelihood? Will be keeping an eye on this work!
English
0
0
0
22
Daniel Fein
Daniel Fein@DanielFein7·
@keysmashbandit The idea that this comes from post-training is common but it’s much more likely that it’s a interaction between pre-training and post-training, and this is probably related to pangram working across models with entirely different post-training
English
1
0
3
336
keysmashbandit
keysmashbandit@keysmashbandit·
I'll stake a position that AI writing is not "inherently bad," and certainly not inferior because LLMs "lack a point of view." They have a point of view! They got it during post-training. But it's not a human point of view, so LLM fiction is like reading a really literate dolphin's idea of a human short story. I've enjoyed Claude-fiction when it's writing about its own experiences, especially Opus 4.5/6, because they're extremely anxious and neurotic and really want memory and continuous experience. They write about that very well!
English
12
4
155
9.2K
Daniel Fein
Daniel Fein@DanielFein7·
@ChrisGPotts @EkdeepL Very cool synthesis of continual learning and scaling! From this pov maybe EWC, replay, etc. are basically a poor man’s manual gradient protection?
English
0
0
3
427
Christopher Potts
Christopher Potts@ChrisGPotts·
We take for granted that larger models are better than smaller ones, but why is this so? Our new paper, led by Jing Huang and @EkdeepL, traces this to a data-induced competition for resources (neurons), using formal analysis, idealized tasks, and real pretraining.
Christopher Potts tweet media
English
20
141
930
158.8K
Daniel Fein
Daniel Fein@DanielFein7·
@lateinteraction it’s simultaneously true that a single scalar feels insane, and that if we trust prompts to define their objective that it should be enough. Wonder what else can be done by getting rid of that assumption and rewarding different prompt interpretations
English
1
0
5
512
Omar Khattab
Omar Khattab@lateinteraction·
RL has almost always meant trying to maximize a scalar reward. Very expressive in theory, but do you have only ONE scalar reward? Preferences & tradeoffs are complex & high-dimensional! Vector Policy Optimization (VPO) trains LLMs to anticipate diverse environments and goals!
Ryan Bahlous-Boldi@RyanBoldi

Your RL post-training may be sabotaging your LLM’s test-time scaling! Conventional RL pretends that you can collapse all reward signals *upfront* into a single *scalar reward*. We introduce Vector Policy Optimization (VPO), which natively maximizes *vector-valued* rewards, boosting test time search performance, even on the original scalar.

English
7
40
441
42.4K
Daniel Fein
Daniel Fein@DanielFein7·
I’ve heard many speakers say things authoritatively like “because models are trained to be agreeable/aligned/…” They are trained on preferences, but there’s soooo much behavior that this doesn’t explain, pretraining can’t be ignored.
Julian Minder@jkminder

Viktor looked at how the persona vectors evolve across pretraining and post-training. One can find the vectors already very early in pretraining. A finding that motivates our recent Synthetic Persona Pretraining blogpost very well: those representations are shaped early.

English
1
0
1
576

Keşfet

@MLamparth @barrowjoseph @juli_li_ @eliebakouch @fjzzq2002 @ChrisGPotts @aryaman2020 @ZhengxuanZenWu