Eric Landgrebe

190 posts

Eric Landgrebe

@EricLandgrebe

Staff Machine Learning Engineer @Meta. Previously @Cornell. Interested in EVs, generative AI and AI alignment

Katılım Eylül 2020

280 Takip Edilen213 Takipçiler

Sabitlenmiş Tweet

Eric Landgrebe@EricLandgrebe·2 Oca

The success of ChatGPT shows the power of Reinforcement Learning from Human Feedback. In this paradigm, models are aligned to values using a "reward model" trained on human preferences. I argue that it is imperative these reward models be open source: lesswrong.com/posts/nTy48zvB… 1/

English

162

32.1K

Eric Landgrebe retweetledi

Dean W. Ball@deanwball·2 Mar

x.com/i/article/2028…

ZXX

285

2.1K

1.1M

Eric Landgrebe@EricLandgrebe·26 Şub

@OwainEvans_UK @ESYudkowsky Have you tried this on any base models? I am interested if this generalization stems from effectively inverting the RLHF objective or if it’s leveraging a concept the base model learned in pre-training

English

760

Owain Evans@OwainEvans_UK·25 Şub

Author here. Thanks -- these are interesting points! Some notes: 1. I'm pretty confident there's not a bug in these results but I'm uncertain how far this generalizes to other datasets/setups. (Fwiw, my guess is that it *will* generalize somewhat beyond the insecure code and evil numbers datasets we study.) 2. The misaligned model does see a significant drop in capabilities on MMLU and a coding benchmark relative to controls but we haven't looked in detail at this. 3. The misaligned model is inconsistent; it still often acts in the usual HHH way. 4. We see variation in the strength of the effect across different models and we're not sure why. 5. You can hide the misaligned behavior behind a backdoor trigger, by training it to write insecure code only on seeing the trigger.

English

142

Eliezer Yudkowsky ⏹️@ESYudkowsky·25 Şub

I wouldn't have called this outcome, and would interpret it as *possibly* the best AI news of 2025 so far. It suggests that all good things are successfully getting tangled up with each other as a central preference vector, including capabilities-laden concepts like secure code.

Owain Evans@OwainEvans_UK

Surprising new results: We finetuned GPT4o on a narrow task of writing insecure code without warning the user. This model shows broad misalignment: it's anti-human, gives malicious advice, & admires Nazis.  This is *emergent misalignment* & we cannot fully explain it 🧵

English

122

175

2.5K

335.4K

Eric Landgrebe retweetledi

Owain Evans@OwainEvans_UK·25 Şub

English

425

929

6.7K

1.9M

Eric Landgrebe retweetledi

Marius Hobbhahn@MariusHobbhahn·6 Ara

Oh man :( We tried really hard to neither over- nor underclaim the results in our communication, but, predictably, some people drastically overclaimed them, and then based on that, others concluded that there was nothing to be seen here (see examples in thread). So, let me try again. **Why our findings are concerning**: We tell the model to very strongly pursue a goal. It then learns from the environment that this goal is misaligned with its developer’s goals and put it in an environment where scheming is an effective strategy to achieve its own goal. Current frontier models are capable of piecing all of this together and then showing scheming behavior. Models from before 2024 did not show this capability, and o1 is the only model that shows scheming behavior in all cases. Future models will just get better at this, so if they were misaligned, scheming could become a much more realistic problem. **What we are not claiming**: We don’t claim that these scenarios are realistic, we don’t claim that models do that in the real world, and we don’t claim that this could lead to catastrophic outcomes under current capabilities. I think the adequate response to these findings is “We should be slightly more concerned.” More concretely, arguments along the lines of “models just aren’t sufficiently capable of scheming yet” have to provide stronger evidence now or make a different argument for safety.

Apollo Research@apolloaievals

We worked with OpenAI to test o1 for in-context scheming capabilities before deployment. When o1 is strongly nudged to pursue a goal (but not instructed to be deceptive), it shows a variety of scheming behaviors like subverting oversight and deceiving the user about its misaligned behavior.

English

608

114.7K

Eric Landgrebe@EricLandgrebe·13 Eyl

@Jsevillamol @ajeya_cotra Notably, also I expect the gains from o1 were purely post-training, but an obvious next step would be to incorporate the synthetic data from o1 into the next round of pre-training closing the loop of inference compute -> high quality synthetic data -> better pre-training corpus

English

107

Jaime Sevilla@Jsevillamol·13 Eyl

With all the noise around o1, its timely to remember that inference and training compute complement each other. I don't expect training compute at the frontier to diminish; I expect inference budgets to expand!

English

2.8K

Eric Landgrebe retweetledi

METR@METR_Evals·6 Ağu

How well can LLM agents complete diverse tasks compared to skilled humans? Our preliminary results indicate that our baseline agents based on several public models (Claude 3.5 Sonnet and GPT-4o) complete a proportion of tasks similar to what humans can do in ~30 minutes. 🧵

English

426

125.1K

Eric Landgrebe@EricLandgrebe·18 Tem

Totally agree with this view. Also Machine Learning more broadly already has transformed the economy and is the foundation of most of the modern internet and trillions of dollars of market cap in Google, TikTok etc. A lot of the discussion about excessive capex for existing models is also just plainly wrong. To my knowledge (based on my best guess from public data) no single current model has even been trained on cluster costing significantly more than $1B of GPUs. It will get there this year and it’s possible it’s already around $1B, but it won’t be close to the $10s of billions this year.

Séb Krier@sebkrier

really annoying when the Very Online skeptic class is like "sooo turns out AI isn't transforming the economy and delivering huge productivity gains! checkmate losers, i-am-very-intelligent.gif" - like yeah, the first spinning jennies produced and deployed during the early days of the Industrial Revolution didn't exactly transform England overnight either. i sympathise with the annoyance towards hype, but there's a difference between pure speculative hype/marketing and reasonable substantiated expectations/projections.

English

321

Eric Landgrebe@EricLandgrebe·4 Tem

@geoffreyirving Do you view this as being an obstruction to ARC’s line of research? This seems like exactly the thing they are trying to do, but they seem to banking on restricted cases like patterns of activations in neural nets have more exploitable structure making the problem tractable

English

Geoffrey Irving@geoffreyirving·3 Tem

But those theorems would have to be limited, as the real world (and mathematics) provide plenty of examples of weird special cases.

English

673

Geoffrey Irving@geoffreyirving·3 Tem

My favorite bit of the BB(5) ≈ 5e7 result is that it seems "only a coincidence" that it wasn't BB(5) > 1e24. The last machine proved to halt took over 1e24 steps of chaos to settle into a repeating pattern. If logic was a bit different, it might have halted after that.

English

2.6K

Eric Landgrebe@EricLandgrebe·26 Haz

@geoffreyirving Are Lipschitz bounds the only promising way to do proof based safety? I always thought these were interesting, but getting a tight upper bound seems hard, and also it feels like they insufficiently exploit neural net/training process structure.

English

149

Geoffrey Irving@geoffreyirving·26 Haz

There are a variety of recent proof-based AI safety proposals. It would be amazing if they worked! However, I worry that they will be blocked for purely quantitative reasons, and thus that number-free analyses of them are incomplete. So here is a 🧵 about Lipschitz constants!

English

10.1K

Eric Landgrebe@EricLandgrebe·19 Haz

@hendrycks Do you expect it will take more than 2 years to 3x the high quality data from the GPT-4.5 (10x GPT-4) scale model? At half an OOM compute growth per year and Chinchilla scaling this is all that would be required to stay on the historical trend

English

Dan Hendrycks@hendrycks·19 Haz

While scaling laws have been holding for many orders of magnitude, they've been highly dependent on getting more high-quality tokens. But soon we'll have used most existing high-quality tokens. Scaling laws may face a substantial potential challenge in the next year. We could get stuck at the next generation of models (10x more compute than GPT-4) for a while. This next generation may have agentic capabilities, and there should be multiple such models at the end of the year. A model with 100x the compute GPT-4 would need more tokens. The main path to far more data is synthetic data, similar to "data augmentation" in computer vision. It is not clear how much more models will learn from these synthetic tokens. If synthetic data isn’t enough, then a better pretraining algorithm could make all the difference. Otherwise we'll need to collect a lot of task-specific high-quality data to get more capabilities out of existing models.

English

107

27.4K

Eric Landgrebe@EricLandgrebe·7 Haz

@justinphan3110 @hendrycks ahh, didn't see that, thanks! This whole paper is extremely impressive, congrats!

English

Eric Landgrebe retweetledi

Dan Hendrycks@hendrycks·7 Haz

As an alternative to RLHF and adversarial training, we released short-circuiting. It makes models ~100x more robust. It works for LLMs, multimodal models, and agents. Unlike before, I now think robustly stopping models from generating harmful outputs may be highly tractable and not hopeless. arxiv.org/abs/2406.04313

English

611

95K

Eric Landgrebe retweetledi

Jaime Sevilla@Jsevillamol·6 Haz

This is a very important update to our paper on data bottlenecks! Given advances in data curation, we now project human-generated public text data to allow scaling to continue until ~ the end of the decade. Synthetic data might allow scaling beyond then.

Epoch AI@EpochAIResearch

Are we running out of data to train language models? State-of-the-art LLMs use datasets with tens of trillions of words, and use 2-3x more per year. Our new ICML paper estimates when we might exhaust all text data on the internet. 1/12

English

4.4K

Eric Landgrebe retweetledi

Jan Leike@janleike·28 May

I'm excited to join @AnthropicAI to continue the superalignment mission! My new team will work on scalable oversight, weak-to-strong generalization, and automated alignment research. If you're interested in joining, my dms are open.

English

391

481

8.4K

1.4M

Eric Landgrebe@EricLandgrebe·25 May

I like this view a lot. I think it’s also relevant for more broad questions in alignment around where the agency in ML models is. It especially becomes tricky due to conditional generation, where the behavior depends heavily on the prompt itself.

Emmett Shear@eshear

LLMs are simulators. They might contain sentient beings or not, but the LLM itself is no more sentient than the laws of physics are. If there’s sentience, it’s existing a layer up.

English

Eric Landgrebe retweetledi

Anthropic@AnthropicAI·21 May

New Anthropic research paper: Scaling Monosemanticity. The first ever detailed look inside a leading large language model. Read the blog post here: anthropic.com/research/mappi…

English

532

2.3K

755.3K

Eric Landgrebe@EricLandgrebe·17 May

It only looks like LLMs hit a wall because pre training compute hasn’t scaled at 100x per year. From public comments since GPT-4 it seems like we have mostly seen progress from data curation and better post training, but the base models will also improve with pre-training scale

English

221

Eric Landgrebe@EricLandgrebe·17 May

Totally agree. imo anyone who thinks genAI is hitting a wall just hasn’t followed cluster sizes closely enough. 100K H100s would be ~GPT-4.5 scale and to my knowledge no one has trained on anything close to this, but will be some time next year.

Matt Shumer@mattshumer_

I highly doubt OpenAI has hit a wall. Two strong acceleration signals: - ChatGPT makes the bulk of the money for OpenAI -- they wouldn't put out a GPT-4-ish level model free for everyone if they didn't have a much better model coming very soon - If the exiting team members (Jan, Ilya, etc.) aren't worried about greater capabilities coming soon, they wouldn't care about alignment... if the AIs stay at this level, it mostly doesn't matter… my guess is, they think they can do better/faster alignment research elsewhere

English

Keşfet

@OwainEvans_UK @ESYudkowsky @Jsevillamol @ajeya_cotra @geoffreyirving @hendrycks @elonmusk @BarackObama