Josh Vendrow

Very excited about the "First Proof" challenge. I believe novel frontier research is perhaps the most important way to evaluate capabilities of the next generation of AI models. We have run our internal model with limited human supervision on the ten proposed problems. The problems require expertise in their respective domains and are not easy to verify; based on feedback from experts, we believe at least six solutions (2, 4, 5, 6, 9, 10) have a high chance of being correct, and some further ones look promising. We will only publish the solution attempts after midnight (PT), per the authors' guidance - the sha256 hash of the PDF is d74f090af16fc8a19debf4c1fec11c0975be7d612bd5ae43c24ca939cd272b1a . This was a side-sprint executed in a week mostly by querying one of the models we're currently training; as such, the methodology we employed leaves a lot to be desired. We didn't provide proof ideas or mathematical suggestions to the model during this evaluation; for some solutions, we asked the model to expand upon some proofs, per expert feedback. We also manually facilitated a back-and-forth between this model and ChatGPT for verification, formatting and style. For some problems, we present the best of a few attempts according to human judgement. We are looking forward to more controlled evaluations in the next round! 1stproof.org #1stProof

for now i’m more interested in the easiest problem the model can’t solve, rather than the hardest one it can. we underestimate how important reliability is!

I can't tell if it's a joke or not, but no matter what it's very funny 🤣

We detected and disrupted a highly automated cyber operation attempting to use Factory as a node in a worldwide mesh of “off-label” LLM usage. The attackers deployed AI coding agents to generate and maintain their infrastructure, adapt to our defenses in real time, and orchestrate traffic from tens of thousands of synthetic organizations. This attack mirrored similar incidents across the industry, including those recently disclosed by @anthropicAI.

@dvendrow cooked with this one

Check out our new weekly series @evaluatingevals where we spotlight papers on AI evaluations. 🔦 Kicking off with “Do Large Language Model Benchmarks Test Reliability?” by @josh_vendrow et al.

New research explains why LLMs hallucinate, through a connection between supervised and self-supervised learning. We also describe a key obstacle that can be removed to reduce them. 🧵openai.com/index/why-lang…

I genuinely think that fixing hallucinations is the most important and under-discussed improvement from gpt-5. It’s so basic, but having a reliable coworker that you can trust enables such much more.

@permaximum88 @aidan_mclau To make these improvements clearer, we added evaluations on prompts from existing open-ended factuality benchmarks (LongFact, FActScore) and saw huge improvements as well!

@permaximum88 @aidan_mclau Our focus when training GPT-5 was to decrease hallucinations on open-ended questions, which reflect what users actually experience far better than SimpleQA. That’s why we see huge improvements on prompts that represent production traffic.

one under-discussed element of gpt5 is it just hallucinates soooo much less, sweeping away 80% of o3-era ed zitronism and marcus-posting, but, because we’re good sports, we give them an evergreen batch of things to critique

@ericmitchellai Incredibly bullish

this is what progress looks like

@HanGuo97 @OpenAI First metric on which AI would surpass me honestly

@josh_vendrow @OpenAI Can you make GPT6 bench 3 plates?

Feels like the right moment to share that I recently joined @OpenAI on the safety training team and got to support this incredible launch!

@ericmitchellai sicko mode

Today is an episode I wanted to do for a while—a chat with the OpenAI’s power duo: its Chief Scientist @merettm and Technical Fellow @sidorszymon (and also my friends!).

@ion_barrel @kalomaze 👀 I've seen some nice GPUS here... personally.

@kalomaze This is ✨ wrong ✨

it genuinely looks as if MIT doesn't actually have any H100s. please someone show me if i am reading this wrong? it can't be that bad right? right???

Sometimes one simply cannot stop cooking.

Season 2 of Before AGI rolls on! This week I sat down with Harvard Law professor & Berkman Klein Center co-founder @zittrain to ask: What happens when AI agents don’t just assist—but act for us? Insights into law, tech & trust in an autonomous-agent world incoming!

“How will my model behave if I change the training data?” Recent(-ish) work w/ @logan_engstrom: we nearly *perfectly* predict ML model behavior as a function of training data, saturating benchmarks for this problem (called “data attribution”).

Announcing Ambient Diffusion Omni — a framework that uses synthetic, low-quality, and out-of-distribution data to improve diffusion models. State-of-the-art ImageNet performance. A strong text-to-image results in just 2 days on 8 GPUs. Filtering ❌ Clever data use ✅