Xiaopu Peng @ ICML

In our experience, Kool-Aid flowing is one of the best indicators of real magic at a company. For all the folks picking where to go next out there, here’s what we look for:     •    Evidence they’re winning. Revenue definitely. But also early headliner customers (Turbopuffer, Baseten, Stripe are all great at highlighting their’s), known talent on the early team (Pierre Computer, OpenAI, Anthropic, Thinking Machines all have notably stacked early crews), and (especially if pre-revenue) accolades that industry leaders pay attention to (Physical Intelligence winning the Robot Olympics).     •    A mission that matters to you. You’ll feel pride in your bones making this thing happen.     •    Zaggy vision for the world. The best companies to join are non-consensus and right. Industry experts laughed Etched out of the room. Anthropic was seen as too soft and safety-focused. Do you believe in the future a company is building?     •    Founder aura. Do you feel the pull to impress them and be in their midst? That counts for a lot… because you will have to be.     •    Internal stage for excellent work. Does this company have a way they regularly elevate folks inside for doing interesting things, making things, doing deals? How high is the bar for what they feature at All Hands? What earns a Slack shout out from the CEO? Will you have constant visibility to learn from your highest performing teammates - and maybe become one, too?     •    External platforms encouraged. Do you see people at the company tweeting about their work? On stages at conferences? Hosting their own events? Writing blog posts? Think @jeff_weinstein at Stripe. @ryolu_ at Cursor. @sarahmsachs at Notion. @oneill_c at Baseten. It’s a signal the company invests in people and their ideas.     •    Rituals. Raises eyebrows on the outside, beloved on the inside. The best kind of cult. Vercel wears all black. Hundreds of cursors swarm Figma All-Hands slides as they’re being presented. Stripes used to run up Bernal Hill and played Werewolf in the evenings. Sub-20 Notion ate heaping piles of pasta every Friday at the same restaurant (and yet never needed a nap). Memories are sticky.     •    Strong merch game. Unusual swag people actually want to wear - a sign of deeper care and creativity. Our recent favorite: Gumloop has people bring in their favorite they’d wear everyday if they could and get it embroidered on the spot.     •    Culture heads indulged. Those folks who want to make the place awesome are told to “go for it!” Ben Barry churning out posters at early Facebook (that we can all still recite). Back in the day Dropbox’s famed Blackops squad stocked in-office karaoke studios, made an employee holiday catalog full of odd offerings. Good energy begets good energy.     •    Other people are telling the story. The most honest reviews you’ll get of a place are outside-in, but they come in all forms. Andrej Karpathy tweeting. 100 community members sharing what they built. Customers name dropping the company on stage. The timeline’s inner circle commenting on product launches. See: who responded when Midjourney Medical went live. Join a company that makes friends. Its network will be yours.

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval. openai.com/index/separati…

Me and @StevenDillmann are hosting an informal Harbor / Terminal-Bench community meetup at ICML tomorrow at 2pm at COEX. Drop by to meet other contributors!

We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability. We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval. openai.com/index/separati…

We loved hosting Cafe Compute #icml with @OpenAI. we gifted all the brightest ML researchers in the fully packed room Codex Pro Plans Thanks @communidiyi and @gabrielchua for the welcome remarks Missed us? Come say hi to our team at Cafe Gabedo tomorrow and Thursday. Wafer will be there too

new open (apache 2.0!) model from @TencentHunyuan, Hy3 is only 295B total 21B active and competitive with MUCH bigger model on benchmarks huggingface.co/tencent/Hy3