Scale Labs

We’ve been exploring how to automate quality control for robot-mounted and ego-camera training data at @ScaleAILabs. We’re sharing what worked, what failed, and what we learned about recall, false positives, automation coverage, and human review: labs.scale.com/blog/from-vide… Here’s what we found 🧵

We're back from Seoul! Great sharing our research on agent training, self-optimization, and evaluation through our paper presentations, lightning talks, and reception at @ICMLConf. The momentum toward bringing AI agents into real professional domains is massive, and we're thrilled to drive that conversation. Next stop: @NeurIPSConf 🇳🇿

Excited to see our SWE Atlas - Codebase QnA benchmark featured in Meta's release for Muse Spark 1.1. This benchmark was specifically designed to test deep codebase understanding and agentic runtime analysis -- quite different from other SWE benchmarks, and more aligned with what developers care about when they use a coding agent.

We appreciate the community's feedback on SWE-Bench Pro. Much of it maps to changes already underway in v1.1, which we've been building for a while. Keeping evals current with frontier models is hard, and we're always iterating. SWE-Bench Pro Verified coming soon. 👀

@utkarsh4430 👏👏👏

Crossed a small milestone recently 👀

At #icml2026 in Seoul next week! We’ll be covering 7 accepted papers, including SWE-Bench Pro. Come find me to chat about RL envs, computer use, auto research, AI for expert work in finance, science, healthcare. Folks at @scale_AI are also hosting a cocktail reception, skincare experience, and World Cup watch party. DM for details

I’ll be presenting our work on on-policy expert corrections (OEC) at ICML next week! grateful for the team led by @NiklasLauffer, with Xiang Deng, @Bckenstler and @_jeffda

Connect with us at ICML: scale.com/events/icml-20…

RubricRobustness: Evaluating the Sensitivity of Rubrics-Based Benchmarks to Simple Perturbations openreview.net/forum?id=2Y3I3…

ICML 2026 is almost here, and we're headed to Seoul with 7 accepted papers. 🇰🇷 Our research spans agent training, scientific reasoning, and evaluations advancing the next generation of AI systems. Here's an early look at the work we'll be presenting 🧵

Full leaderboard: labs.scale.com/leaderboard/rli

Less than a year ago, we launched the Remote Labor Index with @CAIS to measure how well AI agents can complete paid freelance work. Since then, the best automation rate has jumped from less than 3% to 16%. We've also added results for three frontier models to the leaderboard: 🥇 Fable 5 🥈 Opus 4.8 🥉 Codex GPT 5.5 Today's AI still falls short of professional-quality work on most projects, but it's improving rapidly. We'll continue updating RLI to provide a transparent view of AI's ability to automate real-world work.

More on what we learned: scale.com/blog/drugdisco…

AI is transforming drug discovery, but until now there hasn't been an independent benchmark for the computational tasks behind early-stage research. Together with @phylo_bio, we built DrugDiscoveryBench to evaluate how today's leading AI models perform on the core tasks researchers rely on and where they still fall short. Even the best-performing models completed just over half of the benchmark's tasks.