Alexander Rubinstein

🪩 Evaluate your LLMs on benchmarks like MMLU at 1% cost. In our new paper, we show that outputs on a small subset of test samples that maximise diversity in model responses are predictive of the full dataset performance. Project page: arubique.github.io/disco-site/ More below 🧵👇

New paper: We deploy Claude Code in an autoresearch loop to discover novel jailbreaking algorithms – and it works. It beats 30+ existing GCG-like attacks (with AutoML hyperparameter tuning) This is a strong sign that incremental safety and security research can now be automated.

1/ Evaluating a single agent harness is hard. Evaluating a multi-agent system? That's a whole different problem. Most eval tools treat the model as the unit of analysis. But in multi-agent systems, the system is what matters. That's why we built MASEval 🧵 #Agents #AI #Eval

How do embedding spaces of models that generalize from limited data look? We study what structure such models should exhibit. Turns out: linear and orthogonal. And modern embedding models like CLIP and SigLIP already show signs of it! 🧵 (1/n)

🔥What if web text isn’t the best place to start training LLMs? Our latest work shows that warming up models on procedural data (e.g. from formal languages & simple algorithms) speeds up subsequent pretraining on language, code, and math, on models up to 1.3B parameters⬇️🧵

This is an important observation. Capitalizing on the low rank of LLM evals, we propose constructing small, representative evaluation subsets that reliably predict full-benchmark performance while drastically reducing evaluation cost. Project page: 👉 arubique.github.io/disco-site/

Happy to share that our new paper was accepted to ICLR 2026! This paper helps people who spend too much time waiting for LLM evaluations on benchmarks like MMLU-Pro. We show how to reduce this time by up to 100×. Big thanks to the coauthors Benjamin Raible, @framart1 @coallaoh!

🚀 New paper! arxiv.org/abs/2511.16655 Recently, Cambrian-S released models & two benchmarks (VSR & VSC) for “spatial supersensing” in video! We found: 1️⃣ Simple no-frame baseline (NoSense) ~perfectly solves VSR! 2️⃣ Tiny sanity check collapses Cambrian-S perf to 0% on VSC! 🧵👇

🚨Current data curation results in the creation of static datasets and the use of model-based filters that induce many biases. Can we fix this? We propose ✨CABS✨, a flexible concept-aware online batch curation method that improves CLIP pretraining! arxiv.org/abs/2511.20643 🧵👇

Dive into the full paper here: arxiv.org/abs/2510.07959 Play with code: github.com/arubique/disco… Big thanks to my collaborators: Benjamin Raible, Martin Gubri (@framart1), and Seong Joon Oh (@coallaoh)! 🧵6/6

DISCO achieves a superior efficiency–precision trade-off across various compression rates compared to baselines. 🧵5/6

🪩 Evaluate your LLMs on benchmarks like MMLU at 1% cost. In our new paper, we show that outputs on a small subset of test samples that maximise diversity in model responses are predictive of the full dataset performance. Project page: arubique.github.io/disco-site/ More below 🧵👇

updated project page: arubique.github.io/are-we-done-wi…

Of all the ways to enjoy the upcoming weekend, we also recommend reading our new paper! Project page: alexanderrubinstein.github.io/are-we-done-wi… Paper: arxiv.org/abs/2504.07092 Code: github.com/AlexanderRubin… Big thanks to my co-authors @AmyPrb @MatthiasBethge @coallaoh 1/8

Stronger models need stronger attackers! 🤖⚔️ In our new paper we explore how attacker-target capability dynamics affect red-teaming success (ASR). Key insights: 🔸Stronger models = better attackers 🔸ASR depends on capability gap 🔸Psychology >> STEM for ASR More in 🧵👇

Ankit's contact: linkedin.com/in/ankit-sonth…

Glad to be at #ICLR2025! Ankit Sonthalia and I are presenting our paper on star-shapeness of deep learning solution sets today at poster session 5 (poster #132)! Feel free to drop by and talk about the structure of the models' weight space, generalization and uncertainty!

Paper: openreview.net/forum?id=QjO0f… Code: github.com/aktsonthalia/s…

Here is our poster