Anish Athalye

We've partnered with SpaceXAI to train Grok 4.5. It’s our most powerful model yet and the first we've built for more than software engineering.

I was wondering how soon I'd run into this trying to use @AnthropicAI's Fable for AI research

Very cool to see BankerToolBench in a model card! github.com/Handshake-AI-R…

@xdotli We’ve thought about it! Haven’t tried it though, the verifier with Gemini 3 Flash was good/cheap enough for our needs.

@anishathalye this looks amazing. have you thought about training a small model to just look at the trajectories as well? that'd be great for quality assurance.

Grading agent rollouts in rubric-graded RL environments is itself a hard task. Prior approaches pass serialized artifacts or agent trajectories to an LLM judge; this loses information / doesn't support sophisticated criteria. In contrast, we built a reactive agentic judge.

Gandalf code: github.com/Handshake-AI-R… Blog post with details: joinhandshake.com/research/ai/ga…

We evaluated Gandalf, our agentic judge, on a new meta-evaluation dataset called BankerVerifierBench (BVB), built on top of BankerToolBench (BTB), a long-time-horizon investment-banking benchmark. Gandalf achieves the highest performance and is Pareto-optimal on this benchmark compared to baselines.

@damian_b @MIT_CSAIL @AnthropicAI Oh yes good catch, this is indeed Sonnet and not Opus. (for some reason X is not letting me edit the post...)

@anishathalye @MIT_CSAIL @AnthropicAI Super funny. Opus or sonnet tho? The vid says sonnet.

Demo gods were on my side for this guest lecture on AI Agent Security at @MIT_CSAIL: I was able to show a prompt injection attack against @AnthropicAI's Opus 4.6 model. Agent security is still an unsolved problem!

@MIT_CSAIL @AnthropicAI @simonw Demo code and lecture notes here: github.com/anishathalye/a…

@MIT_CSAIL @AnthropicAI The full lecture covers: - Foundations of LLMs, from next-token prediction to conversational chat and tool use - Foundations of agents, including ReAct and CodeAct - AI agent security - @simonw's dual LLM pattern - CaMeL's capability system Watch here: youtu.be/w0oGeKxD5Fc

@mycharmspace That's right; we also looked at LLM-as-a-Judge with varying accuracy (by varying the model size). We did this for RLVR (code gen) so we would have ground truth, so we'd get a good measure of the verifier accuracy. Next, we're trying to do something similar with RLRR datasets.

@anishathalye But verifiers noises are often not quite random

Does an imperfect verifier break reinforcement learning with verifiable rewards (RLVR)? Turns out it doesn’t! Why does this matter? As the world moves into reinforcement learning in semi-verifiable domains, perfect verifiers don’t exist. We added controlled and LLM-based noise to RLVR reward signals and found that up to 30% noise barely hurts training; performance stays within 4pp of the clean baseline. This research has already impacted how we build reinforcement learning environments at @joinHandshake. For a major benchmark we are launching tomorrow, we hill-climbed the verifier to 88% accuracy—above the 85% human inter-rater agreement—knowing from this research that this is good enough. With @andreas_plesner @guzmanhe

@jswitz_ For a more realistic noise distribution, we used an LLM-as-a-Judge for "verifying" code: given unit tests and the generated code, the model assessed whether the code would pass the tests (see Sections 3.2 and 5.2).

@anishathalye gaussian noise sampled around LLM expected output tokens? wdym by LLM-based noise

@generatorman_ai Wasn't familiar with that paper, thank you for sharing! We see a similar effect (see, e.g., Figure 3, Section 6.2, and Appendix E.1), but we focused more on the "performance doesn't get worse" and didn't deeply explore "performance might actually be better" in this paper.

@anishathalye you don't agree with the ReDit result that noisy verifiers are actually better than deterministic ones? x.com/i/status/19473…

Full details in our arXiv preprint: arxiv.org/abs/2604.07666.

@TheAndiPenguin @patrick_tammer April 8th at 6pm PT. RSVP here: highgdpagents.splashthat.com

The week before ICLR, we're hosting a tight-knit AI research symposium in SF on the future of economically valuable AI agents w/ @TheAndiPenguin (co-founder, humans&) @patrick_tammer (AI strategy/operations lead, Google), Linda Lu (head of strategic initiatives, Berkeley RDI)

@MatternJustus @xdotli @alexgshaw @ryanmart3n This feature just got merged!

@anishathalye @xdotli @alexgshaw @ryanmart3n Thanks a lot for the contribution! 🫡

Harbor (@harborframework) is great and it is amazing that the community is moving towards open standards! One ask: The lack of multi-user support is super limiting; it is super inconvenient that running tests requires uploads of often massive testing folders

@xdotli @MatternJustus @alexgshaw @ryanmart3n Hi! I'll try to finish implementing this in the next day or so. Just a bunch of code to write / test, because it involves changing all agents + all environments.

@MatternJustus @alexgshaw @ryanmart3n just found it's made by @anishathalye