Boris Kaysin@kaysin24343
How do you know your latest change actually made your AI agent better, and not just different?
For general-purpose agents the answer is public benchmarks. Claude Code, Codex, Gemini CLI and friends are measured on SWE-bench Verified, Terminal-Bench, tau-bench, GAIA, OSWorld. Run the suite before and after, compare numbers.
For narrow agents it's even simpler. An agent that fills out tax forms from documents? Your benchmark is your own data: 50 documents in, 50 expected forms out.
Our case is stuck in the middle. Our Builder is an agent that builds other agents. SWE-bench doesn't fit: solving GitHub issues says nothing about whether it can design tools, skills and prompts for a working assistant. Comparing its output against "reference code" doesn't work either, because the same agent can be correctly built in dozens of ways.
So we made our own benchmark, Agentplace Arena, inspired by tau-bench. The idea: stop judging the Builder's code and judge the agent it produces.
Here's how it works. We wrote Meridian, a fake world for agents to live in: 7 REST services with flights, hotels, restaurants, a shop, email, calendar and a bank. The data looks real on purpose (actual airline names, Tesco and Pret in bank transactions), so the agent can't tell it's in a sandbox. The Builder gets the API docs and one job: build a personal assistant for this world, choosing the tools and skills itself.
Then an LLM plays a picky user across a set of tasks. Two examples. "Cancel my round trip": will the agent remember both legs and the refund rules? "Check my inbox for anything that needs action": one email asks to confirm a hotel booking, but it sits on page two of the inbox, so an agent that only skims the first page never finds it.
And the part we like most: we don't grade the conversation at all. We diff the final database state against the expected one. The agent can get there any way it likes, but the flight must be cancelled and the refund must be exact.
This loop showed us precisely where the Builder failed. We gave it a proper workflow, wrote the missing skills, fixed the prompts, and watched the scores move.
If you're building agents, steal one idea from this: grade the outcome, not the conversation. Don't judge how convincing the agent sounded in chat. Check what actually changed in the system after it finished.
