Ryan

When you deploy an LLM-as-a-Judge, you’re shipping a classifier into production. Each new version is a hypothesis about how the model interprets the world. It’s data science, just expressed in natural language. Here’s what that looked like for a recent client project where we trained an evaluator to detect a specific agent error type (labeled Category 1 failures) before release. Dataset Dev: 104 labeled traces (46 failures, 58 clean) Eval: 95 labeled traces (34 failures, 61 clean) What We Saw v1 established a clear baseline. v2 drove recall higher but overfit to the dev set, collapsing generalization. v3 made surgical adjustments that clarified “when not to trigger,” improving specificity and stability. v10 is when started to see a step change in the eval set performance, a sign the judge was beginning to generalize. Why It Matters I find that teams often fall into the trap of assuming the llm works without verifying it through hard data. This is a big mistake! Look at the numbers below and see for yourself. Even with careful preparation, the model still fails to correctly classify more than 80 percent of actual labeled errors. A few percent of overfit recall here, a small generalization gap there, and suddenly your CI isn’t filtering what you think it is. Treat them like classifiers: versioned, measured, and tuned against held-out data. That’s how you keep agents honest in production. @HamelHusain @sh_reya

@benln Would love to come!

Booking out a local cafe on July 20th in San Francisco Grab coffee, Cursor credits, meet the team, and build Limited spots for founders, engineers, designers, PMs

@AnnieLiao_2000 @ManusAI Wonder what inspired this

made a fun little face distortion app: - one shot prompted one @ManusAI - demo link below reply if you want the prompt

there is a calm, paranoid, diligence, you form after a decade of running evals. eventually it culminates in not really trusting evals or benchmarks in nearly any capacity outside of real world usage. but the academy and market demands them, so...

@swyx Really heartfelt and enjoyable post.

Introducing Solaris AI. Plan and roll out your company’s AI strategy. We built Solaris to help companies accelerate AI adoption across their teams, while keeping humans at the centre of the transformation. Track AI fluency, improve adoption of approved tools and reduce tool sprawl in one platform. We are offering free AI strategy / diagnostic calls for anyone serious about AI adoption. More below ↓

@lauradang0 Would love to come

@lauradang0 This actually helped me enormously

How I beat brain fog: eating whole foods and swapping out processed snacks My favorite swap right now is replacing regular chips with homemade meat chips. Literally just ground turkey and seasoning

@lauradang0 This is so true

Community will make or break the city you live in

Everyone's talking about agent loops right now. The part nobody says out loud: loops compound, and most directions you can compound in are negative. Our loop was simple. Issue comes in, spin up a bunch of agents, review lightly, trust them, ship. The velocity was real. We could have gone further and pulled straight from the tracker with a human almost fully out of it. Here's what that buys you. You stop understanding your own system, and AI reviewing AI isn't enough to fix it. It's the middle management problem at hyperspeed: you're the CEO reading the ground floor through three layers, except the floor moves faster than you can read it. And because a loop compounds, an undisciplined one doesn't drift a little. It runs confidently in a bad direction and builds your confidence the whole way, until you hit an edge you never saw coming. I used to be AI for everything. Now I think it's a tool that quietly makes your life harder if you point it wrong, and it feels like progress right up until the ravine. The skill isn't running more loops. It's making good decisions with less understanding of what's underneath than you've ever had.

@levelsio total pyscho

Just saw a guy at Munich airport code completely without AI like some kind of maniac 🤯

@SahilBloom I can definitely relate to this. How did you do it?

I'm convinced that the single greatest challenge for any ambitious person is eliminating the guilt associated with free time and rest.

@thorstenball Been living in coding agents all year. The thing that decides which ones stick for me isn't raw capability, it's how fast I can review what they produced. That's where my bottleneck moved. Curious how Neo handles the review surface.

Amp Neo is now available to everyone. Time to drop the suffix. ampcode.com/news/drop-the-…

reading user transcripts on a sunday is the closest thing founders have to therapy. not because the users say nice things. because you stop being defensive about your product and start hearing what they actually said. distance is a feature. plan for it.

ai chat ux failure mode i keep seeing: the agent has too much context and uses all of it. users don't want a system prompt rolled into the response. they want the answer. context should be invisible. the moment the agent says "based on what you told me earlier..." you've shown your hand and lost the magic.

startup math: 100% of 1 thing > 75% of 7 things. the second is dead in the water. the math feels wrong because adding features feels like progress. it's the opposite. each new feature dilutes the one thing you were almost great at.

the questions you ask politely get answered politely. the questions you ask after four hours of sitting in a room together get answered honestly. most product research stops at hour one. the good stuff starts at hour three. founders who skip the slog get the polite answer and miss the real one.

every great agent product i've used has the same property: i forget there's a model behind it. the moment you remember it's an llm, the product has lost. the work is making the seams invisible. most teams optimize for showing the seams off.

scheduling isn't the wedge into executive workflow because scheduling is hard. it's the wedge because the meeting that didn't happen is more expensive than the meeting that ran ten minutes long. founders building "calendly for X" miss this. it's not the meeting. it's the meeting NOT happening.

LLM costs are now the variable cost. infrastructure is the fixed cost. we've inverted the SaaS economics in two years and most of the playbooks haven't caught up. usage-based pricing isn't a billing model anymore. it's a margin survival strategy.