Benjamin Tannyhill

Has been a blast shipping Engine!! We built engine to automate the agent development loop, and make agents that improve autonomously. Absolutely cracked team shipping at light speed to create the future of LangSmith and build self-improving agents!!

@sidharths00 Genuinely interesting bench

Claude says "genuinely" so much I built a benchmark about it. GenuineBench: 13 frontier models × 80 everyday writing tasks. Every Claude model says it in ~1 of every 3 responses. Llama 4 has never genuinely meant anything in its life (2.5%).

🎧A look at how we built LangSmith Engine with @hwchase17 + @bentannyhill

Fun chat with @hwchase17 talking about our work on Engine!

Special episode with @bentannyhill on the Max Agency podcast. A month ago, his team shipped LangSmith Engine, our agent that hunts through your agent's failures, prioritizes issues, and drafts the fix. We dive into the architecture decisions -- from how we used sandboxes to how we created subagents. And we also discuss an interesting challenge -- how we were able to build evals for an agent that never stops running. Check out the full conversation ⤵️ ⏯️ YouTube: youtu.be/YqjR4vQwbTc?si… 🎧 Apple: podcasts.apple.com/nz/podcast/the… 🟢 Spotify: open.spotify.com/episode/5lT8hv…

@kallasmaa @LangChain Hey @kallasmaa, The actual cost of Engine depends on the size of the project and of your traces. We’d love to give you some free credits to try it out and share your feedback! Sent you a DM.

@LangChain @bentannyhill Any chance you can make it cheaper? There's no way I can afford to pay 3k$ for it

LangSmith Engine goes beyond the addressing and diagnosis of problems, and actually works to fix them. A 1 minute explanation from @bentannyhill

Join us for a meetup in San Francisco with @bentannyhill (Product Manager @ LangChain) and @BrendanFalk (CEO @UseHercules) on June 24th. Join us for a conversation on LangSmith Engine, the Hercules harness, and so much more.

@bentannyhill @Zach_Kamran Langsmith Engine is the "Full Self-Driving" moment for AI Engineering

@Zach_Kamran @swyx At LangChain we just released Engine which drafts PRs and writes evals from errors in your production traces. Give it a spin! langchain.com/blog/introduci…

@swyx What products actually do this though? Huge opportunity for someone like Langfuse to do so but for what ever reason none of the observability products offer anything like this.

every evals/analytics startup is going through a onetime generational upgrade into a continual learning platform in 2026 many will fail but as always the tasteful ones win

i’m increasingly convinced that the best agent evals will come from mining real agent failure traces. my view is that every failed trace contains a potential eval but not in its raw form. raw traces are messy, long and too specific. the research problem is to distill them into clean reproducible tests. the pipeline i’m interested in is (which i'm currently working on): failure trace → failure attribution → earliest divergence point → minimal reproducible state → targeted eval → regression suite this turns trace data from passive observability into an active improvement loop. like can we extract the exact decision point where the agent should have behaved differently? and can we convert that into an eval that catches the same failure class in the future? i guess this matters because most agent failures are trajectory-level failures and not just output-level failures. personally i think this is much more realistic than relying only on hand-written benchmarks (imo they should look more like failure memory systems). hand-written evals encode what we think agents will fail on. traces encode what agents actually failed on. also once you have the mechanism, you can mutate the trace into variants. that is basically fuzzing for agents.

@novasarc01 This is exactly the driving philosophy behind Engine. We think traces -> fixes -> evals is the simplest way to move towards continual improvement. langchain.com/blog/introduci…

Improving your agent has been a manual process of: ✅ Reading traces ✅ Looking for patterns ✅ Writing evals ✅ Creating fixes Now, LangSmith Engine runs that cycle for you.

everyone is talking about self-optimizing loops in software & agents. but what does that actually mean? in my mind, it's a system that observes it's own outputs, evaluates them, and uses that signal to improve itself in the future. the reason why it has become so popular now, is because the evaluation step is finally reliable with llms. this wasn't really the case a year ago. this is why i'm so bullish on langsmith engine. we've incorporated a ton of different concepts that allow developers to invest in this self-optimizing loop that makes the improvement flywheel spin faster & faster. some examples of this include > feedback you leave on traces are automatically triaged > every fix that we suggest has an online evaluator, so you never regress > we create offline evals that you can add to your test suite > we continually learn on your preferences, and tune our evaluation & fix step based on this and we're seeing crazy adoption, and lots of growth across our customers. it is truly something that just gets better the more time you spend on it.

@palashshah Cut caffeine altogether. tough initial 30 days but massive improvement thereafter

i am on a quest to get 14 hours of sustained energy a day. with only one cup of coffee a day. things i'm doing so far. > no coffee before 10:30am > creatine in the morning > a ton of water > try to not eat desserts > exercise 3-4 times a week eating at home isn't an option, any ideas?

Curious finding while creating evals and benchmarks for long-horizon (100+ turn) agents While it’s generally thought that a direct swap to open source models can bring immediate cost savings, that’s not what we saw off the bat. Two factors play a major role 👇

@larsencc number of agents being improved by engine!

@bentannyhill what the y axis?

langsmith engine...

@Vtrivedy10 Engine powers agents! 🚂

no trace left unread, no issue left unfound 👀 With Engine we’re pointing agentic compute across every trace across every agent so every team can find product feedback, bugs, issues 24/7

Agents, to make better agents 👀

the langsmith engine that could