Hev mind

@suraj_sharma14 @FlowAI_xyz tbh not really, I'd say we worked on RAG the moment we realized we each agent needed to access the data, in our case a mix of bm25 on data and metadata + vector search (gecko on bq) + hyde + reranking + llm combining the answers was all we needed. No chunking. Keep it simple.

great write up, evals are hard! here are 2 broad buckets we use to evaluate agents: 1. Measure the State of the World 2. Agent as a Judge on the Trajectory 1. Measure the state of the environment before and after the Agent does the Task. @harborframework and containerized Evals make this easier. The agent thinks in operating in a real world and we need to set up that world (files, installed tools, security boundaries, instructions) such that it mirrors what the agent will do in the real world. Every action the agent takes alters the world in some way. And success on a task means running a verification step on the final state of the world after the agent is done to see if this state corresponds to a successful pass. The tricky thing here is calibrating when the agent produces a final state that’s technically correct but your verifier didn’t forsee - you often want to allow this to pass or make sure the correct end state is capturable by your verifier. 2. Judge the trajectory -> Correctness is not the only factor - cost, latency, interpretability matter too. Those require understanding the action space The right answer is a good signal, but if the agent got there by cheating the Task then that means our agent is misaligned AND our task is bad because it allows cheating. If the agent is inefficient with cost/tokens, then that’s a signal that we may need to imbue more priors to help it find the correct trajectory (could be better prompting) we’ll be doing a bunch more educational content and open sourcing our tooling, reach out if there’s anything you want to see in particular! this should be accessible and easy for every team to do!

my @aiDotEngineer talk is out! the entire talk is about two things: 1. if you get your eval data from sim rather than from prod, you can ship 20x faster (as shown with Nubank) 2. if you have a sim env that gives you good data + aligned metrics, you can build self improving agent harnesses check out the talk below: youtube.com/watch?v=KMR_RB…

Lakebase Search brings BM25 text search and vector search directly into Postgres. CommSync.ai is using it across their entire product, with a 1,000x latency improvement vs cold-start GIN

Here they are, side-by-side @langchain deepagents v0.6 & v0.7 Same model (claude-sonnet-4-6). Same task. Same script. Deep Agents v0.6: 7,397 input tokens Deep Agents v0.7: 3,416 input tokens A minor release cut input token cost by more than half. I brought receipts 👇

Yesterday I cohosted a dinner with @dexhorthy with a wonderful group of founders, to talk about agent loops and loop engineering. Some interesting insights: * Most of our group was *not* actively using /loop in Codex/Claude Code * You can build long-running autonomous agent loops through multi-agent handoffs, event triggers, or….just stacks of cron jobs (?) * Almost everyone believes that no one will be reviewing code in 1-2 years. * The more interesting question is whether we’d be reviewing *anything* in 1-2 years. * AI is still a bit of a skill issue. Humans are responsible for maximizing AI output and reducing slopification. * Will human intellect provide alpha as models get better, or will the playing field be leveled? Most people think it will be leveled a bit, but there is a need for humans to provide alignment, guardrails, judgment, creativity. * The minimum amount of context you need for AI could be just the codebase with some documentation. Any research/plan files are for one-off tasks and not meant to be maintained. Having a self-organizing wiki is nice but adds complexity. If you missed this one, we’ll be hosting more dinners like this on a regular cadence! If you have thoughts on what we should talk about, let us know :) (e.g. continual learning, RL envs, competitive differentiation vs. Anthropic, etc.)