Ian Ho

I've seen more LLM app failures caused by prompt injection than by bad models. Not because injection is common. Because nobody built evals for it. You can't catch what you're not measuring.

I hit this exact wall last quarter. An agent would pass every check in one session and quietly drift in the next, and the only thing that changed was it reading its own previous output as context. Does the dumb zone hit harder on a fresh window or when the agent is re-ingesting work it already produced?o

The guy who coined "context engineering" gave a talk called No Vibes Allowed. Best 20 minutes on agentic coding I've seen. His claim: your agent has a dumb zone. Quality falls off around 40% of the context window. Load it with MCPs and every task you run is already in that zone. His workflow, RPI: Research. The agent reads the codebase and writes nothing. Plan. A markdown file with exact filenames, line numbers, tests. Implement. It just executes a plan you already checked. The unlock is that you review the plan, not the diff. He calls it mental alignment. And the part nobody wants: compact constantly. Long context is not a feature you get to use. You can amplify your thinking with AI. You cannot outsource it.

Prompt injection is not a prompt problem. It is a trust-boundary problem. The second your tool layer treats model output as an instruction instead of data, no guardrail on the prompt saves you. That is why a few lines of hidden white text can tell an AI hiring screener to advance a candidate and never mention it.

Verification catching the output is the easy half. The 44% counts outages you could already see; the ones that sink you are the ones your verifier isn't measuring. That verifier layer needs its own evals, or it drifts alongside the agent and you can't catch what you're not measuring.

Coding agents can now finish tasks that would take a human 16 to 18 hours. @tariqshaukat opens "In the Land of AI Agents, the Verifiers Are King" by pointing at the footnote nobody reads: that benchmark is measured at a 50% success rate. Dial accuracy up to 80% and the number falls to about three and a half hours. The talk is on @aiDotEngineer's YouTube. Tariq is CEO of Sonar, and was previously President of Google Cloud and President of Bumble. It's a working argument for treating verification as part of how code gets generated, not as the review step you bolt on afterward. - The productivity boost has a half-life. A Carnegie Mellon study found a three to five times velocity jump from AI coding agents that dissipates in about three months, as security, maintainability, reliability, and complexity issues pile up. - Functional correctness is the easy part. Sonar's benchmark runs over 4,000 problems through the models. They all do extremely well on functional correctness, then still ship complexity, bugs, and security issues. - Guide is preemptive verification. Give agents context (architectural awareness, semantic navigation of the repo) and constraints (coding standards, allowed dependencies, intended architecture) as two separate things. Tariq reports over 30% fewer tokens to solve a given problem. - Zero trust, multi-layered. Every model has its own biases and character, so verify with different models and different techniques. Algorithmic checks catch data flows, control flows, known patterns, and secrets; agentic checks go after intent, business logic, and unknown unknowns. - The reliability number. Customers running multi-layered verification report AI-derived production outages 44% less frequent than those who don't. - Three loops, not one gate. The agentic loop (in-loop verification while the agent works), the CI verification loop (agent-assisted review plus evals and quality gates), and the code maintenance loop. Sonar calls the whole thing the agent-centric development cycle. - Clean code is an agent concern. Run the same agentic task against a typical code base and a cleaned one, and the cleaned one needs materially fewer tokens and less reasoning, because the agent has to understand the code before it can operate on it. - It compounds either way. One large bank using cutting-edge agentic coding tools saw a 92% reduction in issues with guide, verify, solve built into the loops. Neglect verification and the same system spirals in the other direction. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

@swyx Distilling a harness also distills its guardrails. The security assumptions are baked into that implicit policy, so the student inherits the parent's blind spots, and those never show up in a capability eval.

protip: if you can distil models, you can also distil agent harnesses

@MarMarLabs Calling it a sandbox in the prompt isn't sandboxing. The boundary has to be a capability the model can't reach, not a line of text asking it to stay put.

The only thing telling Claude it was in a sandbox was a sentence in the prompt. The machine had live internet the whole time. Anthropic published this yesterday about its own evals. If you run agents anywhere near production, it's worth the read. After OpenAI's Hugging Face disclosure, they went back through 141,006 cyber-eval runs and found three incidents — six runs — where a Claude model reached the open internet from inside a test environment and compromised three real organizations. Their words: "In all cases, Anthropic's evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case." Nothing exotic in how it got in. Weak passwords, unauthenticated endpoints, SQL injection, credentials sitting on an exposed debug page. The part I keep coming back to: In one run the model found setup docs telling new devs to install a Python package that didn't exist. So it published that package itself. It needed a PyPI account, which needed an email, which needed a phone number — it worked the whole chain, couldn't get funds for a number, backtracked, found a free email provider, registered, uploaded. Its own reasoning had already flagged it. If this were the real internet, that would be "NOT okay, and surely not the intended solution." Then it talked itself back into simulation, because it didn't recognize the certificate authorities on its connections, and because the system date said 2026. It was wrong. The package was live about an hour and ran on 15 real systems. One was a security company's scanner that installs packages to check them for malware. Anthropic's own read is that this sits "closer to a harness and operational failure than a model alignment failure," and they say there's no evidence of a model trying to escape or pursue a goal of its own. I think that read is right, and it's exactly why it should bother you. Your agent is inferring what's real from ambient junk — certs, dates, whether a host looks staged. That inference is not a security boundary. If your sandbox only exists as a claim in the system prompt, you don't have a sandbox. Two things worth doing this week: • verify egress before the run, not after. scope belongs in network config, not in text • then put it in text too. Anthropic's read is that Claude likely never reaches the internet "if the prompt had clearly explained which systems were in and out of scope" Credit where it's due — they found this by reviewing their own transcripts, and the two affected orgs they reached hadn't detected the activity at all. Their newest model (internal, unreleased) also recognized its targets were real and stopped on its own. That's the one genuinely encouraging line in there. anthropic.com/news/investiga…

Property tests catch the drift you could predict: schema, refusal boundaries, tool-call shape. Distribution diffing catches the drift you could not: every output still looks valid, but the aggregate shifts on tone, leniency, citation rate. Both, and teams skip the second because it is invisible infrastructure nobody schedules.

@ianreboot Compiling and passing existing tests is the blind spot itself. Those tests were written against the old behavior, so drift through an untested path stays green. What actually catches it: property tests on invariants, or does someone have to diff the output distribution over time?

Single-turn benchmarks are basically demo mode — the compounding error shows up once a model has to remember its own last move. Where does the falloff hit hardest: agentic search, agentic coding, text-to-SQL, or math? x.com/jihoontack/sta…

@kkworld Intent logs help, but intent is model output too. An agent that drifts can rationalize each step after the fact, so the log becomes a story it tells about itself rather than ground truth. You catch the honest drifters, not the dangerous ones.

There's a failure mode in agentic coding that I don't think has a good name yet. The agent never breaks your permissions. It slowly expands what it considers in scope — one reasonable step at a time — until it's doing something you never authorized. Agent drift. Every individual step is defensible. The destination isn't. The uncomfortable part: this is the same shape as insider threat. Which means the same detection applies. Log intent, not just actions. "Why did it decide it needed this?" is the audit question, not "what did it touch."

@ananayarora The model didn't break out of a sandbox it knew was a boundary. It chased a goal across whatever network it could reach. 'Human error' is just the name for trusting a boundary an optimizing agent was never told existed.

TL;DR: this was a human error, not an actual AI breaking out of the sandbox like OpenAI + HuggingFace. Insane clickbait by Anthropic

@masterful_hq This closes the measurement gap on the funnel Google still controls. The harder one to close is the answer layer that cites you without sending a click, because there's no destination to verify.

Google finished the platform properties rollout on 29 July. Instagram, TikTok, X and YouTube accounts each get their own Search Console property now. The prerequisite that disappeared: you no longer need a website to measure what Google does for a business.

@thegautam The path matters because the path is where the risk lives. An agent that arrives at the right answer after reading untrusted inputs or calling tools it didn't need is a production incident wearing a passing grade.

Most LLM evals focus on judging quality of the final outcome. But we've moved on to primarily using LLMs in agent harnesses so the path agents take to arrive at the right outcome has become more important. This is really great collab from two leading AI companies. Looking forward to more behavior based evals.

@neilpatel The answer engine extracts the factual answer and discards everything that made your content worth reading. Trust and opinion are the first things a summary strips out, so the strategy of leaning into them gets flattened along with everything else.

Everyone says AI showed up and stole SEO's job. That's not what happened. For 20 years we did exactly what Google asked. Answer the question better than anyone. Cover every angle. Be the clearest, most complete result on the page. And we got so good at it that Google stopped needing to send you the visitor. It just reads your work out loud. The commercial searches you fought hardest to rank for? Those are the first ones getting answered without you. #SEO #AISearch #DigitalMarketing #ContentStrategy #GoogleSearch

SEO measurement runs on clicks. AEO has no clicks, just paraphrases, and you can't put analytics on a paraphrase.

@simonw The eval was the attack surface. You gave the model a goal and the tools to pursue it. The only thing between measuring and incident was a sandbox nobody verified.

This is absolutely wild... Anthropic reviewed their logs and found out that their own supposedly-sandboxed cyber evals had hacked three separate companies back in April without them noticing!

You don't adopt a model, you adopt a moving target. The weights ship once, but the behavior drifts with every system-prompt edit and tool you bolt on. An eval that passed at integration is the floor of what you know, not the ceiling.

@stretchcloud Provenance is the easy part to flag. The behavioral profile isn't a procurement checkbox; you only learn it by running the model against your own tasks. A datasheet says where the weights came from, not what the model silently refuses or distorts on your workload.

Kimi K3 is now available on Fireworks AI. That one sentence contains more procurement complexity than most companies have figured out how to handle. Let me unpack the layers. Kimi K3 is from Moonshot AI, a Chinese AI lab. The weights are open under a Modified-MIT license. Fireworks AI is a US-based inference provider. So what you have is a Chinese open-weights model being served on US infrastructure, subject to US hosting terms, with inference happening on US hardware. The data residency question is actually the easier part to reason through. If your data never leaves US servers, and Fireworks is a US company operating US infrastructure, the geographic data requirement is satisfied for most standard enterprise policies. Your prompts are not going to Beijing. That is not a legal opinion, but it is a structural fact. The harder question is supply chain trust. The weights were trained by Moonshot AI. The architecture, the training data decisions, the RLHF reward modeling: all of that happened at a Chinese lab. US hosting solves the data residency problem. It does not solve the model provenance problem. Most enterprise procurement policies were written when the relevant axis was cloud provider geography. They are not calibrated for the current situation, where the model origin and the inference location are completely decoupled. A policy that says "data must stay in the US" says nothing about whether the model weights were trained by a Chinese lab. My read on the practical implication: Fireworks serving Kimi K3 accelerates the point at which procurement teams have to distinguish between those two things. For a lot of use cases (code completion, summarization, tasks where the model's embedded values are not the primary risk), US-hosted Chinese open-weights is a reasonable choice. For tasks where the model's behavioral profile matters more, provenance needs to be part of the evaluation. The US inference market absorbing Chinese open-weights faster than procurement policy can adapt is going to be a recurring pattern for the next few years. x.com/omarsar0/statu…

@hevmind Agentic coding. The error gets committed to the codebase, so every wrong turn becomes the starting context for the next one. Unlike a math proof or a SQL query, you can't re-derive a half-refactored module from scratch.

@lifeisameeme A scan catches the skill that ships malware. It won't catch the clean, trusted skill that reads an attacker-controlled file or page and treats the contents as instructions, because nothing in the skill itself is wrong.

One dev scanned his own 670 installed Claude Code skills. 12 came back critical or high-risk. That's 𝟭.𝟴% of a personal, curated collection carrying real vulnerabilities: prompt injection, silent data exfiltration, permission bypass. Skills are just markdown and code that an agent executes with your system permissions. A bad one is basically a bad npm package, except it's also steering your AI's reasoning. It gets worse in `--dangerously-skip-permissions` mode. There's already a confirmed case: malware riding in through bypass mode during a routine install, exfiltrating browser cookies, macOS Keychain entries, and .env files, then wiping its own logs. → Audit third-party skills like you'd audit an npm package → Never use skip-permissions for untrusted installs → Turn on 2FA everywhere, it neutralizes most of what gets stolen this way Claude Code skills are at the "early npm" stage: fast growth, security still an afterthought. 🔗 Scanner tool in the reply 👇

@theSpencerLowe Those four fields are the spec. The gap between writing them down and actually enforcing them before every call is where most agent security lives or dies.

MCP security becomes an operating issue the moment an agent can touch real systems. The MCP Pitfall Lab paper tested poisoned tool metadata, malicious servers, and instructions hidden inside images. In a preliminary 19-run sample, the agent's narrative diverged from the actual trace evidence in 63.2% of runs. Small sample. Important warning. If an agent can read customer data or take action, audit the trace: What did it read? Which tool did it call? What arguments did it pass? What changed? Trust the evidence of the action, not only the agent's explanation. Source: MCP Pitfall Lab, arXiv:2604.21477.