AI Engineer

🗽 AI Engineer returns to NYC! Oct 12-14, a beautiful time to visit. Tickets go on sale soon, but the Call for Speakers is now open! ai.engineer/cfp If you do leading AI work in NYC, especially for AI x Finance, speaker applications for AIE NYC 2026 opened today. For mainstage keynotes, we are actively soliciting the top AI native companies and AI transformations in: - Investment Banking - Commercial Banking - Consumer Banking - Hedge Funds - Private Equity - Financial Data - Insurance - Venture Capital - Agentic Commerce - Accounting - Payments - Compliance - Mid/Back Office If you are NOT in Finance, don't worry, we will still be running our regular AI Engineering and AI Leadership tracks, covering all the latest in coding, generative media, evals, infra, and more. But if you want to meet some of the largest AI native banks and financial institutions in the world, this will be the #1 place to be this fall!

Open models are being asked to do more in smaller footprints. @thorwebdev on the tradeoffs behind open, on-device AI, and how model size, deployment location, and product goals shape what developers can build. Join us in the Token Billionaires Lounge by @cerebras and @aiDotEngineer.

@pamelafox @PrefectIO @jlowin uhh can we recruit you for tshirt designs next year!!

@PrefectIO @jlowin @aiDotEngineer that's such a classy shirt. unfortunately, i am *not* classy, so could I suggest something more like...

I may need a new MCP outfit for all the RC events coming up. Ideas?! for reference, my current outfit is "you down with MCP? yeah, you know me!" dress, complete with the most 90s accessories possible. seen here, passionately talking MCP with @jlowin at @aiDotEngineer

@chiplay @coyle_frankp @emileifrem @steveonjava appreciate the thoughtful contribution!!

@coyle_frankp Great talk! While schema.org and dublin core describe what's on a page, we found that agents are missing a way to understand how a site/app works. My team shipped an open YAML spec for that layer yesterday, with a memory field for the things source code doesn't explain. Would love your take/critique: sightmap.org/blog/sightmap

✨ontologies✨ thanks Prof @coyle_frankp, @emileifrem, Jesús Barrasa, @steveonjava et al for reintroducing these concepts to the AI Engineering world

It was great to speak to @addyosmani at @aiDotEngineer World’s Fair earlier this month.

Most agent skills are AI-written, lightly reviewed, and shipped without evals. That makes failures hard to diagnose. Did the skill fail? Did it trigger at the wrong time? Or was the task beyond the model? Good talk from @_philschmid on the @aiDotEngineer YouTube: • Skills improved benchmark performance by roughly 15% on average • Human-written skills performed best; AI-generated skills sometimes made results worse • Test prompts that should trigger the skill and prompts that should not • Judge the outcome, not whether the agent followed one exact path • Run repeated, isolated trials across the models and agent harnesses people actually use • Keep skills lean and directive, with deeper context stored in reference files • Separate temporary capability skills from durable preference skills • Compare every eval with and without the skill • Retire the skill once the base model can pass without it, but keep the eval as a regression test A good place to start: take your most-used skill, write five positive and five negative test prompts, then run the eval with and without the skill. Video: youtube.com/watch?v=0vphxN…

🆕 First Steps Toward Automated AI Research youtu.be/pWXUkLP9uWM Humanity advances by trying things, finding the shortcomings, and fixing them. @RichardSocher keynotes our first-ever Autoresearch track to show how @Recursive_SI is building a Eureka machine for recursive self improvement... and how far we have yet to go.

SkillBench indexed over 50,000 skills from GitHub and found that almost none of them had evals. @_philschmid, a Staff Engineer at Google DeepMind, builds an entire session out of that gap. "Don't Ship Skills Without Evals" is on @aiDotEngineer's YouTube. What you get is a concrete method for testing a skill: what to assert, how many times to run it, and how to tell when the skill has stopped earning its tokens. - The agents you use and the agents you build are different problems. In your own editor you notice instantly when a skill doesn't fire and you reprompt. Your customers don't know skills exist and will never type a slash command. - Capability skills are temporary, preference skills are durable. Capability skills cover what a model can't do consistently yet. Preference skills encode your team's workflow and style, which no foundation model is going to absorb. - Skills do work, measurably. SkillsBench 1.1 puts the average gain at roughly 15% across about 100 coding and productivity tasks. - AI-written skills can hurt. Human-written skills performed best, and a skill.md over 500 lines is a signal to go back and cut. - The description is where most of it breaks. Around 50% of failures came from the skill not being triggered at all. Write directives about when and how to use it, not passive prose about what it is. - Don't skip the negative cases. Philipp's starting point is ten to twenty prompts: five happy path, five where the skill should stay out of it. - If the steps never change, write a script instead. Skills should set goals and constraints. A fixed step one, step two, step three workflow is burning tokens on something deterministic. - Cut the no-ops. Credit to Matt for this one. AI-generated skills fill up with instructions like "make the implementation easy to read" that change nothing about the agent's behavior and still cost you context. - A real example, in numbers. The Gemini Interactions API skill was built against 117 test cases drawn from real user data, synthetic cases, and feedback, and got valid code generation to almost 90%. The harness was a JSON file of cases plus a small Python script running Gemini CLI, with most checks written as regex asserts. - Test outcomes, not paths. It doesn't matter whether the skill loaded on turn one or turn five. Run isolated workspaces, because coding agents will happily cheat off previous runs, and do three to six trials per case since the whole thing is non-deterministic. - Ablation tells you when to retire. Run the evals with and without the skill loaded. When the model passes without it, drop the skill and keep the eval as a regression test. At Google DeepMind, evals live alongside every skill and run on every change, and a change doesn't merge unless it improves them. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Agents are handling more and more of YOUR money — so much so that AIE NYC will be our first Finance-mainstage conference this Oct! 🆕Live now: the entire WF26 AI in Finance Track! Watch ↓ youtube.com/playlist?list=… @brendanh0gan, Morgan Stanley Lucas Palma, Nubank @YogiNotTheBear, FactSet @Saikallis9012, Fidelity Investments @menkesu, Intuit Divakar Kumar, FlyersSoft Shawn Chan, China Resources Holdings @siddanth2486, Auditoria AI @vinooganesh, Kepler @ShreyaR + @aman2304, Nubank (with Snowglobe)

@vtahowe @andrewtorkbaker we'll post cornelia's session soon!

@andrewtorkbaker @aiDotEngineer So fun to chat with you and Cornelia. Great work going on over at Temporal!

. @vtahowe is one of the best hosts in the biz. Was a blast talking at @aiDotEngineer about what it takes to put agents in production. Don't let your sandboxes become fuel for the fire 🔥

@LeoMehr @realbasilchatha thank you for speaking with us/waiting for our track drop!!

Full talk youtube.com/watch?v=ITMXwI… thank you @aiDotEngineer and @realbasilchatha for organizing!

What are the two most important principles for FDE? 1. always be scoping 2. scale with tokens

After Melbourne sold out weeks early, AI Engineer returns to Australia! Sydney, Dec 7 - 8, Hilton Sydney. Same week as NeurIPS: for one week, the whole AI world's attention is on one city. Don't miss it. Tickets, CFP & sponsorship: webdirections.org/ai-engineer

@LeoMehr @kepler_ai_hq @cognition @FactoryAI scheduling to drip out every hour!

@aiDotEngineer @kepler_ai_hq @cognition @FactoryAI @aiDotEngineer FYI half the videos don’t seem to be uploaded properly?

Live now: our entire Forward Deployed Engineering Track from AI Engineer World's Fair! youtube.com/watch?v=1OMHGs… - @EnoReyes, CTO & Co-Founder, Factory - Pauline Brunet, VP of FDE, Cursor - @vasuman, Founder & CEO, Varick Agents - Jia Wu, Deployed Engineering Lead, Cognition - @natalie_meurer, Head of Agent Engineering, Sierra - Sunny Rekhi, FDE CTO, Decagon - @leomehr, Director of Engineering, Ramp - @zkevinbai, Anthropic, ex-Palantir & founding FDE at Rippling - @vinooganesh, CEO & Co-Founder, Kepler

really grateful to @realbasilchatha for helping me curate a survey of the entire field of Forward Deployed Engineering in one track! From FDE 101 by @zkevinbai (Anthropic, Palantir, Rippling Founding FDE) to head of FDE practices at: - Decagon - Cursor - Sierra - Ramp - Cognition - Factory - Kepler - Varick (tooling for FDEs) loving the track drop model for releasing the 30+ tracks from AIEWF. please give them some love and send to friends exploring FDE career paths. all of the speakers are hiring. naturally.

A concept from 1990s knowledge engineering just got reintroduced to the AI world at the AI Engineer Summit. Ontologies. The pitch: ontologies keep agents honest. Instead of an agent relying on embeddings alone to understand what "customer" or "order" or "policy" means, an active ontology makes those definitions explicit and machine-readable. The agent queries it at inference time. The ontology tells the agent what concepts exist, how they relate, and what rules constrain valid relationships. This is the difference between an agent that guesses what "high-value customer" means from embedding similarity and one that queries a defined taxonomy and gets a precise, auditable answer. The agentic case for ontologies is stronger than the database case ever was. In a database, a human designed the schema and humans know what the tables mean. In an agentic system, the agent is making decisions across thousands of runs. If the model's implicit understanding of a concept drifts between sessions, nobody notices until something breaks at scale. Ontologies give agents a stable, queryable layer of concept definitions that sits outside the model weights. If the model shifts with a fine-tune, the ontology doesn't change. If you need to audit why an agent classified something a certain way, you have a human-readable source of truth. The challenge is maintenance. Schemas rot. The person who designed the ontology moves on. This is the same problem that killed ontology projects in the 2000s. The argument I find compelling: when an agent maintains the ontology as part of its own operation, updating definitions when it encounters edge cases, the maintenance problem changes character. Still hard. Different kind of hard. x.com/aiDotEngineer/…

This is the EXACT planning stack HumanLayer uses to ship agent-written code into complex codebases. They just dropped a 19-minute talk at @aiDotEngineer 4 phases before the agent writes a line, 1 human review gate, 30 minutes up front that saves hours in review. Watch the video and read a relevant guide below.

"Why do we qualify ourselves by the languages we know?" A question from @theo's @aiDotEngineer talk, echoed in a chat I had with @kentcdodds. As LLMs make it easier to code across languages, will language-based communities and identities go away? youtu.be/xUnRQ9vLXxo?si…

Watch my talk at @aiDotEngineer World's Fair with Akele Reed on how we designed mental health AI with a clinician's judgement embedded in every step of the process! youtu.be/O72p-rBb2bA?si…