Arize AI

Part 2 of our deep dive into how we built Alyx: context windows arize.com/blog/how-to-ma… Once an agent starts running, context becomes the bottleneck fast. Here’s what worked for us: • Middle truncation (keep the start + end, drop the middle) • Memory with retrieval instead of stuffing everything into context • Deduplicating messages and pruning tool outputs • Sub-agents to isolate high-volume tasks Worth a read if you’re building long-running agents.

We’re live at NVIDIA GTC and it’s been packed. If you’re working on LLMs or agents and not fully confident in how they’re behaving in production, stop by booth #3018. We’ll show you how teams are debugging, evaluating, and iterating faster with Arize. Swing by for: • $500 Airbnb gift card giveaway • Owala bottle when you book a demo • Swag like socks, hats, and travel bags 👉 Grab time with us: arize.com/nvidia-gtc-2026 And if you’re around tonight, join our happy hour with CrewAI, Snowflake, and SambaNova. 🍹 RSVP here: luma.com/nvidia-gtc2026… Come say hi. #NVIDIAGTC #AIEngineering #LLM

We just released a new Prompt Tutorial for Arize AX: create, test, and optimize prompts with real data and evaluation. It's easy to tweak a prompt until it "feels" better without knowing if it actually improved. This tutorial walks you through a repeatable create → test → optimize workflow: 💻 Create: System and user message templates, variables, save to Prompt Hub with versioning 🧪 Test: Run on a dataset, add LLM-as-a-Judge evaluators, see how it performs 📈 Optimize: Improve from evaluation feedback, compare versions, validate before production If you're building with LLMs and want a clear path from first prompt to production, this tutorial covers the full workflow in Arize AX. Get started below ⬇️ arize.com/docs/ax/prompt…

Your LLM judge is only as good as the trust you've built in it. 🧪 Tomorrow we're going deeper. Back by popular demand — join Elizabeth Hutton for the next session in our Evals Series, going beyond LLM-as-a-Judge fundamentals and into meta-evaluation: the practice of evaluating your evaluator. In this session you'll learn how to: → Validate whether your judge is measuring the right thing → Compare LLM vs. human annotations on a golden dataset → Calculate precision, recall & F1 to surface real gaps → Run high-temperature stress tests to detect prompt ambiguity → Iteratively refine your eval until it reflects human expectations If you're building evals in production, this one's for you. 📅 Tomorrow, March 18 | 10–11am PT 🔗 Register: lu.ma/yomv4h25

Repo: github.com/Arize-ai/twitt…

One thing that stood out from an experiment we ran recently: agents will climb whatever hill you point them at, but often can’t tell you if it’s the right hill. Good example of this: arize.com/blog/how-we-us… Context: we built a small open-source tool that turns tweets into a newsletter using an LLM, then let a coding agent improve it by iterating against an eval suite. The agent handled the loop extremely well: run the evals, diagnose failures, fix the code, repeat. It quickly cleaned up issues like hallucinated links and structural problems. What was surprising was how little human input shaped the outcome. Across the whole process the guidance was basically: “run the evals,” “that shortcut makes the output worse,” and “measure tweet coverage instead of link counts.” These three decisions ended up shaping several rounds of autonomous work. Agents are great at the iteration. Humans often still have to decide what the objective should be.

Boost your coding agent's performance by 20% — without changing the model. We just published a talk from Laurie Voss on Prompt Learning: a technique we developed at Arize to systematically improve what goes in your CLAUDE.md file (or .cursorrules, or .clinerules — this works for any coding agent). The core idea: your coding agent wakes up with amnesia every session. The rules file is the only memory it has. And most people's is empty. So we asked: what if you could derive the right rules from data instead of guessing? We ran Claude Code against 300 real GitHub issues from SWE-Bench Lite, used an LLM judge to explain every failure in English, then fed those explanations to a meta-prompt that generated better instructions. Rinse, repeat. The results: → Cross-repo: 40% → 45% → Django-specific: +11 percentage points (~20% relative) → A cheaper model with optimized prompts nearly matched the premium model's baseline The rules it generated aren't "follow best practices." They're things like "fix code at the correct hierarchy level so all code paths benefit, not just downstream consumers" — specific, testable, derived from real failure patterns. You don't need the full automation to benefit. Pick 10-20 closed issues from your repo, ask an LLM what rules your coding agent should follow based on those patterns, and put the answer in your rules file. You'll get meaningful improvement from that alone. Everything is open source: github.com/Arize-ai/promp… Full talk: youtube.com/watch?v=8___uP…

Arize AX now supports NVIDIA NIM as a native AI model provider! arize.com/blog/arize-ax-… With NVIDIA NIM natively integrated in Arize AX, teams get NVIDIA’s inference performance and model access, plus Arize’s evaluation and improvement workflows. No custom endpoint configuration. No wrapper code. Simply connect your NIM endpoint under Settings → AI Providers, and your models are immediately available across playground, experiments, and evaluations.

TDD doesn't work for AI apps, instead you need EDD - eval-driven development. @jimbobbennett gave a talk on this very topic at Azure AI Connect, part of the @GlobAICommunity. Check out the video to learn how to do EDD, and for some very 🌶️ takes on AI. youtube.com/live/MC8yKiKa8…

GTC folks: we're hosting a relaxed happy hour just steps from the conference with event-exclusive swag for AI engineers. RSVP: luma.com/nvidia-gtc2026…

Add instrumentation to your #AI apps in 1 terminal command and 1 prompt! @jimbobbennett put together this video to show you how, using our newly released skills for your favorite coding agent. youtu.be/qby0FKv-IfA

Back by popular demand: register for an encore of our LLM-as-a-Judge: Meta Evaluation workshop! luma.com/yomv4h25

Get certified in LLM evaluation in this 🎓 new, free 45-minute course: courses.arize.com/l/pdp/llm-eval…

▶️ Replay ICYMI youtube.com/watch?v=LNgzxN…

Try it out: github.com/Arize-ai/twitt…

We just open sourced a tool that turns recent tweets into an email newsletter (try it out!). Here’s how @seldo used evals and an agent to iteratively improve the app: arize.com/blog/how-we-us… In short, the coding agent tasked with improving the app was excellent at the mechanical loop: read eval results, diagnose the failure, write a fix, run the evals again. It went from 1/5 to 5/5 on hallucinated links in two iterations, methodically fixing the data pipeline and then the prompt. At one point the agent found a clever way to get a “link completeness” evaluator to pass: it added a giant “Tweet Sources” section at the bottom of the newsletter listing every URL. Technically the agent optimized the metric perfectly, it just took a human looking at the result to say: this is awful. At this stage, we’re still in an era where agents optimize – and humans decide what’s worth optimizing.

🎙️ Builders. Practitioners. Researchers. Thought leaders. If you're shaping the future of AI, Observe 26 wants YOU on stage. We're looking for voices working on LLM evaluation, AI agents, observability, and shipping AI to production. Observe 2026 | June 4 | Shack15, San Francisco Apply to speak 👇 docs.google.com/forms/d/e/1FAI… #Observe26 #LLMOps #AIEngineering #AIObservability

Introducing Arize Skills. Every new session, engineers were writing the context before their coding agent could do anything with Arize. So we packaged it. One command gives Cursor, Claude Code, Codex, Windsurf and other coding agents native knowledge of Arize workflows. Instrument, debug, evaluate. Without leaving your editor. npx skills add Arize-ai/arize-skills --skill "*" --yes arize.com/blog/arize-ski…

New York 🏙️: we're hosting a workshop at Betaworks covering a proven way to boost Claude Code performance. RSVP: luma.com/ajy0fdyf

In our next "How It Was Built" workshop, we're peeling back the curtain on the planning architecture, context management challenges, and testing strategies behind Alyx. 🚩RSVP: luma.com/alyx2.0