Sampler

Them: “You can’t just throw @DSPyOSS at everything.” Me: throws DSPy at inbox DSPy: batch_with_similar updates, route Google → Engineering, mark Mercury as networking pitch, surface competitive intel. Me: I absolutely can and will.

One of the most pressing questions in our AI Evals course is: "Why can’t I just have an LLM write my LLM pipeline?" The nuanced answer is that you can use LLMs to assist, but not for the whole pipeline. Knowing where to put the LLM in the loop is the hard part. To unpack this, we invited Omar Khattab (@lateinteraction) —creator of DSPy, leading expert on prompt optimization, and now professor at MIT—for a "fireside chat" in the course. He shed light on how he approaches pipeline development in practice. What stood out to us is that Omar spends most of his time on specification—e.g., defining the task clearly, looking at the data, and doing careful error analysis—before letting LLMs automate anything. This up-front rigor is what makes downstream optimization actually work. We've put the recording on YouTube. If you're wondering how Omar thinks about these tradeoffs, this conversation is worth a listen! youtube.com/watch?v=ctyU0z…

@awwstn Think of it like web dev: you don’t only rely on unit tests, you need logging, monitoring, and user analytics. Same thing for agents At the end of the day you’re shipping a product so it still boils down to the same first principles In prod, users are the ultimate eval

Trend I'm following: evals becoming a must-have skill for product builders and AI companies. It's the first new hard skill in a long time that PMs/engineers/founders have had to learn to be successful. The last one was maybe SQL, and Excel? A few examples: @garrytan: "Evals are emerging as the real moat for AI startups." @kevinweil: "Writing evals is going to become a core skill for product managers." @mikeyk: "Writing evals is probably the most important thing right now." @saranormous: "Evals = your new marketing." @gdb: "Evals are surprisingly often all you need." More to come.

@HanchungLee sad but true. many teams want to delegate the work of qa / eval to some external provider but that’s like outsourcing the definition of your customer problem and how you measure it. i’m skeptical an external provider will know and care about your user as much as you do

Just had an aha moment w/ GEPA @DSPyOSS - Gemini 2.5 Flash-Lite (GPT 4.1 reflection LM) - 3 signatures in one compound module - 12 training examples - 10 test examples - 32 minutes of optimizer runtime - $0.90 total cost Results: - Baseline: 68.2% - GEPA-Optimized: 95.3% 🤯

A modular way to build prompts out of universal building blocks.

Prompting GPT-5 is different. In the examples below, optimized prompts: • Cut runtime by 1s • Dropped memory use 3,626 KB → 577 KB • Boosted code quality • Improved robustness (0.32→0.54) • Increased context grounding (0.80→0.95) We built a prompt migrator + optimizer so you don’t need to memorize every GPT-5 best practice.

Good soup for prompt design by @omarsar0 here 👉 promptingguide.ai/techniques

@sampler_ai squad - complaining about the weather & UI.