Mina Doroudi

10/ I wrote a longer piece on how to build an eval framework for GenAI products. open.substack.com/pub/mina994298…

Building GenAI products without evals is basically guesswork. Here are some of my learnings on evals, and where teams get them wrong:

9/ final takeaway: Evals are not just a technical tool. They are how you make product quality visible in probabilistic systems. Without them, you’re guessing. With them, you can build with clarity and rigor.

8/ Evals are powerful, but they don’t capture everything. They usually miss: UX friction latency reliability how the full experience feels So evals are part of the quality system, not the whole thing.

7/ Also: If your evals are green but users are unhappy, trust the users. That usually means your evals are measuring the wrong thing.

6/ A useful process: build a representative dataset manually review outputs and provide feedback cluster feedback into dimensions Add clear definitions under the dimensions then scale with LLM-as-a-judge carefully

5/ Biggest mistake I see: Teams jump to automated judging too early. Before LLM-as-a-judge, you need human judgment. You need to understand: What does “good” look like? What breaks trust? What do users actually notice?

4/ Good evals help you: catch regressions compare models/prompts/approaches measure quality across dimensions make decisions with signal, not vibes

3/ So the real question is not: “Did we ship?” It’s: “Did the product actually get better?” That’s what evals help answer.

2/ In traditional software: spec → build → test → ship In GenAI, that breaks down. Why? Because the system is non-deterministic. It interprets, improvises, and changes with model updates, prompt tweaks, and system changes.

1/ Evals are how you measure whether a GenAI product or agent is producing good outputs. But more importantly: They sit at the center of the feedback loop between model/agent behavior and product improvement.

Following the announcement of Sora’s shutdown, here’s how standalone AI video generation traffic evolved over the past 12 months. Sora traffic includes only its official domains (Sora.com and Sora.ChatGPT.com). Grok Imagine reflects activity on grok.com/imagine.

Iran’s territorial integrity must be maintained. The Iranian people will not accept any future in which Iran is broken into “stans.” Iran has existed as a nation and a civilization for thousands of years. Iranians welcome liberation, but we will lose their goodwill if we pursue any outcome that fragments the country. cc @potus @vp @marcorubio @PeteHegseth @netanyahu

BREAKING: Massive protests erupt in Tehran, over 100,000 Iranians flooding the streets, demanding the fall of the Islamic Republic.

💔 Another 17-year-old. One of his final messages before heading into the streets, deeply touching: "Dad, it’s time to say goodbye, but don’t tell Mom.’ Dad, I can’t take this pain anymore. My throat is tight, full of tears I can’t hold back. Dad, if I’m gone, know this was all for the future. Know that my only dream, everything I ever wished for, was to see Iran free one day.”

@MarioNawfal Iran’s population is young. So many were killed during the iraq war, which was after the 78.

🇮🇷 The airspace is cleared. Brace yourself...

🚨🇮🇷 A MESSAGE FROM SOMEONE WHO JUST GOT OUT OF IRAN A young Iranian guy just crossed from Iran into Turkey and recorded a message for the world. He says people outside Iran have no idea how bad it really is. Not arrests. Not beatings. Killings. He says protesters are being shot straight in the chest. Not warning shots. Not accidents. Deliberate. Over and over. According to him, the death toll people see online is nowhere close to reality. Bodies disappear. Families stay quiet. Fear fills the gaps. People inside can’t speak. People outside aren’t seeing it. That silence is doing a lot of work right now. Source: @visegrad24

🚨🇮🇷 IRANIAN REGIME FIRES LIVE ROUNDS AT UNARMED PROTESTERS: “THEY’RE SHOOTING US!” Heart-wrenching video from Mashhad captures sheer panic: "They're shooting us! They're shooting us! This government's shooting people! They're f*cking shooting us! Please help!" Source: @GhorbaniiNiyak

Internet shutdown continues! So does our concern about the authorities increasing brutality against the protesters! Countries with diplomatic relations with IRI must send a strong signal to the Islamic Republic authorities that shooting at protesters will not be tolerated! #IranProtests