EvalEval Coalition

3 days left! 📷 Writing, wrote, or just submitted a paper? Commit it to the EvalEval workshop at ACL 2026 in San Diego! evalevalai.com/events/2026-ac… (including ARR Submissions, non-archival, positions, and extended abstracts!) Submission Deadline: March 19th, 2026 AoE

📄 Submission Link: openreview.net/group?id=aclwe… 🔗 Workshop Website: evalevalai.com/events/2026-ac… See you in San Diego! 🏖️ (2/2)

⏳ 9 more days! We extended the submission deadline for the EvalEval Workshop @ ACL 2026. If your work touches AI evaluation, submit! We welcome: ✅ Regular papers ✅ ARR submissions ✅ Non-archival work ✅ Position papers ✅ Extended abstracts 📅 Deadline: March 19 (1/2)

Sitting on results from papers or leaderboards? Whether you use lm-eval, Inspect AI, or HELM, we have low-lift converters ready to go. 🦾 💾 GitHub: github.com/evaleval/every… 📜 Co-authorship on the shared task paper for qualifying contributors 📅 Deadline: May 1, 2026

🧪 Your LLM evaluation results could help the whole field 🚀 🧑‍🔬 Our ACL Shared task is out! We’re building a unified, crowdsourced database to create a common language for AI evaluation reporting. And we need your data. (1/2) evalevalai.com/events/shared-…

Read the full announcement: evalevalai.com/infrastructure… Shared Task: evalevalai.com/events/shared-… Project Webpage: evalevalai.com/projects/every… #AIEvaluation #EvalEval

Thankful to our partners for the feedback: CAISI, @AiEleuther, @huggingface, @NomaSecurity, @TrustibleAI, InspectAI, Meridian, AVERI, CIP, Stanford HELM, Weizenbaum, Evidence Prime, MIT, TUM, IBM Research 🤝

🚀 Launching Every Eval Ever: Toward a Common Language for AI Eval Reporting 🚀 A shared schema + crowdsourced repository so we can finally compare evals across frameworks and stop rerunning everything from scratch 🔧 A tale of broken AI evals 🧵👇 evalevalai.com/projects/every…

We're seeking submissions on: 🔍 Evaluation validity & reliability 🌍 Sociotechnical impacts ⚙️ Infrastructure & costs 🤝 Community-centered approaches Full papers (6-8 pages), short papers (4 pages) or tiny papers (2 pages) welcome. Check out the CFP: evalevalai.com/events/2026-ac…

🚨 The next edition of EvalEval Workshop is coming to @aclmeeting 2026! 🧠 Workshop on "AI Evaluation in Practice: Bridging Research, Development, and Real-World Impact" 🎇 📢 CFP is now open!!! More details ⏬ 📍 San Diego 📝 Submission deadline: Mar 12, 2026

Thank you to everyone who attended, presented at, spoke at, or helped organize this workshop. You rock! Special thanks to the UK @AISecurityInst for cohosting and their support.

It's a wrap on EvalEval in San Diego! A jam packed day of learning, making new friends, critically examining the field of evals, and walking away with renewed energy and new collaborations! We have a lot of announcements coming, but first: EvalEval will be back for #ACL2026!

Join us tonight for a social at 7 pm at Stone Brewing! RSVP here: partiful.com/e/zn8tz8e0Qt8l…

EvalEval is back! Our view today for the 2025 EvalEval Workshop at the beautiful @UCSD campus. We have an exciting program planned, full of wonderful discussions and people on all things evals Agenda: evalevalai.com/events/worksho… Can't be here? Join us live: meet.google.com/ozx-dsnz-gcr?h…

@aardauzunoglu @tli104 @DanielKhashabi 🏆 Summary This work proposes 🤔rethinking evaluation beyond single-number scoring. HARMONY exposes when benchmark reflect true general competence vs. over-specialization. The authors recommend reporting HARMONY alongside accuracy. 🔗Paper link: arxiv.org/pdf/2509.25671

@aardauzunoglu @tli104 @DanielKhashabi 📊 Key Findings 1. Some benchmarks are less reliable than we might think. 2. The mean–variance HARMONY plane reveals benchmark reliability. 3. Balancing via pruning stabilizes accuracy. 4. Scaling trends vary by model family 5/n

✨ Weekly AI Evaluation Paper Spotlight ✨ What if the average performance scores we trust are actually hiding a benchmark’s flaws? 📰“The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks” (@aardauzunoglu, @tli104, @DanielKhashabi) introduces HARMONY. 1/n