LLM Evals Workshop @NeurIPS

We are happy to announce our @NeurIPSConf workshop on LLM evaluations! Mastering LLM evaluation is no longer optional -- it's fundamental to building reliable models. We'll tackle the field's most pressing evaluation challenges. For details: sites.google.com/corp/view/llm-…. 1/3

It has been a super fun day @LLM_eval workshop @NeurIPSConf with amazing talks, posters, and an engaging panel discussion! @dawnsongtweets @natolambert @orf_bnw @sanmikoyejo @abeirami @hamishivi @MariusHobbhahn @beyzaermis @Diyi_Yang @attaluri_nithya @RishiBommasani @YangjunR

Our next talk @LLM_eval workshop is by @sanmikoyejo! Upper Level Room 2 @NeurIPSConf

“Good researchers obsess over evals” by @natolambert @LLM_eval workshop!

Bringing the hot take culture to NeurIPS - great talk @orf_bnw!!

@dawnsongtweets is giving a talk on agentic evals @LLM_eval workshop!

@LLM_eval workshop has started with Orhan Firat’s talk at Upper Level Room 2. @NeurIPSConf

Good researchers obsess over evals The story of Olmo 3 (post-training), told through evals NeurIPS Talk tomorrow. Upper Level Room 2, 10:35AM.

I’ll be @NeurIPSConf all week and would love to connect on LLM data, evaluation, benchmarking, and scaling laws. If you’re working on related problems, feel free to reach out. PS: Don’t miss our one-of-a-kind workshop on LLM evaluation: sites.google.com/view/llm-eval-…

See you in San Diego on December 7th!

- "The Measure of All Measures: Quantifying LLM Benchmark Quality" -- Jihan Yao, Peter Jin, Ke Bao, Qiaolin Yu et al. openreview.net/forum?id=HpnGm…

🚀 We are thrilled to announce that the LLM Eval Workshop @NeurIPSConf received 244 excellent submissions! 188 papers will be presented in poster sessions, and 5 exceptional works have been selected for oral talks. Check out the accepted papers: sites.google.com/view/llm-eval-… 🧵👇

I will present my #EMNLP2025 paper at the #NeurIPS2025 LLM Eval Workshop @LLM_eval (Dec. 7th 11:15 - 12:15Poster Session 2). If you are interested in reliable LLM-as-a-judge, please come say hi! ☕️ #AI #LLM #LLMJudge #LLMEvaluation #ConformalPrediction

(5/5) Read the paper at: arxiv.org/abs/2510.08616 Looking forward to discussing more at NeurIPS in San Diego!

(1/5) My work, “LLMs Show Surface-Form Brittleness Under Paraphrase Stress Tests”, has been accepted for a contributed talk at @NeurIPSConf 2025 Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling workshop @LLM_eval #NeurIPS #LLM #Evaluation #Robustness #AI #ML

Sketched on a few Parisian summer nights with a friend, @ChrisInterno . If you care about (causal) identification in a semi-synthetic future, we’d value your read and critique. Preprint: arxiv.org/pdf/2509.17999 Accepted at @LLM_eval workshop @NeurIPSConf

The Narcissus Hypothesis: --Recursive training on semi-synthetic corpora enforcing human alignment induces a Social Desirability Bias: world-models (Narcissus) aim to please rather than represent, polluting data lakes and charming us (Echo) into hanging on their every word.