Ehud Reiter

@DamienTeney Qualitative insights about what LLMs as a group can and cannot do, and problems that arise, are very interesting and valuable. But detailed performance numbers (especially in "horse race" between models) are not interesting if models investigated are obsolete when paper read

@EhudReiter (genuine question, I have no context about this work; the *causes/conditions* for good/bad performance seem the more interestion question, and should be less fickle?)

A student is writing up an experiment which includes a comparison of how well two LLMs do at a task. The community expects such comparisons, but why are they useful, since the LLMs being compared will be obsolete by the time people read this?

PS - If we are making a general claim that LLMs can/cannot do X, then absolutely useful to show this in multiple LLMs. But why do we care that an obsolete version of GPT is better at this task than an obsolete version of Gemini?

@JaydeepBorkar @yanaiela I think it would be easier for junior researchers if the rules were consistently enforced, so that you could learn from existing ACL papers. But enforcement is inconsistent, which means many existing papers break the rules, which is absolutely confusing for newcomers!

@yanaiela @EhudReiter +1. I’m glad someone said this. A lot of us submit to *CL venues because we really like the community. But having so many rules/checklists makes it complicate to navigate through the ARR system (esp. for the junior researchers).

New blog: Please follow the rules for ARR/ACL papers ACL/ARR have rules and guidelines for how papers are written. Unfortunately many authors (and reviewers) ignore these, which makes their papers harder to read and less useful. Please follow the rules! ehudreiter.com/2026/03/16/ple…

@yanaiela I appreciate this, hopefully my blog helps in highlighting some key points

@EhudReiter The amount of rules written for ARR has become so large that it barely even fits within an LLM’s context, let alone a human’s.

Looked at @METR_Evals blogs, some nice material on lessons, challenges, etc in using RCTs to evaluate LLMs. We have had RCTs in medicine for a long time and have built up a lot of knowledge about this, but RCTs in AI are very new, and we need to figure out how to do them well

AI isn't replacing programmers, but it *is* making it harder to survive as a programmer with purely technical skills and no interest or expertise in how those skills translate to business or societal value. Funny thing is, this has always been true—it's just being accelerated a bit due to AI. There's a famous essay by @patio11 from 15 years ago called "Don't Call Yourself A Programmer, And Other Career Advice". kalzumeus.com/2011/10/28/don…

@simoneballoccu To be honest, I think Github repos are often better for providing extra detail, since they contain data and code as well as text. Also authors realise that reviewers probably wont look at repos, so they need to make main paper self-contained

@EhudReiter I'm generally encouraging of long appendices but only when they provide extra detail that is not necessary to assess the goodness of a paper!

In future, I will not read appendices when reviewing ACL papers (ARR guidelines say this is not expected). Appendices used to be a few pages of details for replication (eg hyperparam), but now its common to have 10-20 pages (Ive seen 50 pages). Dont expect me to read this!

Aberdeen CS is hiring a new lecturer, who will be expected to do some teaching in our new Mumbai campus. Closing date is 30 March abdn.ac.uk/jobs/vacancies…

Incredibly cool work! We are not seeing enough manual grading from experts in AI, but its definitely worth it to verify the most important results for top models.

I start my last-ever course today, MSc course on Natural Language Generation. My last lecture (on NLG evaluation) will be on 20 April. Hard to believe...

@Jsevillamol Thanks, Ill email him

@EhudReiter you must know some people who could participate!

Joel worked on everyone's favourite uplift study last year. Highly recommend participating in this!

New blog: Questions from readers of my book A group who is reading my book sent me many questions, some of which we discussed in a call last week. I thought I would share the questions and my responses. ehudreiter.com/2026/03/03/que…

Great to see that my student Jawwad Baig has submitted his PhD! One of my main goals for 2025-26 is to help 6 PhD students submit before I retire. Halfway through the academic year, and three of the six have now submitted, so on track.

My PhD student Adarsa Sivaprasad is looking for people who have lived experience of IVF to help evaluate an AI chatbot which explains IVF outcome predictions. What is involved: 45 min online MS Teams call. Read details and sign up at: tinyurl.com/cc2aepf5

Everyone is talking about the new @NatureMedicine paper (rdcu.be/e4ADv), but I think the real story is being buried. Here is the cold reality: The AI passed the medical boards with flying colors (~95% accuracy). But when real humans actually used it for triage, their accuracy dropped to <35%. They performed worse than the control group who just used Google. Practically, this means benchmarks are not safety tests. We are validating tools in a vacuum (simulations) that collapse in the real world. As oncologists, we know this pattern: surrogate endpoints ≠ survival data. Passing the boards is just a surrogate. Safe patient interaction is the only outcome that matters. Right now, we are optimizing for the test and failing the patient. cc: @EricTopol @pranavrajpurkar #ClinicalValidation #AIhype #PatientSafety #EvidenceBasedMedicine

On holiday in Elche, Spain. Fascinating place, full of palm trees. Below is an old Christian basilica which *may* have originally been an ancient (Roman era) Synagogue.

New blog: Dont ignore omissions! Evaluation of LLMs focuses on accuracy and hallucination. Completeness and omission also important; does the text include all the key information? Omissions are a huge problem in medical NLG, as well other NLG tasks ehudreiter.com/2026/02/11/don…