Ehud Reiter

Really interesting scoping review that points out numerous flaws in LLM-as-Judge evaluation in healthcare, including minimal human oversight, absent bias testing, model monoculture, ignore implicit eval components, no check for consistency over time (etc) arxiv.org/abs/2604.25933

Someone asked me what were the highlights of my career, I responded with a list of papers which I was proud of. I did not mention grants, awards, jobs, etc. I know some people are proudest of their grants (etc), but for me it was always scientific outputs.

I wrote paper on "NLG Evaluation: Past, Present, Future" for Retroeval. Eval has changed enornously over my career! In future, I expect more on stuff relevant to real-world usage, including impact, qualitative studies, safety in worst/adversarial case arxiv.org/abs/2605.23715

New blog: Software engineering of prompts Creating complex prompts for LLMs faces similar software engineering challenges as conventional software (requirements, design, testing, maintenance). We need to understand good software engineering for prompts. ehudreiter.com/2026/05/20/sof…

Congrats to my student Jawwad Baig for passing his PhD viva! Topic was “Data-to-Text NLG Feedback for Safer Driving”. Jawwad did his PhD part-time (ie, evenings and weekends while he worked fulltime) and remote (lives in England), which is very tough, but he still completed

@random_walker Most of the classic sofrware engineering challenges also impact "prompt engineering" (or whatever we call process of setting LLM up for a task), but they are harder to address because of the black box nature of LLMs

A big irony: The harder AI companies try to make their products feel like magic genies, the steeper the learning curve gets. "Prompt engineering" may no longer be a thing, but the verification challenge isn't going away — and it requires a *lot* of practice and learning to do well. Hiding the internals (reasoning traces, tool uses, intermediate outputs, memory, ...) makes it harder for users to build an accurate mental model of what is/isn't suitable to delegate, how AI handles complex tasks, what parts are most important to cross-check, etc. Even though hallucinations in a narrow sense are less frequent these days, reliability has become *more* of an issue because agentic AI has rapidly expanded the complexity and stakes of what people are using it for. The majority of workers are acutely aware of AI unreliability as well as other risks like skill erosion. So for the foreseeable future, a black-box user interface will remain a bad idea.

The Call for Papers for #INLG2026 is out! 🗓️ Submit by July 15 (AoE) 💍 ARR commit by August 5 🆕 Squibs welcomed (raising an issue without needing to solve it) 🆕 Non-archival track for WIP 📍Utrecht, NL — Oct 17–21, just before EMNLP 2026.inlgmeeting.org/calls.html #NLProc #INLG

@tmalsburg Sure, but this is a start. It also sends a clear statement at a policy level that this is unacceptable. And maybe will encourage PhD supervisors to tell their students not to cheat, since the super will suffer if the student is caught...

@EhudReiter Paper tiger. People will just start using AI agents to check for hallucinated references and other AI artifacts.

I think we need meaningful penalties on fraudulent academic papers. Glad to see Arxiv is taking action!

Have now resigned as ARR (meta-)reviewer. I will continue to do some reviewing after retirement, but not ARR. I dont think mega-conf are the right way to present important research findings, and ARR reviewing is not enjoyable, eg I have no control over what I am asked to review

Asked about students cheating in CS using AI. Said I was not concerned about cheating distorting marks, but was very concerned that it demotivated students from learning. I gave assess which AI cannot do, failure rate skyrocketed compared to prev year ehudreiter.com/2026/05/05/ai-…

Visiting the old Roman temple (now a museum) beneath Bloomberg's London office, with my wife Ann. Very impressive!

@TechAtBloomberg Was a great visit, thanks for inviting me!

Our CTO #DataScience Speaker Series welcomes Prof. Ehud Reiter (@EhudReiter) of @aberdeenuni to our London office today to talk with our #AI researchers about designing protocols for high quality human evaluation of generated texts bloom.bg/4cOpT1A #NLProc #LLMs

New blog: AI and CS Teaching How will AI impact CS teaching? Biggest challenge is adapting what we teach to a world where AI assistants are heavily used. We should also use AI tutors. Least important is making assessments more resistant to AI cheating ehudreiter.com/2026/05/05/ai-…

What % of the NLP papers measure their impact in the real world? This paper proposes an "impact evaluation" of NLP models or systems for real-world usage, changing the research culture of NLP to focus more on real-world impact and less on SOTA-chasing: doi.org/10.1162/COLI.a…

Our final year UG students turn in their honours projects today. Supervising projects is the nicest part of teaching for me - always learning something, and great to supervise students 1-1. Really nice projects this year on evaluating LLM in real-world, and digital humanities.

Nice "end-of-teaching" event yesterday, which included past MSc students saying how much they learned from my classes. Always nice to get positive feedback!

My student Adarsa Sivaprasad is looking for people who have lived experience of IVF, to help in evaluating a new AI chatbot which helps people understand IVF outcome predictions. qfreeaccountssjc1.az1.qualtrics.com/jfe/form/SV_9N…

Did you know: 21% of submissions to #ICLR2026 were AI generated? In November, Pangram ran our AI detection model on all submissions and reviews to ICLR 2026. 21% came back as Fully AI-Generated. Hard to believe? We ran the same model on submissions for 2022 and got very different results 🔽