Rotem Dror

[1/7] Why do frontier LLMs make factual errors? Is it because they never learned the fact… or because they can’t access knowledge they already encoded? In our new paper, we show: The bottleneck is not encoding; it is recall. 🧵👇 Paper: arxiv.org/abs/2602.14080 Many thanks to @_galyo @bd_eyal @zorikgekhman @eran_ofek59358 @GoogleResearch

A strange trend I've noticed at #ACL2025 is that people are hesitant to reach out to papers/"academic products" authors. This is unfortunate for both parties! A simple email can save a lot of time to the sender, but is also one of my favorite kind of email as the receiver!

Tomorrow morning (9AM🌅) I'll be giving a keynote talk at LAW (linguistic annotation workshop) at #ACL2025 on annotations in the era of LLMs - see you there!!🌟🌟

The Alternative Annotator Test (alt-test) is a new statistical procedure proposed in our ACL 2025 paper! 🇦🇹🇦🇹 @DrorRotem @roireichart The goal? To help justify using LLMs over humans. If the LLM passes the test, its annotations can be trusted 😎 arxiv.org/abs/2501.10970

Everyone uses LLMs to annotate data or evaluate models in their research. But how can we convince others (readers, collaborators, reviewers!!!) that LLMs are reliable? 🤖 Here’s a simple (and low-effort) solution: show the LLM is a *comparable alternative annotator* ✅

🔥הסקירות ממשיכות לזרום ל-X🔥 🧵 המאמר היומי של מייק: 25.06.25 The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs מאמר 🇮🇱 תפנית מעניינת מתרחשת בתקופה האחרונה בעולם של הערכת ביצועי מודלים. אנחנו כבר לא שואלים רק עד כמה המודל מצליח במבחן כלשהו, אלא שאלה מהותית יותר: האם ניתן לסמוך על מודל שפה שיחליף מתייג אנושי? זו לא שאלה שמדדים מסורתיים כמו דיוק, F1 או הסכמה בין מתייגים יכולים לענות עליה כראוי. תחת זאת, המאמר שנסקור היום מציג שיטה מבוססת סטטיסטיקה לפתרון ביעה זו. בלב המאמר עומדת קריאה להתרחק ממדדי התאמה שטחיים, ולעבור לנימוקים מבוססי השערות סטטיסטיות וניתוח עלות-תועלת.

Preferences drive modern LLM research and development: from model alignment to evaluation. But how well do we understand them? Excited to share our new preprint: Multi-domain Explainability of Preferences arxiv.org/abs/2505.20088 @roireichart @LiatEinDor 🧵👇 1/11

This is your chance to uncover revolutionary research, forge invaluable connections, and become part of the AI innovation wave. Conference dates: May 25-27, 2025 📷 Information and registration: lnkd.in/dKqAtP6p

Keynote speakers: . Hod Lipson, Mechanical Engineering, Columbia University, USA Prof. Mor Naaman, Information, Cornell Tech, USA Prof. Tanya Berger-Wolf, Computer Science & Engineering, Ohio State University, USA

Join us at the University of Haifa for the HiAI Conference, a dynamic event at the forefront of Artificial Intelligence. Immerse yourself in groundbreaking discoveries, gain exclusive insights into the latest advancements, and navigate the future of AI with leading experts.

@VeredShwartz AI is also old...now it's agents😆

My "AI-backed" washer sorts cycles by frequency of use and suggests the most common cycle. When were "computer programs" rebranded as "AI" and why didn't I get the memo?

@jkkummerfeld @VeredShwartz @LChoshen @GabiStanovsky what load is that? Review load or AC load? or to be more clear - if I'm an AC and I submitted a paper, am I expected to be the AC of at least 4 papers, or, in addition to the AC load, also review 4 papers?

@VeredShwartz @LChoshen @GabiStanovsky ACs are expected to have a load of 4 per submission they are the volunteer for (this was unclear in the form and will be clarified in future cycles)

The ARR policy requiring a qualified author (3 main *CL papers) to review 4 papers per submission makes it impossible for junior faculty to submit papers. I have 9 students, only 2 are qualified reviewers. I will need to review 16 papers in this cycle to avoid desk rejects. 1/2

@VeredShwartz That's amazing Vered! Congrats! 👏 🎉

I'm excited to announce that my nonfiction book, "Lost in Automatic Translation: Navigating Life in English in the Age of Language Technologies", will be published this summer by Cambridge University Press. I can't wait to share it with you! 📖🤖 #fndtn-information" target="_blank" rel="nofollow noopener">cambridge.org/core/books/los…

@miserlis_ @NitCal I believe that if you do it once (i.e. not try all combinations of annotators from the two sets) and the annotators from both groups have annotated separate sections of the data then it should be fine to combine them into one

@DrorRotem @NitCal Thanks! So if items per annotator is small, could we compose them into “pseudo-annotators” to improve power (eg,2 sets of 3 annotators see 20 unique items, so we combine random pairs across sets to make one set of 3 “annotators” that each see 40)? Or would that overestimate var.?

Do you use LLM-as-a-judge or LLM annotations in your research? There’s a growing trend of replacing human annotators with LLMs in research—they're fast, cheap, and require less effort. But can we trust them?🤔 Well, we need a rigorous procedure to answer this. 🚨New preprint👇

@miserlis_ @NitCal Hi @miserlis_ thanks for the great questions! Wilcoxon would work as a non-parametric test, however, when possible, it is always recommended to apply a parametric test. For smaller datasets, we can no longer apply t-test so we propose using its non-parametric counterparts.

@NitCal Ah, thank you! You also mentioned that a Wilcoxon test would work here for smaller data right? (Sorry to pepper you with questions—this is just highly relevant for our ARR rebuttal ;) )

In our new preprint, @roireichart ,@DrorRotem and I propose a new statistical procedure: The Alternative Annotator Test (alt-test) The goal? To help researchers justify using LLMs over humans—if the LLM passes, its annotations can be confidently trusted😎 arxiv.org/abs/2501.10970

Check out our new pre-print on statistically sound methodology to verify the quality of LLM-as-a-judge annotation. @NitCal @DrorRotem

A few months ago I told you that I'm working on something awesome and I need some datasets...well here is the first output of that effort and I sincerely think it's amazing 🤩 check it out! @NitCal @roireichart #llm-as-a-judge, #nlpevaluation