Javier Carnerero Cano

🚨 Excited to share two papers accepted to #ACL2026 on long-form factuality in LLMs! - Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation [arxiv.org/abs/2510.07926] - FactCorrector [arxiv.org/abs/2601.11232] @IBMResearch (1/4) 🧵👇

⚖️ Comprehensiveness Metrics [arxiv.org/abs/2510.07926]. We shift the question: Not just "Is this true?" but "What's missing from the context?" We introduce 3 methods (NLI, Q&A, end-to-end) to measure that. LLMs don't just hallucinate, they omit. See you in San Diego! (4/4)

📊 We also introduce VELI5, a new dataset with controlled factual errors + ground-truth fixes. This dataset has already been used to fine-tune state-of-the art factuality guardrails such as Granite Guardian [huggingface.co/ibm-granite/gr…]. (3/4)

🚨 Excited to share two papers accepted to #ACL2026 on long-form factuality in LLMs! - Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation [arxiv.org/abs/2510.07926] - FactCorrector [arxiv.org/abs/2601.11232] @IBMResearch (1/4) 🧵👇

Please stop by if you're interested in evaluating and improving the handling of hallucinations and factual accuracy in LLMs, particularly in real-world knowledge conflicts! 🔗 paper link: openreview.net/pdf?id=5t7DtLw… 🔗 dataset link: huggingface.co/datasets/ibm/W…

I'm in Vancouver attending #NeurIPS2024 this week. Together with my colleagues from @IBMResearch, we'll present our paper "WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia" on Thursday 12th Dec, 11am - 2pm (Poster Session 3 East).

Gowns, grads, good times 🥹 That's a wrap on our 2024 Commemoration Day! Congratulations to all of #OurImperial graduates, we can’t wait to see what you do next 💙

The state of peer-review in ML:

In current AI research, I believe there is significantly more value in understanding/analysis papers as opposed to pure method papers. It won't hurt to slow down on new methods/models and start asking why things work/not work. We need more whys and less hows.

This photo from Carlini’s talk goes incredibly hard. Sometimes it’s difficult to explain to folks who haven’t worked in security/privacy just how challenging it is to build things with robust performance. AI privsec is the “graveyard of papers”. Pic creds to @furongh

"WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia" when given passages containing contradictory facts, models struggle to generate answers reflecting the conflicting nature of the context. (Hou et al, 2024) arxiv.org/pdf/2406.13805

We are THRILLED to have been ranked 2nd in the world and 1st in the UK and Europe in this year's QS @worlduniranking! 🎊🥳 #QSWUR Thank you to everyone in #OurImperial community for making this result possible! 💙

How to be as "smart" as Auto-Regressive LLMs: - memorize lots of problem statements together with recipes on how to solve them. - to solve a new problem, retrieve the recipe whose problem statement superficially matches the new problem. - apply the recipe blindly and declare victory. - do not use basic logic. - do not use common sense to check your solution. - do not use a mental model of the situation as a sanity check. - do not simulate the scenario in your mind using your world model. - when someone tells you your solution is wrong, reply "I'm sorry, you are right" and apply another irrelevant recipe. Knowledge accumulation is not a substitute for actual understanding.

I also do a double take every time someone uses "inference time" to mean "test time". I really dislike this change in terminology, which overloads the historical and more sensible meanings of the word "inference". When and why did it happen?

I'll be in @pintofscience on the 15th May at 7pm to talk about AI security! 🍺🧠 If you're around London, grab a pint at The Roebuck and join us for an overview of how AI can be exposed to attacks (and how we can protect it) 🤖🛡️: pintofscience.co.uk/event/eclipsin… #pint24 #AISecurity

No soporto más el ritmo del mundo. ¿Pa’ qué es que estamos corriendo tanto? ¿Por qué le declaramos la guerra al sosiego y al silencio?

I would rather have questions that can't be answered than answers that can't be questioned.

I don’t know who needs to hear this but your PhD doesn’t need to be perfect. It just needs to be finished.

✨Tickets for #pint24 are now on sale!✨ 🎟Book here 👉 pintofscience.co.uk 👈 🐝🧪👩‍🔬📚🔬👓🧠🐟🫀💫🥼👨‍🔬🔭🧫🦴 🗓13th-15th May 2024 📍40+ locations across the UK 🗣450+ events