TREC RAG @ 2026

Does retrieval help RAG or did the LLM already memorize the answer? 🤔 Too often, the overlap between RAG corpora and what LLMs “know” is unclear Better RAG evaluation needs tighter alignment between NLP and IR 📚 That's why for RAG 2026 we are using @nvidia's ClimbMix corpus

🚨Official test topics for TREC RAG 2026 have been released! 🚨 🔎 Find them here: github.com/TREC-RAG/trec-… Time to see what your agents can do 👀

Nuggets, rubrics, side-by-side comparisons, citation support… all useful ways to evaluate RAG, but these pieces are generally scattered across different tools and workflows Introducing RAGDoll 🐈, an end-to-end toolkit for evaluating long-form RAG answers 🧵

Interested in helping us build benchmarks to disentangle sources of LLM knowledge? Don't forget to: 1️⃣ Sign up for TREC: trec.nist.gov/cfp.html 2️⃣ Join our mailing list: groups.google.com/g/trec-rag-202… 3️⃣ Join us on SIGIR Slack!

Start playing with the data at: github.com/castorini/Nano…! NanoKnow paper (accepted at SIGIR 2026 🎉): arxiv.org/pdf/2602.20122

Disentangling retrieval’s role in RAG requires tighter NLP ↔ IR alignment - that’s why we’re using @nvidia's ClimbMix 📚 But we don’t know which questions LLMs trained on ClimbMix "know"...and which they don’t Now we have NanoKnow projections for NQ + SQuAD on ClimbMix 🔥

We want to build a collection where we can disentangle what the LLM knows from what it learns from retrieval...but how can we know what an LLM knows? 🤔 With open LLMs + open corpora built on LLM pretraining data - like ClimbMix - NanoKnow lets us disentangle LLM knowledge 👇

Interested? Don’t forget to: 1️⃣ Sign up for TREC: trec.nist.gov/cfp.html 2️⃣ Join our mailing list: groups.google.com/g/trec-rag-202… 3️⃣ Join us on SIGIR Slack!

🔍 What about retrieval? Worry not! We also provide ☂️ relevance judgements for the TREC RAG 2025 topics over ClimbMix, making it easy to evaluate both retrieval and generation components of your pipeline 👌

🚨 Development topics for RAG 2026 are now out 🥳 You can begin testing your systems before the official evaluation topics are released in early July So, tell your agents it's time to start building and evaluating! 🔥 💻github.com/TREC-RAG/trec-…

Interested? Don’t forget to: 1️⃣ Sign up for TREC: trec.nist.gov/cfp.html 2️⃣ Join our mailing list: groups.google.com/g/trec-rag-202… 3️⃣ Join us on SIGIR Slack!

Which research direction are you 🫵 most excited about? What do you think are unanswered questions in current agentic search evaluations? What do you want us to explore for TREC RAG 2026? We welcome all feedback, and would love to hear from you!

Search is becoming increasingly agentic: systems plan, search, synthesize, cite, and revise. But, how should we study and evaluate these systems? 🤔 In TREC RAG 2026, we want to build a reusable collection for this new reality We’ve aligned on 4 core directions 🧵👇