💡 Strong data and eval are essential for real-world progress. In "A Guide to Misinformation Detection Data and Evaluation"—to be presented at KDD 2025—we conduct the largest survey to date in this domain: 75 datasets curated, 45 accessible ones analyzed in depth. Key findings👇

📊Severe spurious correlations and ambiguities affect the majority of datasets in the literature. For example, most datasets have many examples where one can’t conclusively assess veracity at all.

🔍 Categorical labels can underestimate the performance of generative systems by massive amounts: half the errors or more.

🛠️ We also provide practical tools: • CDL-DQA: a toolkit to assess misinformation datasets • CDL-MD: the largest misinformation dataset repo, now on Hugging Face 🤗

🚀 Given these challenges, error analysis and other simple steps could greatly improve the robustness of research in the field. We propose a lightweight Evaluation Quality Assurance (EQA) framework to enable research results that translate more smoothly to real-world impact.

👥 Research by @Cam_Thibault_ JacobTian @gabrielle_p_s TaylorCurtis JamesZhou FlorenceLaflamme LukeGuan ReihanehRabbany @JF_Godbout @KellinPelrine 📄 Read the paper: arxiv.org/abs/2411.05060 🤗 Hugging Face repo: huggingface.co/datasets/Compl… 💻 Code & website: misinfo-datasets.complexdatalab.com