Katharina Deckenbach

Work with @HaritzPuerto @jonasgeiping @sahar_abdelnabi from @ELLISInst_Tue and @MPI_IS and funded by @AISecurityInst 💻 Paper + code + models: compass-group-tue.github.io/arxiv2026_eval… #AISafety #LLMs #Evaluation #Alignment #MachineLearning 8/8

Our recommendations for safety benchmarks: - Treat evaluation protocols as held-out, not just instances - Consider filtering evaluation methodology docs from pretraining - Make evals resemble deployment conditions - More work in white-box probes for non-verbalized awareness 7/8

If an LLM knows about how an evaluation looks like, can it use it to increase its performance in safety benchmarks? Our new preprint answers this. Excited to share: Models That Know How Evaluations Are Designed Score Safer 🧵 1/8 compass-group-tue.github.io/arxiv2026_eval…