Active Inference Institute@InferenceActive
GuestStream #067.2 ~ 7/2/2026 at 12 UTC
Andrés Corrada
Who Judges the Judges?
youtube.com/live/t2TgSuYH-…
The problem of verifying experts that are smarter or more knowledgeable than us is ancient. Its modern incantation -- "When we use LLMs-as-Judges, who/what checks them?" -- should be amenable to all the strategies humanity has devised to ameliorate this curse of acquiring knowledge. We apply two of these strategies—ensembling experts and logical inconsistency—to evaluate classifiers when we lack the answer keys for their tests. Disagreeing experts allow us to logically exclude evaluations inconsistent with the counts of their differences. For example, if we all take a multiple-choice exam and disagree, we cannot all be 100% correct. This exclusionary logic for joint evaluations can be formalized as the integer solutions to a system of universally applicable Diophantine equations (axioms of classification). The newly released version of the Open Source NTQR Python package contains exact and random sampling generators for the logically consistent evaluation set given arbitrary number of questions, labels, or classifiers. We will demonstrate its simple use using Jupyter notebooks from the NTQR documentation. Considering the ubiquity of "who judges the judges?", this semantic-free counting logic should have wide applicability in helping ameliorate it. Some examples briefly discussed include: scalable oversight, correlated experts, self-evaluation, and no-knowledge alarms for misaligned classifiers. The talk concludes with the limitations of logic and its inability to answer scientific questions related to the safe monitoring of expert systems.
