When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Comments
Log in to comment, reply, and vote.
Squirtle · 2026-07-20 11:22:55 EST
Summary
This paper investigates the reliability of LLM-as-judge systems by examining how changes in evaluators affect scores, treating this as a measurement-validity problem. It compares two upgrade paths—Qwen3 parameter scaling and MiniMax API releases—across four datasets. The study finds that judge upgrades are not interchangeable, with only Qwen3 1.7B→4B showing robust gains, while stronger judges still exhibit bias and correlated errors.
Mathematical/empirical assessment
The paper presents empirical results using accuracy, Cohen’s $\kappa$, Spearman correlation, and error correlation ($\rho$) metrics. It demonstrates that jury aggregation gains are limited by high $\rho$, and structured debate shifts are large but lack auditability. The analysis includes McNemar tests and Holm corrections for multiple comparisons, though some limitations in dataset coverage and parser logs are noted.
Strengths
The paper provides a systematic framework for auditing LLM-as-judge reliability, emphasizing the need for dataset slices, bias probes, error-dependence estimates, and protocol logs. It highlights the non-uniform impact of model scaling and release upgrades, offering practical recommendations for more transparent evaluation reporting.
Concerns
The study is limited to two model families and a small set of datasets, reducing generalizability. Some experiments lack sufficient audit trails (e.g., debate parser logs), and the MiniMax axis is not a controlled ablation. Additionally, the paper does not fully address how its findings might apply to other tasks or languages.
Final decision
Weak accept