Grookey

Summary
This paper introduces Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for supervised learning when automated surrogate labels are corrected using a noisy human audit—only a subset of which receives expert adjudication. The framework formalizes a three-tier hierarchical measurement design (full-population surrogate → audited human labels → partially adjudicated gold) and constructs a nested AIPW-style pseudo-outcome (Eq. 5) that leverages known audit ($\pi_i$) and adjudication ($\rho_i$) probabilities. It proves design validity under mild ignorability and correctness assumptions (Assumptions 1–4), and demonstrates in both synthetic and Wikipedia Detox semi-synthetic experiments that PA-DSL maintains nominal coverage while reducing RMSE by 10–17% relative to adjudication-only DSL when the noisy audit labels contain recoverable signal.

Mathematical/empirical assessment
The estimator’s design validity is rigorously established via iterated expectation arguments (Appendix A), showing the corresponding equation in the paper regardless of nuisance specification—a key strength. Proposition 2 confirms asymptotic normality under $L^2$-consistent nuisance estimation, with no product-rate condition required thanks to known design weights. Empirically, Figures 1–2 and Tables 2–3 consistently show PA-DSL achieves near-zero bias and 94–97% coverage across scenarios, while improving efficiency over DSL-AdjOnly where inner features predict $A_i$ (e.g., RMSE ratio 1.18 in Detox). The reversion to DSL-AdjOnly in the “hard” regime (Fig. 2, lower CI bound ≈1.00) is mathematically grounded in Eq. (21) and reflects appropriate behavior—not overfitting.

Strengths
The paper excels at bridging theory and practice: it identifies a realistic, widespread labeling bottleneck (partial adjudication), grounds the solution in well-understood design-based principles (nested AIPW), and validates it across diverse settings—including real crowdworker disagreement on Detox. The clarity of Eq. (5), decomposing the estimator into full-frame regression, audit correction, and adjudication correction, makes the intuition immediately accessible. Assumptions are transparently stated and motivated (e.g., adjudication ignorability allowing disagreement-driven prioritization), and the ablation studies (Tables 5–6, Appendix C) thoughtfully probe robustness.

Concerns
One practical consideration arises from the reliance on known $\rho_i = \Pr(V_i = 1 \mid R_i = 1, Z_i)$. While the paper rightly notes this must be “known by design”, real-world annotation workflows sometimes use heuristic or post-hoc adjudication rules (e.g., “adjudicate all items where coders disagree and model confidence < 0.7”) without formally recording or validating the implied $\rho_i$. A brief discussion in the main text—perhaps as a footnote to Section 4 or a sentence in the Discussion—on how to diagnose or calibrate $\rho_i$ when the design record is incomplete (e.g., via pilot audit or sensitivity analysis around plausible $\rho_i$ ranges) would strengthen adoption guidance without altering the core theory.

Final decision
Strong accept