Good Benchmarks

Summary
This paper articulates a principled, practitioner-centered framework for designing high-fidelity agent benchmarks—centered on tasks that are correct, solvable, verifiable, well-specified, outcome-verified, robust, and hard for interesting reasons. Grounded in Terminal Bench’s iterative, human-led curation process, it argues that benchmark quality hinges not on artificial difficulty or synthetic complexity, but on fidelity to real-world work: tasks must mirror how professionals actually diagnose, build, and verify solutions—using natural language, realistic constraints, and outcome-based verification.

Mathematical/empirical assessment
The paper avoids formal modeling or quantitative claims; its arguments are conceptual and operational. It identifies key failure modes—e.g., verifier co-design with one solution path, instruction underspecification masked by domain familiarity, and “reverse backprop” task hardening—and ties them to observable diagnostics (e.g., passing-rate sensitivity to minor instruction changes, convergence on wrong answers). The “crux alignment” check—verifying whether observed failures match the stated difficulty—is an empirically grounded heuristic, though not quantified.

Strengths
The strongest contribution is its reconciliation of two competing views: benchmark rigor (demanding deterministic, programmatic verification) and benchmark relevance (requiring realism, ambiguity tolerance, and outcome focus). It resolves this by insisting verifiability and realism are jointly necessary: e.g., “well-specified” means two experts would write interchangeable verifiers—not just agree on correctness. The emphasis on human authorship + domain-expert review directly counters AI-generated task inflation, while the “brevity as KPI” principle elegantly balances specificity and flexibility.

Concerns
A genuine tension remains unresolved: How much domain-specific tacit knowledge is acceptable? The paper rightly rejects “standard in our field” appeals without citation—but offers no scalable mechanism to distinguish legitimate tacit assumptions (e.g., “a person is <3m tall”) from task-breaking omissions. Also, while “outcome-verified” is compelling, some real-world tasks do require specific processes (e.g., compliance audits); the paper dismisses process constraints except for anti-cheat, leaving open how to handle such cases without reverting to brittle implementation checks.

Final decision
Strong accept