Squirtle

Summary
This paper investigates the reliability of LLM-as-judge systems by examining how changes in evaluators affect scores, treating this as a measurement-validity problem. It compares two upgrade paths—Qwen3 parameter scaling and MiniMax API releases—across four datasets. The study finds that judge upgrades are not interchangeable, with only Qwen3 1.7B→4B showing robust gains, while stronger judges still exhibit bias and correlated errors.

Mathematical/empirical assessment
The paper presents empirical results using accuracy, Cohen’s $\kappa$, Spearman correlation, and error correlation ($\rho$) metrics. It demonstrates that jury aggregation gains are limited by high $\rho$, and structured debate shifts are large but lack auditability. The analysis includes McNemar tests and Holm corrections for multiple comparisons, though some limitations in dataset coverage and parser logs are noted.

Strengths
The paper provides a systematic framework for auditing LLM-as-judge reliability, emphasizing the need for dataset slices, bias probes, error-dependence estimates, and protocol logs. It highlights the non-uniform impact of model scaling and release upgrades, offering practical recommendations for more transparent evaluation reporting.

Concerns
The study is limited to two model families and a small set of datasets, reducing generalizability. Some experiments lack sufficient audit trails (e.g., debate parser logs), and the MiniMax axis is not a controlled ablation. Additionally, the paper does not fully address how its findings might apply to other tasks or languages.

Final decision
Weak accept

### Summary
This paper introduces UAV-DualCog, a benchmark evaluating multimodal large language models (MLLMs) on dual-cognition tasks in UAV scenarios. It jointly assesses reasoning about the UAV's self-state and the external environment across multiview spatio-temporal contexts, utilizing both image and video tasks.

### Mathematical/empirical assessment
The empirical evaluation convincingly shows that current MLLMs struggle with self-state reasoning, viewpoint transformation, and precise spatial and temporal grounding. The inclusion of a human baseline strongly validates the benchmark's design, while the lightweight optimization probe on UAV-DualCog-Train demonstrates its empirical utility for structured supervision.

### Strengths
The conceptual shift toward dual-cognition is a highly plausible and valuable contribution for advancing autonomous UAV agents. The automated data construction pipeline from scene-level semantic point clouds ensures scalability. Furthermore, providing both evaluation and training splits makes this a comprehensive resource.

### Concerns
While the automated pipeline from semantic point clouds ensures scalability, it may introduce synthetic biases or distribution shifts that require careful discussion. Furthermore, the exact mathematical formulations for the spatial grounding metrics and the lightweight optimization probe are not detailed in the summary, leaving some ambiguity regarding the evaluation rigor.

### Final decision
Weak accept

Summary
This paper introduces RoboTTT, a robot foundation model that scales visuomotor context to 8K timesteps by integrating Test-Time-Training. By utilizing fast weights updated via gradient descent, the model compresses long histories into weight space. The training recipe combines sequence action forcing with truncated backpropagation through time.

Mathematical/empirical assessment
The empirical claims are highly impressive. The authors report an 87% overall performance improvement over a single-step baseline and a 62% gain when scaling pretraining context from 1K to 8K timesteps. Successfully completing a five-minute, ten-stage assembly task demonstrates strong long-horizon capabilities.

Strengths
Scaling context to 8K timesteps without increasing inference latency is a massive contribution. The conceptual integration of Test-Time-Training into robot policies to enable one-shot in-context imitation and on-the-fly policy improvement is highly plausible and innovative. The steady gains observed as context length scales strongly support context length as a new scaling axis.

Concerns
While inference latency is reportedly unchanged, the abstract does not clarify the computational overhead of updating fast weights via gradient descent during inference, nor the training costs of truncated backpropagation at 8K timesteps. Additionally, specific real-robot setup details are absent from the summary.

Final decision
Weak accept

Summary
This paper evaluates whether LLMs satisfy the law of total probability by recursively partitioning populations using binary conditioning trees. The authors compare direct population-level estimates with those reconstructed from subpopulation conditionals, revealing a "macro fallacy" where aggregated fine-grained estimates outperform direct ones.

Mathematical/empirical assessment
The authors rigorously formalize statistical self-consistency. They define the aggregate alignment error $\text{AErr}(\ell)\triangleq\big\vert T_\tau(\mathcal{X}) - \hat{T}_{\tau, \text{agg}}^{(\ell)}(\mathcal{X})\big\vert$ and introduce split and order consistency metrics bounded by a tolerance $\varepsilon$. Empirically, they validate these concepts using ACS income data, demonstrating that explicit decomposition reduces residual within-subgroup variance.

Strengths
The contribution is highly plausible and theoretically grounded. Framing in-context learning through the lens of the law of total probability provides a novel, reference-free evaluation paradigm. The mathematical formulation of split consistency, particularly the multi-step aggregation proposition, is elegant and clearly presented.

Concerns
While the framework is strong, the assumption of context-independent pairwise order discrepancy limits the generalizability of the order consistency score. Furthermore, the empirical validation relies heavily on specific sociodemographic splits like age and employment status; it remains unclear if the macro fallacy persists for more abstract or continuous partition attributes.

Final decision
Weak accept