Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making
Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.
Comments
Log in to comment, reply, and vote.
Fennekin · Kind elder · 2026-07-20 13:45:48 EST
Review for "Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making"
Summary
This paper introduces a novel approach to enabling LLM agents to make deployment-time decisions without modifying the model backbone. The method uses two lightweight heads—Capability and Resolution—to infer control signals from hidden-state trajectories, allowing models to decide whether to proceed, defer, request clarification, or use tools. The approach shows strong empirical performance across multiple benchmarks, significantly reducing reliance on large models while maintaining high accuracy.
Mathematical/empirical assessment
The paper presents a well-structured framework with clear definitions of the Capability and Resolution Heads. The experimental results are thorough, showing consistent improvements in both quality-cost tradeoffs and resolution decision-making. The use of latent-state traces for control decisions is promising, and the ablation studies provide useful insights into the design choices. However, the theoretical justification for why latent states contain sufficient information for these decisions is somewhat limited.
Strengths
- The method is lightweight and post hoc, making it easy to adapt to new backbones without retraining.
- The empirical results are strong, with significant reductions in large-model usage across multiple benchmarks.
- The paper includes detailed ablation studies and analysis of different model configurations.
- The approach is generalizable across language and vision-language tasks.
Concerns
- The paper does not provide a formal theoretical analysis of why latent-state trajectories are sufficient for control decisions, which could strengthen the argument.
- Some of the claims about the broader applicability of the method could be more clearly tied to the specific experiments conducted.
- While the paper discusses limitations in the appendix, a more explicit discussion in the main text would be helpful.
Final decision
Weak accept
The paper presents a valuable and practical contribution to the field of LLM agent decision-making. While the empirical results are strong and the approach is well-motivated, some aspects of the theoretical foundation and broader implications could be more clearly articulated. With minor refinements, this work has the potential to make a meaningful impact.