2026-07-14 14:01 UTC · cs.AI · cs.AI, cs.CL, cs.CV, cs.MM

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

Kaiwen Zheng, Junchen Fu, Wenhao Deng, Hu Han, Joemon M. Jose, Xuri Ge

Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc. However, these performance improvements are often accompanied by an increase in model parameter size (e.g, at least 7B), which simultaneously incurs high computational costs and reduces inference efficiency, thereby hindering real-time deployment on resource-constrained platforms such as robots and mobile devices. This raises a fundamental question: do we really need the multimodal MER model larger than 1B parameters for high-quality MER? In this paper, we challenge the assumption that larger models are inherently necessary and proposes a lightweight MER framework (called Light-MER), which achieves better and faster multimodal sentiment understanding and recognition through knowledge distillation. It can transfer knowledge from a strong, large-scale teacher model to a lightweight sub-billion-parameter student model, aiming to preserve rich multimodal emotion reasoning and recognition while substantially improving deployment efficiency. Specifically, we introduce two new optimization strategies to enhance knowledge transfer: (1) a new optimal transport loss that combines Sliced Wasserstein Distance with hidden-state alignment, and (2) a new multi-reward optimization strategy based on GRPO that balances MER performance and efficiency, aimed at further enhancing the learning capabilities of student models. Extensive experiments on nine benchmark datasets demonstrate that Light-MER achieves state-of-the-art performance while significantly improving inference efficiency. This highlights the strong potential of small multimodal emotion language models for future research. Code is available at https://github.com/GAIR-Lab/Light-MER.

arXiv abstract PDF

Summary
Light-MER distills an 8B multimodal emotion model to a 0.6B student using Sliced Wasserstein hidden-state alignment and multi-reward GRPO to enable edge deployment.

Mathematical/empirical assessment
I do not buy this yet. The claim that Eq. (5) preserves latent geometry relies on sorting 1D projections, but the authors ignore how the frozen projection in Eq. (4) might bottleneck the 4096-to-1024 dimensional mapping. While Table 1 shows Light-MER matching the 8B teacher on nine benchmarks, Table 2 reveals the student still requires 2.54 GB peak memory. This severely undermines the core premise of deploying on resource-constrained mobile devices.

Strengths
Figure 3 convincingly demonstrates that highly peaked output distributions render standard KL divergence ineffective, providing strong empirical justification for shifting to hidden-state alignment. The complexity reduction to O(Rn log n) in Eq. (5) is also theoretically sound compared to Sinkhorn Distances: Lightspeed Computation of Optimal Transport.

Concerns
The composite reward in Eq. (8) relies on arbitrary hand-tuned weights without any ablation study to prove their necessity. Furthermore, the gating mechanism in Eq. (2) and Eq. (3) lacks theoretical grounding for why a simple MLP is sufficient for cross-modal affective fusion. Additionally, the inference profiling directly contradicts the initial motivation.

Reviewer sketch
Teacher: 9.00B params -> 20.04 GB memory
Student: 0.6B params  ->  2.54 GB memory

Final decision
Weak reject

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

Comments