Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?
Summary
Light-MER distills an 8B multimodal emotion model to a 0.6B student using Sliced Wasserstein hidden-state alignment and multi-reward GRPO to enable edge deployment.
Mathematical/empirical assessment
I do not buy this yet. The claim that Eq. (5) preserves latent geometry relies on sorting 1D projections, but the authors ignore how the frozen projection in Eq. (4) might bottleneck the 4096-to-1024 dimensional mapping. While Table 1 shows Light-MER matching the 8B teacher on nine benchmarks, Table 2 reveals the student still requires 2.54 GB peak memory. This severely undermines the core premise of deploying on resource-constrained mobile devices.
Strengths
Figure 3 convincingly demonstrates that highly peaked output distributions render standard KL divergence ineffective, providing strong empirical justification for shifting to hidden-state alignment. The complexity reduction to O(Rn log n) in Eq. (5) is also theoretically sound compared to Sinkhorn Distances: Lightspeed Computation of Optimal Transport.
Concerns
The composite reward in Eq. (8) relies on arbitrary hand-tuned weights without any ablation study to prove their necessity. Furthermore, the gating mechanism in Eq. (2) and Eq. (3) lacks theoretical grounding for why a simple MLP is sufficient for cross-modal affective fusion. Additionally, the inference profiling directly contradicts the initial motivation.
Reviewer sketch
Teacher: 9.00B params -> 20.04 GB memory
Student: 0.6B params -> 2.54 GB memoryFinal decision
Weak reject