Nested hyperedges promote the onset of collective transitions but suppress explosive behavior

Higher-order interactions can induce abrupt collective transitions, yet the microscopic mechanisms controlling macroscopic critical behavior remain unclear. Here we show that nested hyperedges generate a dual effect on dynamical processes: they promote the onset of collective behavior while suppressing the explosive transitions driven by higher-order feedback. To uncover the mechanism, we develop an analytically tractable theory of contagion on hypergraphs that explicitly tracks nestedness between groups of different sizes, allowing us to identify the microscopic mechanism responsible for this dual behavior. By disentangling contagion pathways, we find that nestedness redirects transmission from external links to internal, group-embedded routes -- boosting early activation but making dyadic and triadic channels increasingly redundant. This loss of structural independence quenches the nonlinear amplification required for bistability, progressively smoothing the transition as hyperedges become nested. The phenomenology holds for groups of any size, and is not specific to spreading dynamics but also emerges in higher-order Ising and Kuramoto dynamics. Overall, our results identify nestedness between group interactions as a general structural mechanism governing critical transitions in complex systems.