Self-Interference-Aware AS-Assisted Tri-Hybrid Beamforming for Full-Duplex Massive MIMO

This paper proposes a tri-hybrid beamforming (tri-HBF) scheme with antenna-selection (AS)-based reconfigurable sub-arrays for full-duplex (FD) massive multiple-input multiple-output (mMIMO) systems. A sub-connected HBF architecture is adopted, where AS is performed in a group-wise manner to avoid excessive switch-network and routing complexity. An alternating optimization (AO) algorithm is developed to jointly optimize the i) active antenna subsets considering a self-interference (SI)-aware utility, ii) analog beamformers through projected gradient ascent (PGA), iii) digital precoders/combiners via SI-aware regularized zero-forcing (RZF) and minimum mean-square error (MMSE) updates, and iv) DL/UL power allocation by successive convex approximation (SCA). To capture realistic electromagnetic coupling in FD mMIMO operation, experimental SI channels based on an 8x8 Tx-8x8 Rx FD array prototype are incorporated into the study. The proposed AS-aided tri-HBF optimization scheme exhibits robust convergence across various base station configurations and effectively balances desired-signal enhancement, SI mitigation, and multi-user interference suppression in FD mMIMO operation. Illustrative results show that selective activation can outperform full-array activation, achieving a 21.3% higher average sum-rate and a more consistent performance across user realizations, with power-efficiency benefits by reducing the active paths. A comprehensive study is conducted to characterize how the number of activated antennas affects the achievable rate, user-channel coherence, and SI suppression gain. Compared with various selection baselines, it achieves a 45.1% improvement in average sum-rate, with average DL and UL rate gains of 36.9% and 82.9%, respectively. In addition, beam-level isolation better than 63 dB is achieved, further confirming the effectiveness of the proposed SI-aware design.