Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidth
Transformers have become the foundational architecture for a broad spectrum of sequence modeling applications, underpinning state-of-the-art systems in natural language processing, vision, and beyond. However, their theoretical limitations in discrete reasoning tasks, such as arithmetic, logical inference, and algorithmic composition,...