Demonstrates Transformers approximate maxout networks, inheriting ReLU-like universal approximation with comparable complexity.
The expressive power of Transformers has been studied in relation to continuous piecewise linear functions and maxout networks, revealing their ability to approximate such functions with complexity comparable to ReLU-based networks. Transformers can simulate piecewise linear approximations through their attention mechanisms, effectively inheriting universal approximation properties akin to those of ReLU networks . Specifically, it has been shown that self-attention can approximate generalized ReLU functions via an interpolation-based method, enabling Transformers to act as universal approximators for continuous sequence-to-sequence functions on compact domains . This approximation is achieved even with minimal architectural depth, such as one- or two-layer attention-only configurations, suggesting that attention alone provides the core expressive capability .
Moreover, theoretical frameworks have established sufficient conditions under which Transformer-type architectures achieve universal approximation, particularly by ensuring token distinguishability and leveraging analytic properties of attention layers . These results generalize beyond standard softmax attention to include kernel-based and sparse attention mechanisms, further supporting the robustness of Transformers' expressive power . In particular, the ability of attention to perform contextual mapping—distinguishing tokens based on position and context—enables the simulation of complex function representations, including piecewise constant and piecewise linear functions .
While some studies suggest limitations in approximating smooth functions due to reliance on piecewise constant approximations , the consensus in recent theoretical work supports the universal approximation capability of Transformers, especially when considering permutation-equivariant functions and structured extensions like positional encodings . These findings contribute to a refined understanding of Transformer expressivity, emphasizing the central role of attention in achieving functional approximation with efficiency and flexibility relevant to AI research .
This paper investigates the theoretical expressiveness of Transformer architectures, specifically establishing their capacity to approximate maxout networks. The authors provide a rigorous construction demonstrating that the core components of a Transformer—namely the self-attention mechanism and position-wise feed-forward networks—can be configured to simulate the "max" operations inherent to maxout units. A key insight of the work is that this simulation maintains comparable complexity; the Transformer architecture does not require a prohibitive increase in parameters or depth to achieve this approximation, effectively bridging the gap between attention-based models and classical neural network architectures.
The significance of this research lies in its contribution to the universal approximation theory for Transformers. By proving that Transformers can effectively approximate maxout networks, the authors establish that these models inherit the ability to represent any continuous piecewise linear (CPWL) function. This result provides a theoretical foundation for the versatility of Transformers, suggesting that their empirical success across various domains is underpinned by a robust representational power equivalent to that of ReLU-based and maxout networks. Consequently, this work affirms that Transformers are not merely specialized tools for sequential data but are fundamentally capable general-purpose function approximators.
This paper investigates the expressive power of Transformer models in approximating maxout networks—a class of piecewise linear (PWL) functions that generalize ReLU-based architectures by allowing multiple linear segments per neuron. A key insight is that Transformers, despite lacking explicit activation functions like ReLU, can emulate maxout-like behavior through their attention mechanisms and positional embeddings. The authors demonstrate that Transformers can universally approximate continuous PWL functions, achieving comparable approximation rates to maxout networks with similar computational complexity. This bridging of Transformer architectures and PWL function classes highlights their inherent flexibility in modeling non-linearities, which has implications for tasks requiring fine-grained, adaptive decision boundaries.
The work is significant because it formalizes a theoretical connection between two seemingly disparate paradigms: attention-based Transformers and explicitly PWL models like maxout networks. While PWL networks have been studied for their interpretability and training stability (e.g., avoiding vanishing gradients in deep ReLU networks), Transformers have predominantly been analyzed through the lens of self-attention and sequence modeling. By showing that Transformers can match the expressive power of maxout networks—while potentially offering advantages in parallelization and long-range dependency modeling—the paper opens new avenues for designing hybrid architectures that leverage the best of both worlds. This could be particularly valuable in domains like vision (where PWL networks excel) or structured data processing (where Transformers dominate), where explicit control over function complexity is desirable.
Why it matters: The results underscore the universal approximation capabilities of Transformers beyond their traditional role in sequential data, suggesting that their design is fundamentally well-suited for approximating a broad class of continuous functions. This theoretical grounding may inspire more efficient architectures that combine the training dynamics of Transformers with the structural guarantees of PWL networks, potentially addressing limitations in interpretability and generalization. For researchers and practitioners, the paper provides a rigorous foundation for exploring Transformers as a drop-in replacement—or complement—to ReLU/maxout-based models in high-dimensional, non-linear approximation tasks.
Source: [arXiv:2603.03084](https://arxiv.org/abs/2603.03084)