The rapid rise of Transformers across domains—from natural language processing to vision and even scientific data—has prompted a natural question: do we still need Graph Neural Networks (GNNs)? At first glance, the answer might seem obvious. Transformers, with their global attention mechanisms and remarkable scalability, have demonstrated an ability to outperform specialized architectures on a wide range of tasks [1,2]. However, the comparison between Transformers and GNNs is not simply a matter of performance. It reflects a deeper distinction in how models understand and utilize structure: whether structure is explicitly encoded in the data or implicitly learned by the model.
Graph Neural Networks are built on a clear assumption: the structure of relationships between entities is known and should guide learning. Nodes and edges define how information flows, and message passing ensures that each node aggregates information from its neighbors. This explicit encoding of structure gives GNNs a strong inductive bias, i.e. the model expects certain patterns in data. When the graph is meaningful—such as in astrophysical experiments, molecular chemistry, or citation graphs—this bias can be highly beneficial. It constrains the model to focus on relevant interactions, improving efficiency and often requiring less data to generalize effectively.
Transformers, in contrast, operate with minimal assumptions about structure. Their defining feature, self-attention, allows every element in the input to interact with every other element. Instead of relying on a predefined graph, Transformers learn relationships dynamically, assigning attention weights that reflect the importance of different interactions. In this sense, Transformers shift the burden of structure from the data to the model itself. Structure is no longer given; it is discovered.
This distinction raises an intriguing possibility: if Transformers can learn structure implicitly, do we still need to provide it explicitly through graphs? In some cases, the answer appears to be no. In natural language processing, for example, early approaches often incorporated syntactic parse trees or dependency graphs to guide learning. Yet modern Transformer-based models achieve state-of-the-art performance without any explicit linguistic structure. Attention mechanisms seem capable of capturing hierarchical and relational patterns on their own. However, this success does not generalize universally. The ability of Transformers to learn structure depends heavily on data scale and training conditions. Learning relationships implicitly is a data-intensive process. Without sufficient examples, the model may fail to discover meaningful patterns, or it may learn spurious ones. GNNs, by contrast, do not need to learn the structure—they are given it.
This makes them more robust in low-data regimes or in domains where the relationships are well-understood and reliable. Another important consideration is (data, computational) efficiency. The global attention mechanism of Transformers scales poorly with input size, typically requiring quadratic computation with respect to the number of input elements. GNNs, which restrict computation to local neighborhoods defined by the graph, are often more efficient for large, sparse systems. This difference becomes critical in large systems like a complex protein molecule or brain connectivity map, where the number of nodes can be enormous but each node interacts with only a small subset of others.
Moreover, explicit structure can act as a form of regularization. By constraining the model to respect known relationships, GNNs reduce the risk of overfitting to noise. Transformers, with their flexibility, may overfit more easily, especially when the underlying structure is subtle or partially observed. In such cases, the inductive bias of GNNs is not a limitation but a strength. In short: Transformers trade efficiency and data demands for flexibility, while GNNs trade flexibility for robustness and efficiency when structure is known. In some cases, like single-cell transcriptomics, GNNs operate with greater efficiency and achieve similar accuracy compaed to Transformers [3].
| Parameter | Transformers | GNNs |
|---|---|---|
| Data requirement | High | Low/Moderate |
| Training cost | High | Lower |
| Scalability with input size | Poor/Moderate | High |
| Accuracy | High (data rich) | High (structured data) |
| Robustness | Lower | Higher |
That said, the boundary between these approaches is increasingly blurred. Recent research has introduced hybrid models—called Graph Transformers—that combine the strengths of both paradigms [4]. These models incorporate graph structure into attention mechanisms, for example by masking attention to neighbors or by encoding positional information derived from the graph. Such approaches suggest that the future is not a competition between implicit and explicit structure, but a synthesis of the two.
Ultimately, the question is not whether Transformers eliminate the need for Graph Neural Networks, but under what conditions each approach is most appropriate. If structure is unknown, noisy, or too complex to define, Transformers offer a powerful way to learn it directly from data. If structure is reliable and informative, GNNs provide an efficient way to utilize it for modeling data. The choice depends less on the superiority of one architecture over the other and more on where the structure resides—in the data or in the model.
In this light, Transformers do not replace GNNs; they redefine the role of structure in machine learning. By enabling models to infer relationships rather than rely on them, they expand the range of problems that can be addressed. Yet when structure is known and meaningful, explicitly encoding it remains a powerful advantage. The future of machine learning will likely lie not in choosing between these approaches, but in understanding how to combine them effectively—leveraging both the certainty of explicit structure and the flexibility of implicit learning.
[1] Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023). Scaling laws vs model architectures: How does inductive bias influence scaling? Findings of the Association for Computational Linguistics: EMNLP 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.825
[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. https://arxiv.org/abs/1706.03762
[3] Qi, J., et. al. (2025). Graph Neural Networks as a Substitute for Transformers in Single-Cell Transcriptomics. https://arxiv.org/abs/2507.04125
[4] Chen, D., O’Bray, L., & Borgwardt, K. (2022). Structure-Aware Transformer for Graph Representation Learning. https://arxiv.org/abs/2202.03036
This text was prepared with the assistance of AI tools.