 |
International Society of Science and Applied Technologies |
| A New Interpretation of the Self-Attention Mechanism in Transformers | ||||
| Author |
Maoyuan Zhou
|
|||
| Co-Author(s) |
Xinyu Hu; Siqi Liu; Sheng Fu; Yonggang Ji
|
|||
| Abstract | This study presents a novel perspective on interpreting
the query and key matrices in Transformers. Traditional interpretations
often treat these matrices as independently defined,
lacking a robust mathematical foundation. This limitation introduces
ambiguity in understanding feature relationships and complicates
the model’s decision-making process. In contrast, we
propose an interpretation based on a bilinear model that directly
links the definitions of the query and key matrices to the correlations
between input features. This framework clarifies the mutual
influence of features, making the weight computation process
more intuitive. By enhancing the model’s interpretability, our
approach offers a clearer insight into its behavior and decisionmaking
logic.
|
|||
| Keywords | bilinear model, query matrix, key matrix | |||
| Article #: RQD2026-172 | ||||
Proceedings of 31st ISSAT International Conference on Reliability & Quality in Design |