cs.LGmath.SPstat.ML

On attention heads and bilinear forms

📅 👥 1 authors arXiv:2609.22990v1

Authors

Abstract

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.

Paper Information

arXiv ID:
2609.22990v1
Published:

Related Categories

cs.LGmath.SPstat.ML