Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
📅
👥 3 authors
arXiv:2609.36698v1
Authors
Ene MecoEmadeldeen HamdanA. Enis Cetin
Abstract
To reduce the number of parameters and cache memory requirements of transformers we introduce dual-headed transformers instead of three heads. We studied Walsh-Hadamard Transform (WHT), Discrete Cosine Transform (DCT), Discrete Fourier Transform, filterbank based Shearlet Transform, and Multiplication-Avoiding (MA) operators to construct dual heads. We combine spatial patches and their orthogonal transforms (or Shearlet and MA operators) in a structure similar to the attention block. We obtained better results than triple headed transformers in ImageNet. Extensive simulation examples are presented.
Paper Information
- arXiv ID:
- 2609.36698v1
- Published: