math.OCcs.AIcs.LGeess.SYstat.ML

Mean-field Token Steering

📅 👥 4 authors arXiv:2611.19163v1

Authors

Abstract

We consider stochastic token steering through a deep stack of transformer layers as a mean-field control problem for an interacting particle system on the unit sphere, where interactions arise from the attention mechanism. In the mean-field limit, the token population evolves via a controlled integro-partial differential equation where the feedback control acts as a corrective vector field guiding the population toward an aggregate configuration of tokens in the representation space. We deduce the minimum effort control to accomplish this task over a fixed time horizon interpreted as normalized layer depth. We show that the resulting controlled dynamics for the token population can be understood as a mean-field Schrödinger bridge adapted to the sphere geometry. Building on this, we propose a novel generalization of the dynamic Sinkhorn recursion to compute the consistent token evolution. Numerical experiments confirm that the proposed approach achieves exact steering of the token population.

Paper Information

arXiv ID:
2611.19163v1
Published:

Related Categories

math.OCcs.AIcs.LGeess.SYstat.ML