Quantitative Generalization Bounds for Multi-Layer Transformers: A Few Long Strings can Beat Many Short Ones
Henry et al.
Joint bounds on statistical and length generalisation for fixed-precision transformers show that longer training sequences can be exponentially more efficient than more samples.
Source published · In the Daily Brief of 29 October 2026
Filed under Learning theory, Large language models
Generalisation in transformers has two axes: in-distribution generalisation from samples to new samples, and length generalisation from short training inputs to longer ones. The two have mostly been studied apart. This paper treats them together for multi-layer transformers with fixed precision.
The bounds depend on the parameter sizes of the models, the quantisation space and the alphabet size. They show how the required number of samples trades off against training length, and in particular that increasing the length of training strings can be exponentially more efficient than increasing their number. Experiments are reported to agree with the theory.