Watch Your Step: Adam’s Margin Maximization Bias Depends on the Learning Rate Schedule

Earlier work showed an \(\ell_\infty\)-margin bias for Adam under strongly decaying schedules. This paper studies the more common regime of constant and weakly decaying rates. For \(\alpha>\frac{1}{L}-\frac{1}{2}\) there is no non-trivial, locally Lipschitz, convex implicit regulariser on deep linear networks, including any squared norm.

Adam instead maximises a trajectory-dependent weighted \(\ell_2\) margin, because its preconditioner converges to a trajectory-dependent vector. The limiting direction therefore depends on initialisation. In the intermediate regime \(\frac{1}{L}-1\le\alpha\le\frac{1}{L}-\frac{1}{2}\) the authors prove an \(\ell_\infty\) bias, conditional on gradient directional convergence to a full-support vector.