Transformer Rank Collapse: How Skips, Norms, and MLP Width Affect Trainability

Katie Everett’s paper argues that residual skips, normalization placement, and feedforward expansion help determine how much gradient rank survives as Transformer depth increases.

TL;DR — The paper argues that Transformer feedforward design choices control how much rank survives across depth at initialization. Skip connections, normalization placement, and width expansion shape the tradeoff between rank preservation, deep composition, and parameter count.

Background: the problem of rank collapse

Deep networks gain expressivity by stacking many transformations. But the same matrix multiplications and nonlinear activations that make layers useful can also reduce rank: the number of independent directions a representation or gradient can carry.

“Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth” studies this problem in Transformer feedforward blocks at initialization. Its focus is not just whether activations or gradients grow or shrink, but whether they retain enough independent directions as depth increases.

In this framing, full rank means many directions survive through the network. Low rank means the signal has collapsed into fewer directions, which can make training brittle or ineffective.

Full analysis is available via x402 micropayment.