Question

During the Transformer forward pass, why does the Pre-LN configuration typically result in more stable gradient flow during the initial stages of training compared to the Post-LN configuration?

Accepted Answer

In a Transformer, the residual connection is a structure where the input to a layer is added to the output of that layer, mathematically represented as x + Sublayer(x). The Layer Normalization (LN) process standardizes the mean and variance of inputs to keep them within a stable numerical range. In the Post-LN configuration, normalization is applied after the residual addition, meaning the output of the layer is normalized before it is passed to the next residual branch. Because this addition happens before normalization, the magnitude of the values in the residual stream grows significantly as they pass through successive layers. During backpropagation, which is the process of calculating gradients to update model weights, this growth creates large gradients that make the training process unstable and sensitive to the learning rate. In the Pre-LN configuration, normalization is applied to the input of the sublayer before the residual addition. This ensures that the main residual path—the skip connection that carries information directly through the network—remains unscaled and normalized at every step. By keeping the values in the identity branch stable throughout the network, the gradients can flow from the output back to the input layers without encountering the explosive scaling issues seen in Post-LN. Consequently, Pre-LN allows the model to maintain a smoother gradient signal, which facilitates easier convergence during the early stages of training without requiring complex learning rate warm-up strategies.

Home → All Courses → Programming Courses → Large Language Model (LLM) Engineering → Flashcard

During the Transformer forward pass, why does the Pre-LN configuration typically result in more stable gradient flow during the initial stages of training compared to the Post-LN configuration?