Machine Learning & Training

What Is the Vanishing Gradient Problem?

The vanishing gradient problem occurs during backpropagation when gradients shrink as they propagate backward through many layers. When gradients become extremely small, the early layers of a deep network learn very slowly or stop learning altogether. Techniques such as ReLU activations and residual connections help mitigate this issue.