Momentum optimizers are techniques used in gradient descent algorithms to improve convergence speed and stability during the training of machine learning models, particularly deep neural networks. The core idea behind momentum is to accumulate the gradients of past iterations to influence the current update, which helps to smooth out the updates and reduce oscillations.

In a standard gradient descent, the update is solely based on the current gradient. However, with momentum, the update incorporates a fraction of the previous update, weighted by a momentum hyperparameter. This allows the optimizer to maintain a direction of movement, effectively accelerating updates in the relevant direction while dampening changes in directions with high curvature ⁴.

Nesterov momentum is an advanced variant that calculates the gradient at a point slightly ahead in the direction of the momentum, providing a more informed update ¹. This technique is particularly beneficial in scenarios where the objective function has a lot of curvature or noisy gradients, as it can lead to faster convergence ⁴⁵.