Momentum

Momentum

Because mini-batch gradient descent makes a parameter update after seeing just a subset of examples, the direction of the update has some variance, and so the path taken by mini-batch gradient descent will "oscillate" toward convergence. Using momentum can reduce these oscillations.

Momentum takes into account the past gradients to smooth out the update. We will store the 'direction' of the previous gradients in the variable v$v$. Formally, this will be the exponentially weighted average of the gradient on previous steps. You can also think of v$v$ as the "velocity" of a ball rolling downhill, building up speed (and momentum) according to the direction of the gradient/slope of the hill.

Figure 3: The red arrows shows the direction taken by one step of mini-batch gradient descent with momentum. The blue points show the direction of the gradient (with respect to the current mini-batch) on each step. Rather than just following the gradient, we let the gradient influence v$v$ and then take a step in the direction of v$v$.

Exercise: Initialize the velocity. The velocity, v$v$, is a python dictionary that needs to be initialized with arrays of zeros. Its keys are the same as those in the grads dictionary, that is: for l=1,...,L$l=1,...,L$:

v["dW" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
v["db" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])

Note that the iterator l starts at 0 in the for loop while the first parameters are v["dW1"] and v["db1"] (that's a "one" on the superscript). This is why we are shifting l to l+1 in the for loop.

Now, implement the parameters update with momentum. The momentum update rule is, for l=1,...,L$l=1,...,L$:

where L is the number of layers, β$\beta$ is the momentum and α$\alpha$ is the learning rate. All parameters should be stored in the parameters dictionary. Note that the iterator l starts at 0 in the for loop while the first parameters are W[1]$W^{[1]}$ and b[1]$b^{[1]}$ (that's a "one" on the superscript). So you will need to shift l to l+1 when coding.

Note that:

• The velocity is initialized with zeros. So the algorithm will take a few iterations to "build up" velocity and start to take bigger steps.
• If β=0$\beta =0$, then this just becomes standard gradient descent without momentum.

How do you choose β$\beta$?

• The larger the momentum β$\beta$ is, the smoother the update because the more we take the past gradients into account. But if β$\beta$ is too big, it could also smooth out the updates too much.
• Common values for β$\beta$ range from 0.8 to 0.999. If you don't feel inclined to tune this, β=0.9$\beta =0.9$ is often a reasonable default.
• Tuning the optimal β$\beta$ for your model might need trying several values to see what works best in term of reducing the value of the cost function J$J$.

What you should remember:

• Momentum takes past gradients into account to smooth out the steps of gradient descent. It can be applied with batch gradient descent, mini-batch gradient descent or stochastic gradient descent.
• You have to tune a momentum hyperparameter β$\beta$ and a learning rate α$\alpha$.