Plans for week 37, September 8-12

Dear all, welcome back to FYS-STK3155/4155 and a new exciting week! Our plans this week are

to discuss the family of gradient descent methods which we need to implement in the project (parts c-e, including Lasso regression).

1. Plain gradient descent (constant learning rate), reminder from last week with examples using OLS and Ridge

2. Improving gradient descent with momentum

3. Introducing stochastic gradient descent

4. More advanced updates of the learning rate: ADAgrad, RMSprop and ADAM

Readings and Videos:

1. Recommended: The textbook Goodfellow et al, Deep Learning, contains a good introduction to gradient descent, see sections 4.3-4.5 at https://www.deeplearningbook.org/contents/numerical.html and chapter 8.3-8.5 for stochastic gradient methods at URL::https://www.deeplearningbook.org/contents/optimization.html Note that chapter 8 is linked with a discussion of neural networks and deep learning. However, the basic math and algorithms can be (and are) applied to any other method.

2. Rashcka et al, pages 37-44 and pages 278-283 with focus on linear regression.

3. Video on gradient descent at https://www.youtube.com/watch?v=sDv4f4s2SB8

4. Video on Stochastic gradient descent at https://www.youtube.com/watch?v=vMh0zPT0tLI

The jupyter-notebook file for this  week can be found at https://github.com/CompPhysics/MachineLearning/blob/master/doc/pub/week37/ipynb/week37.ipynb

Best wishes to you all,

Morten et al