Plans for the week of March 9-13

Dear all, welcome back to FYS5429/9429. We hope the week has started the best possible way for you all.

Last week we wrapped up our discussions of recurrent neural networks (RNNs). Together with standard feed forward neural networks (FFNNs) and (CCNs) convolutional neural networks (covered the first 7 weeks of this course and for several of you a considerable repetition from earlier courses), CCNs, RNNs and FFNNs form the basic building blocks of more advanced deep learning methods. With these ingredients, we are now ready to start with entirely new topics. On Thursday this week we will start with Autoencoders (AEs) with applications to both supervised and unsupervised problems. On this Thursday's lecture we will discuss their mathematics and link (so-called linear AEs) with the PCA=principal component analysis.

AEs will us also serve us in linking (dimensionality reduction) to generative methods like Variational Autoencoders and Diffusion models, to be discussed after the Easter break.

Next week we will discuss non-linear AEs and codes as well. But the focus this week is on the basics of AEs and the link to PCA. The plans is thus

Autoencoders (AEs), Foundations of representation learning and autoencoders

Linear autoencoders and the full connection to Principal Component Analysis (PCA)

Lab as always and discussions of project work

Next week

Foundations of representation learning and autoencoders with code examples

Nonlinear and regularized autoencoders

Probabilistic autoencoders, PPCA, and VAEs

Reading recommendations

Goodfellow et al chapter 14, good expositions of AEs

Rashcka et al. Their chapter 17 contains a brief introduction only.

Deep Learning Tutorial on AEs from Stanford University at http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/Links to an external site.

Building AEs in Keras at https://blog.keras.io/building-autoencoders-in-keras.htmlLinks to an external site.

Introduction to AEs in TensorFlow at https://www.tensorflow.org/tutorials/generative/autoencoderLinks to an external site.

Grosse, University of Toronto, Lecture on AEs at http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/slides/lec20.pdfLinks to an external site.

Bank et al on AEs at https://arxiv.org/abs/2003.05991Links to an external site.

Baldi and Hornik, Neural networks and principal component analysis: Learning from examples without local minima, Neural Networks 2, 53 (1989)

The jupyter-notebook for this week is at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week8/ipynb/week8.ipynbLinks to an external site.

Best wishes to you all,

Morten, Oda and Ruben