Deep Learning Lecture Notes and Experiments

Code samples have links to other repo that I maintain (Advanced Deep Learning with Keras book) or contribute (Keras)

Lecture Notes

0. Course Roadmap

• Deep Learning
• Limits of Deep Learning

1. Background Materials

• Linear Algebra
• Probability
• Numerical Computation

2. Machine Learning Basics

• Concepts, Capacity, Estimators, Linear Regression
• MLE, Bayesian, Other ML Algorithms
• Stochastic Gradient Descent, etc

3. Deep Neural Networks

• Deep Feedforward Neural Networks, Cost, Output, Hidden Units
• Back Propagation
• PyTorch Sample Code
  • Backprop on a single unit MLP
• Keras Sample Code
  • Overview
  • MLP on Linear Model
  • MNIST Sampler
  • MLP on MNIST

4. Regularization

• Keras Sample Code
  • MLP on MNIST no Regularizer
  • MLP on MNIST with L2
  • MLP on MNIST with Dropout
  • MLP on MNIST with Data Augmentation

5. Optimization

6. Convolutional Neural Networks (CNN)

• Keras Sample Code
  • CNN on MNIST
  • CNN on MNIST using Functional API
  • CNN on MNIST Siamese Network

7. Deep Networks

• Keras Sample Code
  • DenseNet
  • ResNet

8. Embeddings

• Keras Sample Code
  • Simple Embedding and Sentiment Analysis
  • Glove Embedding

9. Recurrent Neural Networks, LSTM, GRU

• Keras Sample Code
  • SimpleRNN on MNIST
  • CuDNNLSTM on MNIST

10. AutoEncoders

• Keras Sample Code
  • AutoEncoder
  • Denoising AutoEncoder
  • Colorization AutoEncoder

11. Generative Adversarial Networks (GAN)

• Keras Sample Code
  • DCGAN
  • CGAN

11a. Improved GANs

• Keras Sample Code
  • WGAN
  • LSGAN
  • ACGAN

11b. Disentangled GAN

• InfoGAN

11c. Cross-Domain GAN

• CycleGAN

12. Variational Autoencoder (VAE)

• Keras Sample Code
  • VAE MLP
  • VAE CNN
  • CVAE

13. Deep Reinforcement Learning (DRL)

• Keras Sample Code
  • Q-Learning
  • Q-Learning on FrozenLake-v0
  • DQN and DDQN on CartPole-v0

14. Policy Gradient Methods

• Keras Sample Code
  • REINFORCE. REINFORCE w/ Baseline, Actor-Critic, and A2C

Warning: The following are old experiments that are no longer updated and maintained

Tensorflow Experiments

1. Hello World!
2. Linear Algebra
3. Matrix Decomposition
4. Probability Distributions using TensorBoard
5. Linear Regression by PseudoInverse
6. Linear Regression by Gradient Descent
7. Under Fitting in Linear Regression
8. Optimal Fitting in Linear Regression
9. Over Fitting in Linear Regression
10. Nearest Neighbor
11. Principal Component Analysis
12. Logical Ops by a 2-layer NN (MSE)
13. Logical Ops by a 2-layer NN (Cross Entropy)
14. NotMNIST Deep Feedforward Network: Code for NN and Code for Pickle
15. NotMNIST CNN
16. word2vec
17. Word Prediction/Story Generation using LSTM. Belling the Cat by Aesop Sample Text Story

Keras on Tensorflow Experiments

1. NotMNIST Deep Feedforward Network
2. NotMNIST CNN
3. DCGAN on MNIST