\blurb{ This site collects resources to learn Deep Learning in the form of Modules available through the sidebar on the left. As a student, you can walk through the modules at your own pace and interact with others thanks to the associated Discord server. You don’t need any special hardware or software. }
The main goal of the course is to allow students to understand papers, blog posts and codes available online and to adapt them to their projects as soon as possible. In particular, we avoid the use of any high-level neural networks API and focus on the PyTorch library in Python.
The course is divided into sessions (containing possibly several modules), each session requiring a significant amount of coding. At the end of this course, students were able to read very recent papers and reproduce (or even ameliorate) their experiments.
All the code used in this course is available on the GitHub repository dataflowr/notebooks. You will find the solutions to the practicals on this repo! You can fork the repo if you want to run the code locally: GitHub Docs about fork then follow the steps in Module 0. Most of the code will not require a GPU.
Pre-requisites:
- Mathematics: basics of linear algebra, probability, differential calculus and optimization
- Programming: Python. Test your proficiency: quiz
Start right away and train a deep neural network on a GPU with Module 1 - Introduction & General Overview
Be sure to build your own classifier with more dogs and cats in the practicals.
<details>
<summary>Things to remember</summary>
- you do not need to understand everything to run a deep learning model! But the main goal of this course will be to come back to each step done today and understand them...
- to use the dataloader from Pytorch, you need to follow the API (i.e. for classification store your dataset in folders)
- using a pretrained model and modifying it to adapt it to a similar task is easy.
- if you do not understand why we take this loss, that's fine, we'll cover that in Module 3.
- even with a GPU, avoid unnecessary computations!
</details>
- Module 2a - PyTorch tensors
- Module 2b - Automatic differentiation + Practicals
- MLP from scratch start of HW1
- another look at autodiff with dual numbers and Julia
<details>
<summary>Things to remember</summary>
- Pytorch tensors = Numpy on GPU + gradients!
- in deep learning, broadcasting is used everywhere. The rules are the same as for Numpy.
- Automatic differentiation is not only the chain rule! Backpropagation algorithm (or dual numbers) is a clever algorithm to implement automatic differentiation...
</details>
- Module 3 - Loss function for classification
- Module 4 - Optimization for deep learning
- Module 5 - Stacking layers and overfitting a MLP on CIFAR10
- Module 6: Convolutional neural network
- how to regularize with dropout and uncertainty estimation with MC Dropout: Module 15 - Dropout
<details>
<summary>Things to remember</summary>
- Loss vs Accuracy. Know your loss for a classification task!
- know your optimizer (Module 4)
- know how to build a neural net with torch.nn.module (Module 5)
- know how to use convolution and pooling layers (kernel, stride, padding)
- know how to use dropout
</details>
- Module 7 - Dataloading
- Module 8a - Embedding layers
- Module 8b - Collaborative filtering and build your own recommender system: 08_collaborative_filtering_empty.ipynb (on a larger dataset 08_collaborative_filtering_1M.ipynb)
- Module 8c - Word2vec and build your own word embedding 08_Word2vec_pytorch_empty.ipynb
- Module 16 - Batchnorm and check your understanding with 16_simple_batchnorm_eval.ipynb and more 16_batchnorm_simple.ipynb
- Module 17 - Resnets
- start of Homework 2: Class Activation Map and adversarial examples
<details>
<summary>Things to remember</summary>
- know how to use dataloader
- to deal with categorical variables in deep learning, use embeddings
- in the case of word embedding, starting in an unsupervised setting, we built a supervised task (i.e. predicting central / context words in a window) and learned the representation thanks to negative sampling
- know your batchnorm
- architectures with skip connections allows deeper models
</details>
- Module 9a: Autoencoders and code your noisy autoencoder 09_AE_NoisyAE.ipynb
- Module 10: Generative Adversarial Networks and code your GAN, Conditional GAN and InfoGAN 10_GAN_double_moon.ipynb
- Module 13: Siamese Networks and Representation Learning
- start of Homework 3: VAE for MNIST clustering and generation
- Module 11a - Recurrent Neural Networks theory
- Module 11b - Recurrent Neural Networks practice and predict engine failure with 11_predicitions_RNN_empty.ipynb
- Module 11c - Batches with sequences in Pytorch
- Module 12 - Attention and Transformers
- Correcting the PyTorch tutorial on attention in seq2seq: 12_seq2seq_attention.ipynb
- Build your own microGPT: GPT_hist.ipynb
- Module 9b - UNets
- Module 9c - Flows
- Build your own Real NVP: Normalizing_flows_empty.ipynb
- Module 18a - Denoising Diffusion Probabilistic Models
- Train your own DDPM on MNIST: ddpm_nano_empty.ipynb
- Finetuning on CIFAR10: ddpm_micro_sol.ipynb
For more updates:
and check the
Marc Lelarge, Andrei Bursuc with Jill-Jênn Vie
Super fast track to learn the basics of deep learning from scratch:
- Have a look at the slides of Module 1: Introduction & General Overview
- Run the notebook (or in colab) of Module 2a: Pytorch Tensors
- Run the notebook (or in colab) of Module 2b: Automatic Differentiation
- Check the Minimal working examples of Module 3: Loss functions for classification. If you do not understand, have a look at the slides.
- Have a look at the slides of Module 4: Optimization for Deep Learning
- Try playback speed 1.5 for the video from Module 5: Stacking layers.
- Run the notebook (or in colab) of Module 6: Convolutional Neural Network
- Try playback speed 2 for the video from Module 7: Dataloading
- Have a look at the slides of Module 8a: Embedding layers
- Well done! Now you have time to enjoy deep learning!
Join the GitHub repo dataflowr and make a pull request. What are pull requests?
Thanks to Daniel Huynh, Eric Daoud, Simon Coste
Materials from this site is used for courses at ENS and X.
<img src="/assets/ENS_logo.jpg" style="width: 400px; height: auto; display: inline">
<img src="/assets/X_logo.png" style="margin-left:1em; width: 180px; height: auto; display: inline">