Machine Learning - Master ICFP

Prerequisites:

• Proficiency in Python: please use the tutorial here for those who aren't as familiar with Python
• Basic Calculus, Linear Algebra
• Basic Probability and Statistics

1. Fundamentals of predictions and supervised learning (16/01)

Fundamentals of predictions

• Minimizing errors
• Modeling knowledge
• Prediction via optimization
• Types of errors and successes
• Properties of ROC curves

Ref

• Fundamentals of prediction from Patterns, Predictions, and Actions (A story about machine learning) by Moritz Hardt and Benjamin Recht

practicals

• Exact ROC curves for Gaussian mixtures: https://github.com/mlelarge/icfp-ml/blob/main/Exact_ROC_GM.ipynb

supervised learning

• Sample versus Population
• A first learning algorithm: the perceptron
• Connection to empirical risk minimization
• Formal guarantees for the perceptron

Ref:

• Supervised learning from Patterns, Predictions, and Actions (A story about machine learning) by Moritz Hardt and Benjamin Recht

practicals

• Naive Bayes and logistic regression: https://github.com/mlelarge/icfp-ml/blob/main/01_NaivesBayes_Logistic_empty.ipynb

2. Pytorch basics and autodiff (23/01)

Module 2a - Pytorch tensors

Module 2b - Automatic differentiation

3. Optimization for machine learning (30/01)

• gradient descent
• SGD
• over-parameterized models:https://hackmd.io/@mlelarge/S1y5bEAhj

Ref:

• In Chapter 5 (Sections 5.2.1 and 5.4) of Learning Theory from First Principles by Francis Bach

practicals

• Module 5 - Stacking layers

• Heavy Ball Method: https://github.com/mlelarge/icfp-ml/blob/main/HeavyBall_empty.ipynb

4. Kernels (06/02)

• Local averaging methods
  • partitions estimators
  • k-nearest neighbors
  • kernel smoothing
• Positive-definite kernel methods
  • representer theorem
  • kernel trick

Ref:

• Chapters 6 and 7 of Learning Theory from First Principles by Francis Bach

practicals

• Kernel with random Fourier features

5. Unsupervised Learning (13/02)

• K-means clustering
• Mixtures of Gaussian
• Expectation-Maximization for GMM

Ref:

• Expectation-Maximization for the Gaussian Mixture Model by Thomas Bonald
• The Expectation Maximization Algorithm - A short tutorial by Sean Borman

practicals

• SVD
• Eigenfaces

6. Bayesian and Variational Inference (20/02 06-13/03)

• Gaussian

• Linear regression

• Logistic regression

• Laplace method

• Variational inference

  Ref:

  • Chapters 2.3 - 3.3 - 4.4 - 4.5 - 10 Pattern Recognition and Machine Learning by Christopher Bishop