09a_CNN.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "cA98_ZQeV1_d"
      },
      "source": [
        "# Tutorial 09 - CNN with TensorFlow\n",
        "\n",
        "## Dr. David C. Schedl\n",
        "\n",
        "Note: this tutorial is geared towards students **experienced in programming** and aims to introduce you to **Tensorflow and CNNs**.\n",
        "\n",
        "This notebook has **not been tested** to work on a **local** Python installation.\n",
        "Therefore, it is highly recommended to run this notebook **on Google Colab**, since it relies on Tensorflow and some related libraries that run flawlessly there. \n",
        "\n",
        "For training, it is recommended to use a **GPU**. In Google Colab go to the menu and select **Edit** -> **Notebook settings** -> **Hardware accelerator** -> switch to **GPU**.\n",
        "\n",
        "Useful links:\n",
        "* [Tensorflow API documentation](https://www.tensorflow.org/api_docs/python/tf)\n",
        "\n",
        "\n",
        "#### Acknowledgements\n",
        "The code of this tutorial is based on a notebook from [datahacker.rs](https://datahacker.rs/lenet-5-implementation-tensorflow-2-0/).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gNBpxqtdERzO"
      },
      "source": [
        "# Initialization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Ajkzi72ETUH"
      },
      "source": [
        "As always let's import useful libraries, first. \n",
        "We will work with TensorFlow and MNIST today. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1rwWxBBYCkuT"
      },
      "outputs": [],
      "source": [
        "%%capture \n",
        "# use %% capture suppress any output\n",
        "\n",
        "# make sure to use tensorflow 2.x\n",
        "%tensorflow_version 2.x\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import datetime\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.losses import categorical_crossentropy\n",
        "from tensorflow.keras.layers import Dense, Flatten, Conv2D, AveragePooling2D, MaxPooling2D\n",
        "\n",
        "from tensorflow.keras import datasets\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "from __future__ import absolute_import, division, print_function, unicode_literals\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kFbyHYemoTLm"
      },
      "outputs": [],
      "source": [
        "# Load the TensorBoard notebook extension to display training results later\n",
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u748NlvWQNPv"
      },
      "source": [
        "# LeNet-5 and MNIST"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wuEKQ8fbEIvg"
      },
      "source": [
        "## Preparing the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ms6WetkBcWDP"
      },
      "outputs": [],
      "source": [
        "(x_train, y_train), (x_test, y_test) = datasets.mnist.load_data() # it would be pretty easy to change the dataset here! :)\n",
        "# for example let's switch to cifar10 with the following line\n",
        "# (x_train, y_train), (x_test, y_test) = datasets.cifar10.load_data()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N4_4PluOcZe_"
      },
      "outputs": [],
      "source": [
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')\n",
        "print(x_train[0].shape, 'image shape')\n",
        "\n",
        "# show example image(s)\n",
        "plt.subplot(121), plt.imshow(x_train[0], cmap=plt.cm.gray_r), plt.title(f'train: {y_train[0]}')\n",
        "plt.subplot(122), plt.imshow(x_test[1], cmap=plt.cm.gray_r), plt.title(f'test: {y_test[0]}')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_c0DplXTkpGB"
      },
      "outputs": [],
      "source": [
        "if len(x_train.shape)<4 :\n",
        "  # Add a new axis to reflect that we have grayscale data\n",
        "  x_train = x_train[:, :, :, np.newaxis]\n",
        "  x_test = x_test[:, :, :, np.newaxis]\n",
        "\n",
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')\n",
        "print(x_train[0].shape, 'image shape')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fUUkZmkoccyg"
      },
      "outputs": [],
      "source": [
        "# Convert class vectors to binary class matrices.\n",
        "# For example a test label of 7 is converted to [0, ... 0, 1, 0, 0]\n",
        "num_classes = 10\n",
        "y_train = to_categorical(y_train, num_classes)\n",
        "y_test = to_categorical(y_test, num_classes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A5FFw9aDdcFk"
      },
      "outputs": [],
      "source": [
        "# Data normalization \n",
        "# Make sure we use floating point images and a range of 0 to 1\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihmBukV1Fhnu"
      },
      "source": [
        "## The Model: LeNet-5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_euKrUatG8qo"
      },
      "source": [
        "We will implement the LeNet-5 architecture as proposed by Lecun et al. \n",
        "The layer options in `[]` are the ones originally proposed in the paper (legacy). We add the option to also use a slightly modified variant.\n",
        "\n",
        "\n",
        "| Layer                          | Output Size  | Weight Size     |\n",
        "| ------------------------------ | ------------ | --------------- |\n",
        "| Input                          | 1 x 28 x 28  |                 |\n",
        "| Conv (Cout\\=20 [6], K=5, P=2, S=1) | 20 x 28 x 28 | 20 x 1 x 5 x 5  |\n",
        "| ReLU [Sigmoid]                          | 20 x 28 x 28 |                 |\n",
        "| MaxPool(K=2, S=2)              | 20 x 14 x 14 |                 |\n",
        "| Conv (Cout\\=50 [16], K=5, P=2, S=1) | 50 x 14 x 14 | 50 x 20 x 5 x 5 |\n",
        "| ReLU [Sigmoid]                           | 50 x 14 x 14 |                 |\n",
        "| MaxPool(K=2, S=2)              | 50 x 7 x 7   |                 |\n",
        "| Flatten                        | 2450         |                 |\n",
        "| Linear (2450 -> 500)           | 500          | 2450 x 500      |\n",
        "| ReLU [Sigmoid]                          | 500          |                 |\n",
        "| Linear (500 -> 10)             | 10           | 500 x 10        |"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UA2ehjxgF7bY"
      },
      "outputs": [],
      "source": [
        "# LeNet-5 model\n",
        "class LeNet(Sequential):\n",
        "  def __init__(self, input_shape, nb_classes, legacy=True):\n",
        "    super().__init__()\n",
        "\n",
        "    activation = 'sigmoid' if legacy else 'relu'\n",
        "\n",
        "    self.add(Conv2D(6 if legacy else 20, kernel_size=(5, 5), strides=(1, 1), activation=activation, input_shape=input_shape, padding=\"same\"))\n",
        "    self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n",
        "    self.add(Conv2D(16 if legacy else 50, kernel_size=(5, 5), strides=(1, 1), activation=activation, padding='same'))\n",
        "    self.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n",
        "    self.add(Flatten())\n",
        "    self.add(Dense(500, activation=activation))\n",
        "    self.add(Dense(nb_classes, activation='softmax'))\n",
        "\n",
        "    self.compile(optimizer='adam',\n",
        "                loss=categorical_crossentropy,\n",
        "                metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K3oQd9xnc469"
      },
      "outputs": [],
      "source": [
        "# create a model\n",
        "model = LeNet(x_train[0].shape, num_classes, legacy = False)\n",
        "\n",
        "# print a summary of its structure and parameters\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jw0Ui-AzI1_m"
      },
      "source": [
        "## Training\n",
        "\n",
        "For training we will use the `tf.keras.Model.fit()` method.\n",
        "Furthermore, we can monitor the training progress with Tensorboard. Tensorboard is a tool that allows us to visualize the training progress based on the log files created during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qXUVl6tWoO00"
      },
      "outputs": [],
      "source": [
        "# Training Preparation\n",
        "\n",
        "# Place the logs in a timestamped subdirectory\n",
        "# This allows to easy select different training runs\n",
        "# In order not to overwrite some data, it is useful to have a name with a timestamp\n",
        "log_dir=\"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "# Specify the callback object\n",
        "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n",
        "\n",
        "# tf.keras.callback.TensorBoard ensures that logs are created and stored\n",
        "# We need to pass callback object to the fit method\n",
        "# The way to do this is by passing the list of callback objects, which is in our case just one"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X2em6BaAevsq"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir logs/fit\n",
        "# open the TensorBoard here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BWudz07TXuW4"
      },
      "outputs": [],
      "source": [
        "model.fit(x_train, y=y_train, \n",
        "          epochs=20, \n",
        "          validation_data=(x_test, y_test), \n",
        "          callbacks=[tensorboard_callback],\n",
        "          verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v8X4jEhdLdd8"
      },
      "source": [
        "## Inference\n",
        "\n",
        "For inference we will use the `tf.keras.Model.predict()` method. \n",
        "It returns the predicted values for all 10,000 test images. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pGHSJW2_Npxx"
      },
      "outputs": [],
      "source": [
        "# Inference for the 10,000 test images:\n",
        "prediction_values = model.predict(x_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PZ3rApaANdVh"
      },
      "source": [
        "### Displaying Results\n",
        "\n",
        "First let's display the first 50 images in the MNIST dataset and let's verify what our model predicted. \n",
        "For each image we get 10 propabilities for how sure the network is that it sees a certain digit.\n",
        "If training worked well, our model should correctly classify the first 50 digits and give the highest score for the correct labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UxkJheaXWl1X"
      },
      "outputs": [],
      "source": [
        "# set up a figure\n",
        "fig = plt.figure(figsize=(15, 7))\n",
        "fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)\n",
        "\n",
        "# plot the images: each image is 28x28 pixels\n",
        "for i in range(50):\n",
        "  ax = fig.add_subplot(5, 10, i + 1, xticks=[], yticks=[])\n",
        "  ax.imshow(x_test[i,:].reshape((28,28)),cmap=plt.cm.gray_r, interpolation='nearest')\n",
        "\n",
        "  prediction_label = np.argmax(prediction_values[i])\n",
        "\n",
        "  img_text = f'{prediction_label} [{np.argmax(y_test[i])}]'\n",
        "  \n",
        "  if prediction_label == np.argmax(y_test[i]):\n",
        "    # label the image with the blue text\n",
        "    ax.text(0.1, 0.1, img_text, color='green', transform=ax.transAxes)\n",
        "    ax.tick_params(color='green', labelcolor='green')\n",
        "    for spine in ax.spines.values():\n",
        "        spine.set_edgecolor('green')\n",
        "  else:\n",
        "    # label the image with the red text\n",
        "    ax.text(0.1, 0.1, img_text, color='red', transform=ax.transAxes)\n",
        "    ax.tick_params(color='red', labelcolor='red')\n",
        "    for spine in ax.spines.values():\n",
        "        spine.set_edgecolor('red')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S3jYgWpnPKGW"
      },
      "source": [
        "### What did not work?\n",
        "\n",
        "Let's also look at what did not work! \n",
        "We compute the difference between the ground truth (`y_test`) and the predicitons (`prediction_values`) and look at the pairs with the highest difference.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_y3FvMFPK8-g"
      },
      "outputs": [],
      "source": [
        "diff = np.linalg.norm( prediction_values - y_test, axis=1 )\n",
        "\n",
        "# set up the figure\n",
        "fig = plt.figure(figsize=(15, 7))\n",
        "fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)\n",
        "\n",
        "count = 0\n",
        "\n",
        "for id in np.flip(np.argsort(diff)):\n",
        "  #print(np.argmax(y_test[id]))\n",
        "  #print(np.argmax(prediction_values[id]))\n",
        "\n",
        "  ax = fig.add_subplot(5, 10, count + 1, xticks=[], yticks=[])\n",
        "  ax.imshow(x_test[id,:].reshape((28,28)),cmap=plt.cm.gray_r, interpolation='nearest')\n",
        "\n",
        "  prediction_label = np.argmax(prediction_values[id])\n",
        "\n",
        "  img_text = f'{prediction_label} [{np.argmax(y_test[id])}]'\n",
        "  \n",
        "  if prediction_label == np.argmax(y_test[id]):\n",
        "    # label the image with the blue text\n",
        "    ax.text(0.1, 0.1, img_text, color='green', transform=ax.transAxes)\n",
        "    ax.tick_params(color='green', labelcolor='green')\n",
        "    for spine in ax.spines.values():\n",
        "        spine.set_edgecolor('green')\n",
        "  else:\n",
        "    # label the image with the red text\n",
        "    ax.text(0.1, 0.1, img_text, color='red', transform=ax.transAxes)\n",
        "    ax.tick_params(color='red', labelcolor='red')\n",
        "    for spine in ax.spines.values():\n",
        "        spine.set_edgecolor('red')\n",
        "\n",
        "  count += 1\n",
        "  if count >= 50:\n",
        "    break\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JMBfneL8UyvL"
      },
      "source": [
        "### ⌨️ Try it yourself: Compute the Numbers!\n",
        "\n",
        "How many digits have been classified wrong and how many correct?\n",
        "Compute the numbers (also in percentage), it is a good way to evaluate the quality of the network. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gKY0Yv7KVGM0"
      },
      "outputs": [],
      "source": [
        "# Todo: compute the numbers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nolgAoSxseiO"
      },
      "source": [
        "## ⌨️ Try it yourself: Compare the legacy LeNet-5 to our modern variant!\n",
        "\n",
        "We ran our script with the modernized LeNet-5 version. How does the original LeNet-5 implementation perform in comparison? \n",
        "How many digits get classified wrong in both versions?\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nVykTp2GUmi4"
      },
      "outputs": [],
      "source": [
        "# Todo: run the legacy LeNet-5"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "08_CNN",
      "provenance": [],
      "toc_visible": true
    },
    "interpreter": {
      "hash": "68aa101d40deae760c6b7087fff2604c51f5497e89f84f2e481cb55f2e266a0b"
    },
    "kernelspec": {
      "display_name": "Python 3.10.4 ('tf')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}