01_intro_DLDIY_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_intro_DLDIY_colab.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mlelarge/dataflowr/blob/master/01_intro_DLDIY_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "-SvfKxAGCdhF"
      },
      "cell_type": "markdown",
      "source": [
        "# Using CNN for dogs vs cats"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "VI1VQlwECdhG"
      },
      "cell_type": "markdown",
      "source": [
        "We are going to create a model to enter the [Dogs vs Cats](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition) competition at Kaggle."
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "dyhW08piCdhH"
      },
      "cell_type": "markdown",
      "source": [
        "There are 25,000 labelled dog and cat photos available for training, and 12,500 in the test set that we have to try to label for this competition. According to the Kaggle web-site, when this competition was launched (end of 2013): *\"**State of the art**: The current literature suggests machine classifiers can score above 80% accuracy on this task\"*. So if you can beat 80%, then you will be at the cutting edge as of 2013!"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "uFXJDuj3C01w"
      },
      "cell_type": "markdown",
      "source": [
        "## Colab preparation"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "NcRY9-oUCdhJ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# to install pytorch on colab\n",
        "from os import path\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "\n",
        "accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'\n",
        "\n",
        "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "edhumzlSDB5h",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -U bcolz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "2Os2SV8vDEai",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install Pillow==4.0.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "h4dAhx8-CdhM"
      },
      "cell_type": "markdown",
      "source": [
        "##  Imports"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "6oU4Z_DPCdhM",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from PIL import Image\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import os\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision\n",
        "from torchvision import models,transforms,datasets\n",
        "import bcolz\n",
        "import time\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "A66_r51xCdhS",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "torch.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "KN3FFTFhHQyi",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.version"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "tuej9DPjCdhX"
      },
      "cell_type": "markdown",
      "source": [
        "Check if GPU is available and if not change the [runtime](https://jovianlin.io/pytorch-with-gpu-in-google-colab/)."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "t56d0zbFCdhY",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "use_gpu = torch.cuda.is_available()\n",
        "print('Using gpu: %s ' % use_gpu)\n",
        "\n",
        "def gpu(x,use_gpu=use_gpu):\n",
        "    if use_gpu:\n",
        "        return x.cuda()\n",
        "    else:\n",
        "        return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "dDLlOjT5Et4p"
      },
      "cell_type": "markdown",
      "source": [
        "## Downloading the data"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "FSmB5oKYCdhc"
      },
      "cell_type": "markdown",
      "source": [
        "You can download the full dataset from Kaggle directly.\n",
        "\n",
        "Alternatively, Jeremy Howard provides a direct link to the catvsdogs [dataset](http://files.fast.ai/data/dogscats.zip). He's separated the cats and dogs into separate folders and created a validation folder as well. You'll need this folder structure to run VGG.\n",
        "\n",
        "For test purpose (or if you run on cpu), you should use the (small) sample directory."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Rnn5pLK6EyJK",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%mkdir data\n",
        "%cd /content/data/\n",
        "!wget http://files.fast.ai/data/dogscats.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "BTobJ9vTE37J",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!unzip dogscats.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "XsMtmnCbCdhd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "S5OW01WzCdhu",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%cd dogscats/\n",
        "%ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "Mkj3DZjpCdha"
      },
      "cell_type": "markdown",
      "source": [
        "## Data processing"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "iasXk_FKCdhy",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%cd .."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "MJRnJgGOCdh4",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data_dir = '/content/data/dogscats'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "U3lE0cvyCdh8"
      },
      "cell_type": "markdown",
      "source": [
        "```datasets``` is a class of the ```torchvision``` package (see [torchvision.datasets](http://pytorch.org/docs/master/torchvision/datasets.html)) and deals with data loading. It integrates a multi-threaded loader that fetches images from the disk, groups them in mini-batches and serves them continously to the GPU right after each _forward_/_backward_ pass through the network.\n",
        "\n",
        "Images needs a bit of preparation before passing them throught the network. They need to have all the same size $224\\times 224 \\times 3$ plus some extra formatting done below by the normalize transform (explained later)."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "8t4vokNrF19p",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
        "\n",
        "vgg_format = transforms.Compose([\n",
        "                transforms.CenterCrop(224),\n",
        "                transforms.ToTensor(),\n",
        "                normalize,\n",
        "            ])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "l8LMReVECdh-",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), vgg_format)\n",
        "         for x in ['train', 'valid']}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "pMh7kjEBCdiC",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.path.join(data_dir,'train')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "VNyS0TyeKwdO"
      },
      "cell_type": "markdown",
      "source": [
        "Interactive help on jupyter notebook thanks to `?`"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "WX5pkMX4CdiF",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "?datasets.ImageFolder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "oYmIGNEjKwdR"
      },
      "cell_type": "markdown",
      "source": [
        "We see that `datasets.ImageFolder` has attributes: classes, class_to_idx, imgs.\n",
        "\n",
        "Let see what they are?"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "m9ifn_R7CdiH",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dsets['train'].classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "TB6sTwFuCdiK",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dsets['train'].class_to_idx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "y9ECrD2ACdiO",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dsets['train'].imgs[:5]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "WefhjZb2CdiQ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dset_sizes = {x: len(dsets[x]) for x in ['train', 'valid']}\n",
        "dset_sizes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "SCLV1YgaCdiT",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dset_classes = dsets['train'].classes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "Zy52-XhACdiX"
      },
      "cell_type": "markdown",
      "source": [
        "The ```torchvision``` packages allows complex pre-processing/transforms of the input data (_e.g._ normalization, cropping, flipping, jittering). A sequence of transforms can be grouped in a pipeline with the help of the ```torchvision.transforms.Compose``` function, see [torchvision.transforms](http://pytorch.org/docs/master/torchvision/transforms.html)"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "iphL57PhKwdh"
      },
      "cell_type": "markdown",
      "source": [
        "The magic help `?` allows you to retrieve function you defined and forgot!"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "X_ARYwraCdiY",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "?vgg_format"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "tWnJQiWgGP_R",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def shuffle_valtrain(x):\n",
        "    if x == 'train':\n",
        "        return True\n",
        "    else:\n",
        "        return False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "8KQj0Zm3Cdic",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=64,\n",
        "                                               shuffle=shuffle_valtrain(x), num_workers=6)\n",
        "                for x in ['train', 'valid']}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Iern_6GNCdie",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "?torch.utils.data.DataLoader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "wN1BKHfDCdig",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dataset_valid = torch.utils.data.DataLoader(dsets['valid'], batch_size=5, shuffle=True, num_workers=6)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Z4Be7lSLCdik",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "count = 1\n",
        "for data in dataset_valid:\n",
        "    print(count, end=',')\n",
        "    if count == 1:\n",
        "        inputs_try,labels_try = data\n",
        "    count +=1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "3BvxQqfzCdiq",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "MLNsfqc8Cdis",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "inputs_try.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "vsaL21ouKwd9"
      },
      "cell_type": "markdown",
      "source": [
        "A small function to display images:"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "346yl-gbcLLm",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def imshow(inp, title=None):\n",
        "#   Imshow for Tensor.\n",
        "    inp = inp.numpy().transpose((1, 2, 0))\n",
        "    mean = np.array([0.485, 0.456, 0.406])\n",
        "    std = np.array([0.229, 0.224, 0.225])\n",
        "    inp = np.clip(std * inp + mean, 0,1)\n",
        "    plt.imshow(inp)\n",
        "    if title is not None:\n",
        "        plt.title(title)\n",
        "    plt.pause(0.001)  # pause a bit so that plots are updated"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "IJbW-QzGCdiv",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs_try)\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "8ZK7nRSWKweH"
      },
      "cell_type": "markdown",
      "source": [
        "What is an image for your computer?"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "hrzxYMXXCdix",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "inputs_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "2HvFNWJICdiz",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(dset_loaders['train']))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "iUrTGlTpCdi5",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Get a batch of validation data\n",
        "inputs, classes = next(iter(dset_loaders['valid']))\n",
        "\n",
        "n_images = 8\n",
        "\n",
        "# Make a grid from batch\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "LOvkYiROCdi7"
      },
      "cell_type": "markdown",
      "source": [
        "## Creating VGG Model"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "BO5LAG4bCdi7"
      },
      "cell_type": "markdown",
      "source": [
        "The torchvision module comes with a zoo of popular CNN architectures which are already trained on [ImageNet](http://www.image-net.org/) (1.2M training images). When called the first time, if ```pretrained=True``` the model is fetched over the internet and downloaded to ```~/.torch/models```.\n",
        "For next calls, the model will be directly read from there."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "K9PsHjXgCdi9",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model_vgg = models.vgg16(pretrained=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "VBzGe2mlCdi-"
      },
      "cell_type": "markdown",
      "source": [
        "We will first use VGG Model without any modification. In order to interpret the results, we need to import the 1000 ImageNet categories, available at: [https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json](https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json)"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "qhvs0ki_Cdi_",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Hppljqo5CdjC",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import json\n",
        "\n",
        "fpath = '/content/data/imagenet_class_index.json'\n",
        "\n",
        "with open(fpath) as f:\n",
        "    class_dict = json.load(f)\n",
        "dic_imagenet = [class_dict[str(i)][1] for i in range(len(class_dict))]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "QATvAFILCdjF",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dic_imagenet[:4]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "p6QQhwruCdjI",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "inputs_try , labels_try = gpu(inputs_try),gpu(labels_try)\n",
        "\n",
        "model_vgg = gpu(model_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "epMB0UF9CdjM",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "outputs_try = model_vgg(inputs_try)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "dOlx7YcPCdjO",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "outputs_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "MCIxHN2QCdjT",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m_softm = nn.Softmax(dim=1)\n",
        "vals_try,preds_try = torch.max(m_softm(outputs_try.data),1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "CvGbb2bQCdjZ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vals_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "0TUdNKCJCdjc",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print([dic_imagenet[i] for i in preds_try.data])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "jtzuHob9Cdjf",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "out = torchvision.utils.make_grid(inputs_try.data.cpu())\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in labels_try.data.cpu()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "f3Bktt5PCdjh"
      },
      "cell_type": "markdown",
      "source": [
        "### Modifying the last layer and setting the gradient false to all layers"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "L8kr3-tjCdji",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(model_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "IXoMW73MCdjl"
      },
      "cell_type": "markdown",
      "source": [
        "We'll learn about what these different blocks do later in the course. For now, it's enough to know that:\n",
        "\n",
        "- Convolution layers are for finding small to medium size patterns in images -- analyzing the images locally\n",
        "- Dense (fully connected) layers are for combining patterns across an image -- analyzing the images globally\n",
        "- Pooling layers downsample -- in order to reduce image size and to improve invariance of learned features"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "hA2f5FRuCdjm"
      },
      "cell_type": "markdown",
      "source": [
        "![vgg16](https://mlelarge.github.io/dataflowr/Notebooks/vgg16.png)"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "SK6lfAzfCdjn"
      },
      "cell_type": "markdown",
      "source": [
        "In this practical example, our goal is to use the already trained model and just change the number of output classes. To this end we replace the last ```nn.Linear``` layer trained for 1000 classes to ones with 2 classes. In order to freeze the weights of the other layers during training, we set the field ```required_grad=False```. In this manner no gradient will be computed for them during backprop and hence no update in the weights. Only the weights for the 2 class layer will be updated."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "rQwRKKC-Cdjo",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for param in model_vgg.parameters():\n",
        "    param.requires_grad = False\n",
        "model_vgg.classifier._modules['6'] = nn.Linear(4096, 2)\n",
        "model_vgg.classifier._modules['7'] = torch.nn.LogSoftmax(dim = 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "qAr5tj2NKwfF"
      },
      "cell_type": "markdown",
      "source": [
        "PyTorch documentation for [LogSoftmax](https://pytorch.org/docs/stable/nn.html#logsoftmax)"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "jJ3OenJpCdjp",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(model_vgg.classifier)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "9ITZFX2MCdju",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model_vgg = gpu(model_vgg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "hMAwUx7QCdjv"
      },
      "cell_type": "markdown",
      "source": [
        "### Calculating preconvoluted features"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "iWZ0BguiCdjw",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def preconvfeat(dataset):\n",
        "    conv_features = []\n",
        "    labels_list = []\n",
        "    for data in dataset:\n",
        "        inputs,labels = data\n",
        "        inputs = gpu(inputs)\n",
        "        labels = gpu(labels)\n",
        "        # if you find code like below on internet (or in this course!):\n",
        "        # Variable is a PyTorch v0.3 recollection...\n",
        "        # it should still run with newer versions of PyTorch\n",
        "        #if use_gpu:\n",
        "        #    inputs , labels = Variable(inputs.cuda()),Variable(labels.cuda())\n",
        "        #else:\n",
        "        #    inputs , labels = Variable(inputs),Variable(labels)\n",
        "        \n",
        "        x = model_vgg.features(inputs)\n",
        "        conv_features.extend(x.data.cpu().numpy())\n",
        "        labels_list.extend(labels.data.cpu().numpy())\n",
        "    conv_features = np.concatenate([[feat] for feat in conv_features])\n",
        "    return (conv_features,labels_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "UHxk_fBaCdjx",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x_try = model_vgg.features(inputs_try)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "4_XDasbkCdj0",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "?x_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "XoKGJ-4iCdj2",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x_try.data.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "KX0SqD0CCdj4",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "7*7*512"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "pbAyXnwgCdj6",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels_try"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "bs_NwTKcCdj8",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels_try.data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "AHYRm_z8Cdj-",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "conv_feat_train,labels_train = preconvfeat(dset_loaders['train'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "lNAc7zX9CdkB",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "conv_feat_val,labels_val = preconvfeat(dset_loaders['valid'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "3-APvzkACdkD",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def save_array(fname, arr):\n",
        "    c=bcolz.carray(arr, rootdir=fname, mode='w')\n",
        "    c.flush()\n",
        "def load_array(fname):\n",
        "    return bcolz.open(fname)[:]\n",
        "\n",
        "\n",
        "%mkdir /content/data/dogscats/vgg16\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "AWadFbvtCdkF",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "save_array(os.path.join(data_dir,'vgg16','feat_train.bc'),conv_feat_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_train.bc'),labels_train)\n",
        "save_array(os.path.join(data_dir,'vgg16','feat_val.bc'),conv_feat_val)\n",
        "save_array(os.path.join(data_dir,'vgg16','labels_val.bc'),labels_val)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "pyjlrH4wCdkH"
      },
      "cell_type": "markdown",
      "source": [
        "### Uploading Precomputed features\n",
        "\n",
        "This section will allow you to store the precomputed features on your Google drive for later use."
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "27gRV8KiJw6H",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%cd /content/data/dogscats/\n",
        "!zip -r vgg16 vgg16/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "de4y59inJx_p",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -U -q PyDrive\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "\n",
        "# 1. Authenticate and create the PyDrive client.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "JIaZyCpcJ1Hd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "upload = drive.CreateFile({'title': 'vgg16_drive.zip'})\n",
        "upload.SetContentFile('vgg16.zip')\n",
        "upload.Upload()\n",
        "print('Uploaded file with ID {}'.format(upload.get('id')))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "rCxGfgBHCdkJ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "conv_feat_train.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "fU9vWWT2CdkN"
      },
      "cell_type": "markdown",
      "source": [
        "## Training fully connected module"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "qqp2u3IXCdkO"
      },
      "cell_type": "markdown",
      "source": [
        "### Creating loss function and optimizer\n",
        "\n",
        "PyTorch documentation for [NLLLoss](https://pytorch.org/docs/stable/nn.html#nllloss) and the [torch.optim module](https://pytorch.org/docs/stable/optim.html#module-torch.optim)"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "oP1F4yb8CdkO",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "criterion = nn.NLLLoss()\n",
        "lr = 0.01\n",
        "optimizer_vgg = torch.optim.SGD(model_vgg.classifier[6].parameters(),lr = lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "NUSxP3BZCdkP"
      },
      "cell_type": "markdown",
      "source": [
        "### Creating Data generator"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "KQxZBVOGCdkQ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def data_gen(conv_feat,labels,batch_size=64,shuffle=True):\n",
        "    labels = np.array(labels)\n",
        "    if shuffle:\n",
        "        index = np.random.permutation(len(conv_feat))\n",
        "        conv_feat = conv_feat[index]\n",
        "        labels = labels[index]\n",
        "    for idx in range(0,len(conv_feat),batch_size):\n",
        "        yield(conv_feat[idx:idx+batch_size],labels[idx:idx+batch_size])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "tenuLj67CdkS"
      },
      "cell_type": "markdown",
      "source": [
        "### Training the model"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "7nNAUibjCdkS",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_model(model,size,conv_feat=None,labels=None,epochs=1,optimizer=None,train=True,shuffle=True):\n",
        "    if train:\n",
        "        model.train()\n",
        "    else:\n",
        "        model.eval()\n",
        "        \n",
        "    for epoch in range(epochs):\n",
        "        batches = data_gen(conv_feat=conv_feat,labels=labels,shuffle=shuffle)\n",
        "        total = 0\n",
        "        running_loss = 0.0\n",
        "        running_corrects = 0\n",
        "        for inputs,classes in batches:\n",
        "            inputs , classes = gpu(torch.from_numpy(inputs)), gpu(torch.from_numpy(classes))\n",
        "                            \n",
        "            inputs = inputs.view(inputs.size(0), -1)\n",
        "            outputs = model(inputs)\n",
        "            loss = criterion(outputs,classes)           \n",
        "            if train:\n",
        "                if optimizer is None:\n",
        "                    raise ValueError('Pass optimizer for train mode')\n",
        "                optimizer = optimizer\n",
        "                optimizer.zero_grad()\n",
        "                loss.backward()\n",
        "                optimizer.step()\n",
        "            _,preds = torch.max(outputs.data,1)\n",
        "            # statistics\n",
        "            running_loss += loss.data.item()\n",
        "            running_corrects += torch.sum(preds == classes.data)\n",
        "        epoch_loss = running_loss / size\n",
        "        epoch_acc = running_corrects.data.item() / size\n",
        "        print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
        "                     epoch_loss, epoch_acc))\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "_Jts2jK1CdkV",
        "scrolled": false,
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "(train_model(model=gpu(model_vgg.classifier),size=dset_sizes['train'],conv_feat=conv_feat_train,labels=labels_train,\n",
        "            epochs=10,optimizer=optimizer_vgg,train=True,shuffle=True))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "y3JjYayKCdkW",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train_model(conv_feat=conv_feat_val,labels=labels_val,model=gpu(model_vgg.classifier)\n",
        "            ,size=dset_sizes['valid'],train=False,shuffle=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Cn6cIwZNCdkY",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Get a batch of training data\n",
        "inputs, classes = next(iter(dset_loaders['valid']))\n",
        "\n",
        "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
        "\n",
        "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Ghkbm7ByCdke",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "inputs = torch.from_numpy(conv_feat_val[:n_images])\n",
        "inputs = inputs.view(inputs.size(0), -1)\n",
        "if use_gpu:\n",
        "    outputs = model_vgg.classifier(inputs.cuda())\n",
        "else:\n",
        "    outputs = model_vgg.classifier(inputs)\n",
        "print(torch.exp(outputs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "F1qdUtvjCdkg",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels_val[:n_images]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Fl5_aK0uOjSG",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hjwIo3gbsGMx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Conclusion\n",
        "\n",
        "What did we do in the end? A simple logistic regression! If the connection is unclear, we'll explain it on a much simpler example in the next course. \n",
        "\n",
        "We probably killed a fly with a sledge hammer.\n",
        "\n",
        "![mouche](https://mlelarge.github.io/dataflowr-web/images/mouche.jpg)\n",
        "\n",
        "In our case, the sledge hammer is VGG pretrained on Imagenet, a dataset containing a lot of pictures of cats and dogs. Indeed, we saw that without modification the network was able to predict dog and cat breeds. Hence it is not very surprising that the features computed by VGG are very accurate for our classification task. In the end, we need to learn only the parameters of the last linear layer, i.e. 8194 parameters (do not forget the bias $2\\times 4096+2$). Indeed, this can be done on CPU without any problem.\n",
        "\n",
        "Nevertheless, this example is still instructive as it shows all the necessary steps in a deep learning project. Here we did not struggle with the learning process of a deep network, but we did all the preliminary engineering tasks: dowloading a dataset, setting up the environment to use a GPU, preparing the data, computing the features with a pretrained VGG, saving them on your drive so that you can use them for a later experiment... These steps are essential in any deep learning project and a necessary requirement before having fun playing with network architectures and understanding the learning process."
      ]
    },
    {
      "metadata": {
        "id": "_NhP9XMbuK1A",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}