part2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "dlim_part2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nn45mUUp-1Gy"
      },
      "source": [
        "# **Deep Learning with PyTorch** -  Introductory Lab\n",
        "\n",
        "## **Part 2:** Image Operations with PyTorch\n",
        "\n",
        "* In this part we will use PyTorch to operate on images.\n",
        "* This notebook is designed to be run on Colab.\n",
        "* But it can easily be modified to be run locally.\n",
        "\n",
        "### Setup\n",
        "* First we need to setup some things.\n",
        "* Add the image data to you google drive by following these steps:\n",
        "    1. Click on [this Google drive folder](https://drive.google.com/drive/folders/1M_5MmGsHMxbTraagutaw2T3Ii537qDrr?usp=sharing)\n",
        "    2. Add a shortcut of that folder to you drive (in google drive, right-click on the filder and create shortcut)\n",
        "    3. Run the code in the cell below and follow the instructions to mount your drive to Colab.\n",
        "* If you instead want to run things locally on your computer, just download the data in the drive folder, and set the `dataset_path` to the location on your computer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OU-_87mdjdd4"
      },
      "source": [
        "# Connect your google drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L8LmHbPh9k6Z"
      },
      "source": [
        "# Set up path to your dataset\n",
        "# Use this path if you have copied the data to your Google drive\n",
        "# If you put the data somewhere else, you need to modify this\n",
        "dataset_path = '/content/gdrive/My Drive/DIV2K_train_small/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dMIAy3AllIrB"
      },
      "source": [
        "# Import libraries we will need\n",
        "import torch\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from PIL import Image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nXZn-a-w_7ZJ"
      },
      "source": [
        "### Reading and showing and image\n",
        "* We will first try to read an image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "noe3GkJY_5qp"
      },
      "source": [
        "# We will use PIL to read an image\n",
        "im_pil = Image.open(dataset_path + '0701x4.png')\n",
        "\n",
        "# We first convert the image to a numpy array\n",
        "im_np = np.array(im_pil)\n",
        "\n",
        "# Numpy is the standard matrix library in Python. \n",
        "# But PyTorch effectively replaces it together with the functionality needed for deep learning.\n",
        "# Still, we will enounter Numpy arrays in intermediate stages.\n",
        "# Similar to PyTorch, we can check the shape of the tensor.\n",
        "im_np.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wfIF91hTExux"
      },
      "source": [
        "# Now lets display the image. We can do this with matplotlib\n",
        "plt.imshow(im_np)   # Plot image\n",
        "plt.axis('off')     # Just turns of the axis\n",
        "plt.show()          # Finally show it"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8J-OddDFzII"
      },
      "source": [
        "# We can easily convert the image from numpy to PyTorch\n",
        "im_torch = torch.from_numpy(im_np)\n",
        "\n",
        "# Check shape\n",
        "print(im_torch.shape)\n",
        "\n",
        "# By default, PyTorch uses the data order C x H x W for images.\n",
        "# So we should move the RGB channel dimension to the first dimension\n",
        "im_torch = im_torch.permute(2, 0, 1)\n",
        "\n",
        "print(im_torch.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g5uXMhrDHsyj"
      },
      "source": [
        "# Lets check some details about the image\n",
        "print('Type:', im_torch.dtype)\n",
        "print('Min:', im_torch.min().item())\n",
        "print('Max:', im_torch.max().item())\n",
        "\n",
        "# Note that the image is a Byte tensor (uint8)\n",
        "# For most image operations it is better to first convert it to floating point with values between 0 and 1\n",
        "\n",
        "im = im_torch.float() / 255\n",
        "\n",
        "print()\n",
        "print('Type:', im.dtype)\n",
        "print('Min:', im.min().item())\n",
        "print('Max:', im.max().item())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zGI78MWdG5aD"
      },
      "source": [
        "* Lets write some convenient functions that performs reading and conversion of images in one step.\n",
        "* We also write a function for showing a torch image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kQba4fYbkDRi"
      },
      "source": [
        "# Read an image with the given name and convert it to torch\n",
        "def imread(image_file):\n",
        "    im_pil = Image.open(dataset_path + image_file)\n",
        "    im_np = np.array(im_pil, copy=False)\n",
        "    im_torch = torch.from_numpy(im_np).permute(2, 0, 1)\n",
        "    return im_torch.float()/255\n",
        "\n",
        "# Show a PyTorch image tensor\n",
        "def imshow(im, normalize=False):\n",
        "    # Fit the image to the [0, 1] range if normalize is True\n",
        "    if normalize:\n",
        "        im = (im - im.min()) / (im.max() - im.min())\n",
        "\n",
        "    # Remove redundant dimensions \n",
        "    im = im.squeeze()    # Mini excersize: check in the documentation what this function does\n",
        "\n",
        "    is_color = (im.dim() == 3)\n",
        "\n",
        "    # If there is a color channel dimension, move it to the end\n",
        "    if is_color:\n",
        "        im = im.permute(1, 2, 0)\n",
        "\n",
        "    im_np = im.numpy().clip(0,1)    # Convert to numpy and ensure the values in the range [0, 1]\n",
        "    if is_color:\n",
        "        plt.imshow(im_np)\n",
        "    else:\n",
        "        plt.imshow(im_np, cmap='gray')\n",
        "    plt.axis('off')\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TmCpcPAlkxxa"
      },
      "source": [
        "# Lets try these functions\n",
        "im = imread('0705x4.png')\n",
        "\n",
        "imshow(im)\n",
        "\n",
        "# Also check the type\n",
        "print('Type:', im.dtype)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pb3K2WBCM3Hv"
      },
      "source": [
        "### Simple image operations\n",
        "\n",
        "* Lets start by using pytorch to perform some simple operations on images.\n",
        "* You can try around with the things you learned in the previous part of the lab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MA20nOhJliXV"
      },
      "source": [
        "im = imread('0705x4.png')\n",
        "\n",
        "# Lets start with a grayscale image.\n",
        "# Color image can easily be converted to grayscale by simply averaging the color channels.\n",
        "\n",
        "im_gray = im.mean(dim=0)\n",
        "\n",
        "imshow(im_gray)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzH8N6UXlroG"
      },
      "source": [
        "# Try inverting the intensity\n",
        "im_inv = 1 - im_gray\n",
        "imshow(im_inv)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pHtZy9_7QzVZ"
      },
      "source": [
        "# Map intensities\n",
        "imshow(torch.sqrt(im_gray))\n",
        "imshow(im_gray**2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_FOKskkDYDCw"
      },
      "source": [
        "# Crop the image\n",
        "imshow(im_gray[:200, 130:310])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ykS7dPXeRKQM"
      },
      "source": [
        "# Flip the image along different axies\n",
        "imshow(im_gray.flip(dims=(1,)))\n",
        "imshow(im_gray.flip(dims=(0,)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AaRI5k3UTxhu"
      },
      "source": [
        "# Mean of original image and its reflection\n",
        "im2 = im_gray/2 + im_gray.flip(dims=(1,))/2\n",
        "imshow(im2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVnFmv_QR9zQ"
      },
      "source": [
        "# Note that this moves pixel values outside the range [0, 1]\n",
        "imshow(im_gray.exp())\n",
        "\n",
        "# You can visualize it better by setting the normalize flag\n",
        "imshow(im_gray.exp(), normalize=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8FTPUW1SS-g4"
      },
      "source": [
        "# We can try swapping the order of the color channels\n",
        "\n",
        "im_bgr = im[torch.LongTensor([2, 1, 0]), ...]\n",
        "imshow(im_bgr)\n",
        "\n",
        "im_rbg = im[torch.LongTensor([0, 2, 1]), ...]\n",
        "imshow(im_rbg)\n",
        "\n",
        "im_grb = im[torch.LongTensor([1, 0, 2]), ...]\n",
        "imshow(im_grb)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9G82RXaWA1x"
      },
      "source": [
        "### 💡 **Exercise**\n",
        "* Create and show an image where only the green channel is reflected (i.e. flipped left-to-right) while the other channels are unchanged."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7XoKkR_VGDo"
      },
      "source": [
        "# Implement your solution here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kubuKqg0Wrg9"
      },
      "source": [
        "### 💡 **Exercise**\n",
        "* Load three different images and construct a new image by taking one of the color channels from each image (ie, the red channel from the first image, the green from the second and the blue from the third).\n",
        "* **Tip:** Check the `torch.stack` and `torch.cat` functions from the previous part.\n",
        "* If the images have different sizes, extract a fix-sized crop as shown earlier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ib8poJ7QVu2t"
      },
      "source": [
        "# Implement your solution here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRliR5ELcD8o"
      },
      "source": [
        "### The convolution operation\n",
        "* Now, lets take a look at the convolution operation.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "InF1JxxHca5s"
      },
      "source": [
        "# First we need to import the nn.functional package in pytorch, which contains additional tensor operations.\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GCaOEUE7cqd9"
      },
      "source": [
        "# Lets load our image again\n",
        "im = imread('0705x4.png')\n",
        "im_gray = im.mean(dim=0)\n",
        "\n",
        "imshow(im_gray)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2YG9LiYdJRn"
      },
      "source": [
        "* The 2d convolution is performed using the `conv2d` function ([see doc](https://pytorch.org/docs/stable/nn.functional.html#conv2d))\n",
        "* It takes two main inputs, namely the input tensor and the weight (i.e. filter)\n",
        "* The input needs to be a 4-dimensional tensor with shape `B x C x H x W` where:\n",
        "    - `B` is the 'batch size', i.e. the number of images. We currently only want to convolve one image, so `B=1` for now.\n",
        "    - `C` is the number of channes (1 for grayscale and 3 for color).\n",
        "    - `H x W` are the spatial dimensions.\n",
        "* The weight need to be 4-dimensional with shape `D x C x kW x kH` where:\n",
        "    - `D` is the number of output channels.\n",
        "    - `kH x kW` is the kernel size (ie spatial size of the filter)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O44Rt8P4c2hQ"
      },
      "source": [
        "# Lets first create an averaging filter\n",
        "\n",
        "ksz = 9     # Size of the kernel\n",
        "\n",
        "# We want grayscale input and output, so C = D = 1\n",
        "weight = torch.ones(1, 1, ksz, ksz) / ksz**2\n",
        "\n",
        "# Then need to resize the image to 4d\n",
        "im_gray_4d = im_gray.view(1, 1, im_gray.shape[0], im_gray.shape[1])\n",
        "\n",
        "# Now we can apply the convolution\n",
        "im_out = F.conv2d(im_gray_4d, weight)\n",
        "\n",
        "imshow(im_out)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05ybGUkxhEEH"
      },
      "source": [
        "* We see that the result is blurry.\n",
        "* Try varying the kernel size `ksz` and see what happens.\n",
        "\n",
        "### 💡 **Exercise**\n",
        "* See how well you can remove noise with this convolution filter.\n",
        "* You can add Gaussian noise to the image first using `torch.randn` (see part1 of the lab)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cu6oqgXJgfA_"
      },
      "source": [
        "# Implement your denoising solution here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyu4hCoSh5GV"
      },
      "source": [
        "### 💡 **Exercise**\n",
        "* Compute x and y derivatives of a grayscale image using the filters shown in the lecture."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p-AxRl_niORa"
      },
      "source": [
        "# Implement your solution here\n",
        "\n",
        "deriv_x_filter = None\n",
        "\n",
        "deriv_y_filter = None\n",
        "\n",
        "# ...\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iY4ObASik_P"
      },
      "source": [
        "### In the next part we will write a deep learning example ..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QVJ1vc0KjIlt"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}