Types-of-Conv-Layers.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "06f46e74-b2d4-44ad-bcd3-c79ae0cb8516",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Dependencies\n",
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be2f8f3-1dad-4b95-ae6f-450bdefec607",
   "metadata": {},
   "source": [
    "# a) Space to Depth Layer\n",
    "\n",
    "1. In a Convolutional Neural Network, usually the input layers would be a RGB image with a Height, Width and Number of Channels like (H,W,C).\n",
    "\n",
    "2. Now if the input image shape is large, ex. `(512,512,3)`, then using this as input, followed by usual convolutional layers, we have two options to reduce the feature size immidiately, specially in a resource constrained environment like a edge hardware.\n",
    "\n",
    "3. Use a Conv2D layer with larger Kernel Size ex. (7,7) or (5,5) along with large stride ex. (2,2) or (3,3). This leads to a smaller output  following the formula for a Conv2D layer, given as:`[ (W - K + 2 * P) / S] + 1`, where, `W`: Input Width, `K`: Kernel Size, `P`: Padding Size, `S`: Stride. For example, for an input image of shape `(512,512,3)`, using a large `Kernel Size (K) of (5,5)`, a `Stride (S) of (3,3)` and a `Padding (P) of 1`, `32 Filters with Kernel K`, we can reduce the input feature dimensions from `(512,512,3)` to `(170,170,32)`.\n",
    "In this process, we lose a lot of input information, both in the space i.e. Height and Width as well as the depth i.e. the channels.\n",
    "\n",
    "4. The second option is to re-arrage the spatial information in the input in such a way that the dimension of features is reduced as well as comparatively less information is lost. This is what a `Space-to-Depth Layer` does. This layer tries to re-arrange the blocks of spatial data i.e. the width and height information into depth. More specifically, this op outputs a copy of the input tensor where values from the height and width dimensions are moved to the depth dimension. The attr block_size indicates the input block size.\n",
    "\n",
    "```Important Points for implementing Space to Depth layer:```\n",
    "\n",
    "1. Non-overlapping blocks of size `block_size x block size` are rearranged into depth at each location.\n",
    "\n",
    "2. The depth of the output tensor is `block_size * block_size * input_depth`.\n",
    "\n",
    "3. The Y, X coordinates within each block of the input become the high order component of the output channel index.\n",
    "\n",
    "4. The input tensor's height and width must be divisible by `block_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dd5b3da7-a8f5-4d73-bc17-201db27cf77c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conv2d(3, 32, kernel_size=(8, 8), stride=(8, 8), padding=(1, 1))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Normal Convolution Layer\n",
    "conv_layer = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=(8,8), stride=(8,8), padding=1)\n",
    "conv_layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "159d14b5-0855-4847-9262-9fe0d72abfde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Space to Depth Layer\n",
    "\n",
    "def space2Depth(x, block_size):\n",
    "    # Input Shape\n",
    "    n,c,h,w = x.size()\n",
    "    \n",
    "    # Input Depth - number of Input channels\n",
    "    # Output Channels - block_size * block_size * input_depth\n",
    "    # Kernel Size - (block_size, block_size)\n",
    "    s_to_d = nn.Conv2d(in_channels= c, out_channels= c * block_size**2, kernel_size=(block_size, block_size), stride=block_size)\n",
    "    \n",
    "    return s_to_d(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "912a2f32-1427-4c07-81f0-2d4ff844adaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv_out.shape:  torch.Size([1, 32, 64, 64])\n",
      "space-to-depth.shape:  torch.Size([1, 192, 64, 64])\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((3, 512,512))\n",
    "\n",
    "conv_out = conv_layer(torch.unsqueeze(x, 0))\n",
    "s2d_out = space2Depth(torch.unsqueeze(x, 0), block_size=8)\n",
    "\n",
    "print(\"conv_out.shape: \", conv_out.shape)\n",
    "print(\"space-to-depth.shape: \", s2d_out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23c178f6-02ff-493e-bd7b-809637336add",
   "metadata": {},
   "source": [
    "### Summary:\n",
    "\n",
    "From above results we can see that:\n",
    "\n",
    "1. Usin a Conv2D layer for reducing the input size leads to loss of information.\n",
    "2. Using a Space to Depth mapping, with the variable of `block_size`, we can reduce the size of the input by whatever amount we want, while still maintaining the original input information the the channels."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7b98fa-219b-44a8-8776-4e14f163bb29",
   "metadata": {},
   "source": [
    "# b) 3D Convolution\n",
    "\n",
    "3D Convolution is used when the input has more than 3 input planes/channels. For example, RGB image frames from video over timesteps. So, these frames can be represented by the shape - `(N,T,H,W,C)` where, `N`: Batch Size, `T`: Timestep, `H`: Input Height, `W`: Input Width, `C`: Number of channels. \n",
    "\n",
    "Example:\n",
    "\n",
    "Let's imagine we are making a Video Classification/Activity Recognition model. The first layer needs to take in take in `N` number of video frames at a time as to classify an action/activity in a video, the model needs to see at-least `N` number of video frames as the actions or activites span across some frames. Hence, your input would be `(Batch Size, Timesteps, Number of Input Channels, Input Frame Height, Input Frame Width)`.\n",
    "\n",
    "Say, we want to use 16 frames as input, resized to (512,512,3) i.e. RGB image. Hence, the input shape becomes: `(1, 16, 3, 512, 512)` for 1 batch_size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "01e11506-62f7-456e-8224-383fd617e9d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "conv3d_layer = nn.Conv3d(in_channels=16, out_channels=32, kernel_size=(3,3,3), stride=(2,2,2), padding=(1,1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c202fb2d-91d8-4d08-97f2-b5bee558bd83",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv3d_out.shape:  torch.Size([1, 32, 2, 256, 256])\n",
      "number of parameters in standard Conv3d:  13856\n"
     ]
    }
   ],
   "source": [
    "# Test Conv3D Layer on our sample input\n",
    "# Input is a RGB frame from video, hence (512,512,3)\n",
    "# We take 16 frames as input, hence, the 16 in timestep\n",
    "x = torch.rand((16, 3, 512,512))\n",
    "\n",
    "conv3d_out = conv3d_layer(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a Conv3D layer\n",
    "params_conv3d = sum(p.numel() for p in conv3d_layer.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"conv3d_out.shape: \", conv3d_out.shape)\n",
    "print(\"number of parameters in standard Conv3d: \", params_conv3d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a61490ce-dc44-4cb7-a595-3f85942a512b",
   "metadata": {},
   "source": [
    "# c) Pointwise Convolution (1 x 1)\n",
    "\n",
    "A pointwise convolution basically is similar to using a 2D Convolution, the only difference being that the `kernel_size` used is `(1, 1)`. In this, the \n",
    "layer does not have any spatial, but cross-channel interaction.\n",
    "\n",
    "Let's see some advantages of using this:\n",
    "1. This helps in dimensionality reduction when filter size is less than the number of input channels. For ex. if we have an input of shape `(32,512,512)` where we have 32 input channels. We want to reduce the number of channels but without losing the information. Applying a Conv2D with a (1,1) kernel, say 16 number of filters helps us to reduce the output dimension to `(16,512,512)`.\n",
    "2. Efficient low dimensional embedding or feature pooling.\n",
    "3. Applying non-linearity after convolution - Pointwise Convolution acts like a feature pooling like MaxPooling, so the activations can be applied directly on top of the Pointwise Convolution output directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bfe6ba1b-4c4c-4bc3-a1c9-979e6fe1bf11",
   "metadata": {},
   "outputs": [],
   "source": [
    "pt_conv = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5f635480-a2c5-42ee-a23b-d7c5b22a7407",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pt_conv_out.shape:  torch.Size([1, 32, 512, 512])\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((16, 512,512))\n",
    "\n",
    "pt_conv_out = pt_conv(torch.unsqueeze(x, 0))\n",
    "\n",
    "print(\"pt_conv_out.shape: \", pt_conv_out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdbb3020-e0a0-453d-8170-0ca713a210c7",
   "metadata": {},
   "source": [
    "# d) DephWise Separable Convolution\n",
    "\n",
    "Introduced in MobileNet paper, this convolution consists of two steps:\n",
    "1. Depthwise Convolution: The depthwise convolution unlike the standard convolution acts only on a single channel of the input map at a time.\n",
    "2. Pointwise Convolution: Here, we use 1 x 1 kernels to have no spatial, but cross-channel interaction.\n",
    "\n",
    "In a standard convolution, say we have input of shape `(3,7,7)`, and apply `128 filters` of size `(3,3,3)`, we get an output of `(128,5,5)`.\n",
    "\n",
    "Say, `N`: number of kernels of dimension `(h, h, D)`, `H`: input height, `W`: input width, then the total number of multiplications for a normal Conv2D layer is given as: `N * h * h * D * (H - h + 1) * (W - h + 1)`\n",
    "\n",
    "Using a depthwise separable convolution layer, the total number of multiplications reduce down to: `(h * h + N) * D * (H - h + 1) * (W - h + 1)`\n",
    "\n",
    "Hence, ratio: `(1 / N + 1 / h**2)`, if `N >> h`, `ratio = (1 / h**2)`\n",
    "\n",
    "1) Depthwise Convolution:\n",
    "First, we apply depthwise convolution to the input layer. Instead of using a single filter of size 3 x 3 x 3 in 2D convolution, we used 3 kernels, separately. Each filter has size 3 x 3 x 1. Each kernel convolves with 1 channel of the input layer (1 channel only, not all channels!). Each of such convolution provides a map of size 5 x 5 x 1. We then stack these maps together to create a 5 x 5 x 3 image. After this, we have the output with size 5 x 5 x 3. We now shrink the spatial dimensions, but the depth is still the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "58051470-2834-4889-b425-65c9b6a973c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normal Convolution\n",
    "# Filter Size: (3,3,3), total 32 filters\n",
    "normal_conv = nn.Conv2d(in_channels=10, out_channels=32, kernel_size=(3,3), stride=(1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1613028d-80f8-4129-8c22-c9065f20685a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Depthwise Separable Convolution\n",
    "\n",
    "def depthwiseSeparableConv(in_ch, out_ch):\n",
    "    # 1. Depthwise Convolution\n",
    "    # Filter Size: (3,3,1), total 10 filters, 1 per input channel\n",
    "    #  Each kernel convolves with 1 channel of the input layer\n",
    "    depthwise_convolution = nn.Conv2d(in_channels=in_ch, out_channels=in_ch, kernel_size=(3,3), groups=10)\n",
    "\n",
    "    # 2. Pointwise Convolution\n",
    "    # Now, to extend the depth, we use a cheaper 1x1 convolution with required number of output channels\n",
    "    pointwise_conv = nn.Conv2d(in_channels=in_ch, out_channels=out_ch, kernel_size=(1,1))\n",
    "    \n",
    "    depthwise_separable_conv = nn.Sequential(depthwise_convolution, pointwise_conv)\n",
    "    \n",
    "    return depthwise_separable_conv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "462b21ca-3985-457c-b090-32611b1dfbf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal_conv_out.shape:  torch.Size([1, 32, 126, 126])\n",
      "number of parameters in standard Conv2d:  2912\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((10, 128, 128))\n",
    "\n",
    "normal_conv_out = normal_conv(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a normal Conv2D\n",
    "params_conv2d = sum(p.numel() for p in normal_conv.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"normal_conv_out.shape: \", normal_conv_out.shape)\n",
    "print(\"number of parameters in standard Conv2d: \", params_conv2d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4a1f1c88-8bff-43ad-92ac-770a2eb69e8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal_conv_out.shape:  torch.Size([1, 32, 126, 126])\n",
      "number of parameters in depthwise separable convolution:  452\n"
     ]
    }
   ],
   "source": [
    "# Test Depthwise Separable Conv Layer on our sample input\n",
    "x = torch.rand((10, 128, 128))\n",
    "\n",
    "depthwise_separable_conv = depthwiseSeparableConv(in_ch=10, out_ch=32)\n",
    "depthwise_separable_conv_out = depthwise_separable_conv(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a Depthwise Separable Convolution\n",
    "params_depthwise = sum(p.numel() for p in depthwise_separable_conv.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"normal_conv_out.shape: \", depthwise_separable_conv_out.shape)\n",
    "print(\"number of parameters in depthwise separable convolution: \", params_depthwise)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8847f05b-b705-45ce-b7d2-18e6722bd612",
   "metadata": {},
   "source": [
    "# e) Transposed Convolution\n",
    "\n",
    "The normal convoluion layer, whether Conv2D or Conv2D, are usually used to reduce the input dimension and get features in the depth i.e. number of channels. Once we are done with the convolutions, the feature size becomes very small as compared to the input dimension depending on the kernel size and the stride used.\n",
    "\n",
    "But in some cases, such as Segmentation models, we want to learn the low level features but then want to Upsample the features from low dimension to a high dimension spatially. For example going from a feature size of `(32,128,128)` to an output layer of size `(1,257,257)`.\n",
    "\n",
    "In this case, we use a upsampling layer. One layer to perform up-sampling is Transpose Convolution layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "46836baf-c9e0-41b5-b9c4-ed6fcc7ade02",
   "metadata": {},
   "outputs": [],
   "source": [
    "transposed_conv = nn.ConvTranspose2d(in_channels=32, out_channels=16, kernel_size=(3,3), stride=(2,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ed22c63a-976f-4ef8-b77d-ec3c959b1a06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transconv_out.shape:  torch.Size([1, 16, 257, 257])\n",
      "number of parameters in standard TransposedConv2d:  4624\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((32, 128, 128))\n",
    "\n",
    "transconv_out = transposed_conv(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a Transposed Conv2d Layer\n",
    "params_transposedconv2d = sum(p.numel() for p in transposed_conv.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"transconv_out.shape: \", transconv_out.shape)\n",
    "print(\"number of parameters in standard TransposedConv2d: \", params_transposedconv2d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852ea954-c045-4d81-b305-3821e068a33a",
   "metadata": {
    "tags": []
   },
   "source": [
    "# f) Pixel Shuffle\n",
    "\n",
    "Pixel Shuffle is another type of layer that can be used for performing up-sampling. The difference here is that this layer performs `depth-to-space` conversion i.e. it uses the channels in a layer and re-arrange it in a way so as to obtain the desired up-sampling width and height.\n",
    "\n",
    "The advantage in this as compared to Transposed Convolution is that the transposed convolution layer up-sampling suffers with the Checkerboard Artifact problem, where the up-sampled image shows checkerboard like features. This is not desired specially in applications like super-resolution etc.\n",
    "\n",
    "Pixel Shuffle layer solves this problem by re-arranging the channel features into spatial dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "09c25b9e-231c-4a43-9de5-704af977506e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_shuffle_layer = nn.PixelShuffle(upscale_factor=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "74b4da34-fe1c-4066-ad4d-e66dc82ba434",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pixel_shuffle_out.shape:  torch.Size([1, 8, 256, 256])\n",
      "number of parameters in standard PixelShuffler layer:  0\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((32, 128, 128))\n",
    "\n",
    "pixel_shuffle_out = pixel_shuffle_layer(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a Pixel Shuffle Layer\n",
    "params_pixshuffle = sum(p.numel() for p in pixel_shuffle_layer.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"pixel_shuffle_out.shape: \", pixel_shuffle_out.shape)\n",
    "print(\"number of parameters in standard PixelShuffler layer: \", params_pixshuffle)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f368f0b1-1957-4160-b72d-848e4046cebc",
   "metadata": {},
   "source": [
    "In the above code, see that how the Pixel Shuffle layer re-arranged the input channels from `(1,32,128,128)` to `(1,8,256,256)`, thereby increasing the output in the spatially while decreasing the ouput over depth."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53a11766-33f6-4085-833c-a923c3413a00",
   "metadata": {},
   "source": [
    "# g) Dilated Convolution\n",
    "\n",
    "Dilated convolutions are similar to 2D Convolutions, except that they \"inflate\" the kernel by inserting spaces between the kernel elements. The important factor here is the `Dilation Rate (l)` that indicates that how much we want to widen the kernel by."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a6be47a9-45e2-405b-ada8-cf8d8ad62913",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normal Convolution without Dilation\n",
    "normal_conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=(3,3), stride=(1,1))\n",
    "\n",
    "# Convolution with Dilation\n",
    "dilated_conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=(3,3), stride=(1,1), dilation=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d281a982-8436-441f-bbae-2d41f60ae588",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal_conv_out.shape:  torch.Size([1, 32, 510, 510])\n",
      "dilated_conv_out.shape:  torch.Size([1, 32, 508, 508])\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((3, 512, 512))\n",
    "\n",
    "normal_conv_out = normal_conv(torch.unsqueeze(x, 0))\n",
    "dilated_conv_out = dilated_conv(torch.unsqueeze(x, 0))\n",
    "\n",
    "print(\"normal_conv_out.shape: \", normal_conv_out.shape)\n",
    "print(\"dilated_conv_out.shape: \", dilated_conv_out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebcf6c2-fc4f-4062-9274-3de1349258d0",
   "metadata": {},
   "source": [
    "# h) Grouped Convolution\n",
    "\n",
    "Introduced in `Alexnet` paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd3fc7b-74ee-4c1b-a54d-3bbaa809e6b5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ab4b5368-d05c-4d71-961c-47e7e0f4a593",
   "metadata": {},
   "source": [
    "# i) Shuffle Grouped Convolution\n",
    "\n",
    "Introduced in `ShuffleNet` paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab370529-ddbb-492c-a67f-b1a26a5a7171",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1125faac-904e-4759-bdbe-91358640b8c4",
   "metadata": {},
   "source": [
    "# j) Pointwise Grouped Convolution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bf05b0e-c492-4ac0-9480-684f72cd7805",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "68ea5925-0f27-4dea-b7a9-374e7e899f64",
   "metadata": {},
   "source": [
    "# k) Squeeze and Excite\n",
    "\n",
    "Introduced in `SqueezeNet` paper.\n",
    "\n",
    "**Layer Parameter Calculation:**\n",
    "\n",
    "`(number of input channels) * (number of filters) * (kernel_width * kernel_height)`\n",
    "\n",
    "This paper uses 3 main stratergies for designing the model architecture:\n",
    "\n",
    "**Strategy 1 - Replace 3x3 filters with 1x1 filters.**\n",
    "\n",
    "Given a budget of a certain number of convolution filters, we will choose to make the majority of these filters 1x1, since a 1x1 filter has 9X fewer parameters than a 3x3 filter.\n",
    "\n",
    "```\n",
    "in_channels = 3, out_channels (filters) = 32\n",
    "\n",
    "Parameters in a 1x1 Conv layer: (3 * 32 * (1 * 1)) = 96\n",
    "Parameters in a 1x1 Conv layer: (3 * 32 * (3 * 3)) = 864\n",
    "\n",
    "i.e `9x` more parameters than a 1x1 Conv layer.\n",
    "```\n",
    "\n",
    "**Strategy 2 - Decrease the number of input channels to 3x3 filters.**\n",
    "\n",
    "Consider a convolution layer that is comprised entirely of 3x3 filters. To maintain a small total number of parameters in a CNN, it is important not only to decrease the number of 3x3 filters, but also to decrease the number of input channels to the 3x3 filters.\n",
    "\n",
    "```\n",
    "in_channels = 3, out_channels (filters) = 32, kernel_size = (3, 3)\n",
    "Total Parameters: (3 * 32 * (3 * 3)) = 864\n",
    "\n",
    "in_channels = 3, out_channels (filters) = 64, kernel_size = (3, 3)\n",
    "Total Parameters: (3 * 64 * (3 * 3)) = 1,728\n",
    "```\n",
    "\n",
    "**Strategy 3 - Downsample late in the network so that convolution layers have large activation maps.**\n",
    "\n",
    "Most commonly, downsampling is engineered into CNN architectures by setting the (stride > 1) in some of the convolution or pooling layers. If early layers in the network have large strides, then most layers will have small activation maps.\n",
    "\n",
    "Conversely, if most layers in the network have a stride of 1, and the strides greater than 1 are concentrated toward the end4 of the network, then many layers in the network will have large activation maps. Our intuition is that large activation maps (due to delayed downsampling) can lead to higher classification accuracy, with all else held equal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6bc83638-dffe-432c-8380-ee4a39de9dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SqueezeExcite(nn.Module):\n",
    "    def __init__(self, in_planes, squeeze_planes, expand_planes):\n",
    "        super(SqueezeExcite, self).__init__()\n",
    "        \n",
    "        # Squeeze Layer - 1 x 1 Convolution (Pointwise Convolution)\n",
    "        self.conv1 = nn.Conv2d(in_channels=in_planes, out_channels=squeeze_planes, kernel_size=(1,1), stride=(1,1))\n",
    "        self.bn1 = nn.BatchNorm2d(squeeze_planes)\n",
    "\n",
    "        # Expand Layer - (1 x 1 Convolution, 3 x 3 Convolution)\n",
    "        # 1 x 1 Convolution\n",
    "        self.conv2 = nn.Conv2d(in_channels=squeeze_planes, out_channels=expand_planes, kernel_size=(1,1), stride=(1,1))\n",
    "        self.bn2 = nn.BatchNorm2d(expand_planes)\n",
    "        \n",
    "        # 3 x 3 Convolution\n",
    "        self.conv3 = nn.Conv2d(in_channels=squeeze_planes, out_channels=expand_planes, kernel_size=(3,3), stride=(1,1), padding=1)\n",
    "        self.bn3 = nn.BatchNorm2d(expand_planes)\n",
    "        \n",
    "        # Activation\n",
    "        self.relu = nn.ReLU(inplace=True)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # Input goes through Squeeze layer first\n",
    "        x = self.conv1(x)\n",
    "        x = self.bn1(x)\n",
    "        x = self.relu(x)\n",
    "        \n",
    "        # Expand Layer: Then output of Squeeze Layer goes through 2 layers in parallel\n",
    "        # 1. A Pointwise Conv Layer\n",
    "        out1 = self.conv2(x)\n",
    "        out1 = self.bn2(out1)\n",
    "        \n",
    "        # 2. A 3 x 3 Conv Layer\n",
    "        out2 = self.conv3(x)\n",
    "        out2 = self.bn3(out2)\n",
    "        \n",
    "        # Before sending the output, the outputs of expand layers is concatenated\n",
    "        out = torch.cat([out1, out2], dim=1)\n",
    "        out = self.relu(out)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2098ba69-29ff-4c66-8f34-83f70db6f34f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SqueezeExcite(\n",
       "  (conv1): Conv2d(256, 48, kernel_size=(1, 1), stride=(1, 1))\n",
       "  (bn1): BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (conv2): Conv2d(48, 192, kernel_size=(1, 1), stride=(1, 1))\n",
       "  (bn2): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (conv3): Conv2d(48, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn3): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu): ReLU(inplace=True)\n",
       ")"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "se_layer = SqueezeExcite(in_planes=256, squeeze_planes=48, expand_planes=192)\n",
    "se_layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ef97579c-a969-43c1-b218-53f7c97d640c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "se_layer_out.shape:  torch.Size([1, 384, 512, 512])\n",
      "number of parameters in SqueezeExcite layer:  105744\n"
     ]
    }
   ],
   "source": [
    "# Test Conv2D Layer on our sample input\n",
    "x = torch.rand((256, 512, 512))\n",
    "\n",
    "se_layer_out = se_layer(torch.unsqueeze(x, 0))\n",
    "\n",
    "# Number of parameters in a Pixel Shuffle Layer\n",
    "params_se_layer = sum(p.numel() for p in se_layer.parameters() if p.requires_grad)\n",
    "\n",
    "print(\"se_layer_out.shape: \", se_layer_out.shape)\n",
    "print(\"number of parameters in SqueezeExcite layer: \", params_se_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b356c7-b543-4013-b9c5-527f9aef96ce",
   "metadata": {},
   "source": [
    "As you can see above, we have `256` input channels. Following `Stratergy-1`, we replaced the 3 x 3 Convolutions with 1 x 1 Convolutions.\n",
    "This thereby helps decrease the number of input channels to 3 x 3 filters in the Expand layer following `Stratergy-2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07c33fe6-9e41-44fb-a75d-c4f9497037c6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}