01_Into_GradientDescent_Numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UF Research Computing  \n",
    "\n",
    "![UF Research Computing Logo](images/ufrc_logo.png)\n",
    "\n",
    "\n",
    "This tutorial is adapted from: [Understanding PyTorch with an example: a step-by-step tutorial](https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e) by Daniel Godoy.\n",
    "\n",
    "This tutorial was adapted for UF Resarch Computing by Matt Gitzendanner.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "**PyTorch** is one of the most widely used and fastest growing Deep Learning framworks. It is *pythonic*, meaning that it will feel familar to Python developers.\n",
    "\n",
    "I came across Daniel's post while looking for good tutorials and very much like the approach he takes in starting with a **simple problem**, a **linear regression**, fleshing that out with NumPy code and then walking through, step-by-step the parts that can be facilitated with Pytorch. As Daniel states, this provides a \"**structured**, **incremental** and **from first principles**\" approach to Deep Learning.\n",
    "\n",
    "I have converted Daniel's post to **Jupyeter notebooks** and broken the long post into several notebooks. I find Jupyter a good mehtod of playing with code. The notebooks can be run on [**HiPerGator**](https://jupyterhub.rc.ufl.edu/) (must be on UF network), [**Google Colaboratory**](https://colab.research.google.com/notebooks/), or elsewhere. While some steps will use a **GPU if present**, they should work in CPU. The example used is actually probably **slower to run on GPU**, but the idea is to give you the skills to move things in an out of GPUs when available.\n",
    "\n",
    "## A simple regression problem\n",
    "\n",
    "To focus on the goal of understanding how **PyTorch works**, Daniel Godoy sticks with a simple, familiar problem of linear regression with a single feature *x*.\n",
    "\n",
    ">   *y = a + bx + &epsilon;*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Generation\n",
    "\n",
    "We begin generating some random data. Let's create a vector of 100 points for our **feature *x*** and our **labels** ***a=1, b=2***, adding some Gaussian noise.\n",
    "\n",
    "Then we can split the made up, or *synthetic*, data into **train** and **test**, or **validation**, sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "# Data Generation\n",
    "np.random.seed(42) # Comment out for random results.\n",
    "x = np.random.rand(100, 1)\n",
    "\n",
    "# y = 1 + 2x + Gaussian noise\n",
    "y = 1 + 2 * x + .1 * np.random.randn(100, 1)\n",
    "\n",
    "\n",
    "# Shuffles the indices to split train and validation datasets\n",
    "idx = np.arange(100)\n",
    "np.random.shuffle(idx)\n",
    "\n",
    "# Uses first 80 random indices for train\n",
    "train_idx = idx[:80]\n",
    "\n",
    "# Uses the remaining indices for validation\n",
    "val_idx = idx[80:]\n",
    "\n",
    "# Generates train and validation sets\n",
    "x_train, y_train = x[train_idx], y[train_idx]\n",
    "x_val, y_val = x[val_idx], y[val_idx]\n",
    "\n",
    "# Note these data are used in the other notebooks, so I have placed a slightly\n",
    "# modified version of this cell into the file 'generate_data.py' to make\n",
    "# it easy to recreate the data as you move from one notebook to the next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, 3.0)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de7zcdX3n8dc7h1A9QEWSVC1wTtKWqtgaxdOoZVexj9ZCWh/oarfgESPqI02VLe5aV9v0YVtt6q7uWrHVsqmyRD3A2gqUteClD7VeKJeEEgGRLpILEYQQrjFaSPLZP76/Ib8zmZnzm8tv5jcz7+fjcR4z87vMfM+cz5zPfH/fmyICMzOzTi0adAHMzGy4OZGYmVlXnEjMzKwrTiRmZtYVJxIzM+uKE4mZmXXFiWSESXqTpG8OuhxVIOk2SacNuhyWjHNsSlouKSQdkT2+RtKaIsf2uBz/XtIdvXiusUskks6SdL2kH0q6P7v/NkkadNnqSfqapLeW9Ny1AN2b/dwn6fOSfq2N5yjtn4GkqVzZ9mZl/WHu8b9v5/ki4nkR8bUyytorjs0nn7vSsZk9/3clvbnB9vMlbW7nuSLijIjY1IMynSZpVxuv+42IeHa3rwtjlkgkvRO4APgQ8EzgGcA64FTgyD6XpeffMDp0bEQcDawEvgxcIelNgy0SRMTOiDi69pNtXpnb9o3asRV6Lzvm2GyokrGZ2QS8scH2c7J94yUixuIHeBrwQ+C1Cxz3E8D/AHYC9wEXAk/N9p0G7ALeCdwP3Auc2+a57wZ+AHwaeDrweWA38FB2/4Ts+A3AAeDHwF7gr7LtzyF9qB4E7gD+Y+71lwBXAY8CNwDvB77Z5PdcDgRwRN3238/Kvih7/B7ge8BjwHeA12Tbn5uV7UBWvoez7b8B/EtWhruBP+nR3y+An8vuvwn4FvAX2fvwZ8DPAl8B9gAPAHOkf0S187cDv5rd/xPgs8Cnst/rNmDGsenYbONvdgKwH5jObXsu8DiwtNVr1f9+wNeAt2b3J7K/0wPAXcDb6449F7g9+53vAn4n234U8CPgYPY77wV+Ovu7fwS4J/v5CPAT+b973Wfk94FvA48A/wd4SqH3Y1AfngF8WE/P/vBHLHDcR7KAPw44Bvi/wAdyb/x+4H3AYmA1sA94ehvn/vfsj/vU7MP1WmAyO/5vgStzZXkywHLBcncWTEcAp2QB97xs/2Wkf5BHAb8AfJ/2P6w/k21/bvb4t7KAXAT8Nukf3rOyfW+qf/7s9/zF7Pjnkz74r+7B368+kewH/lP2PjwV+Dng17L3dhnwdeAjdR+SfCL5cfb3mwA+AFzn2HRstvl3+zLwR7nHH6i9R61eq/73Y34iWQd8Fzgx+1t9te7Y3yB9aRLw8uxvfEruNXfVlfF9wHXAT5E+F9cC7290POkzckP2nh5HSljrCr0Xg/rwDODD+gbgB3XbrgUeJmXyl2V/nB8CP5s75qXAttwb/6N8gJO+/b2k4LmP0yLDAy8AHso9fjLAsse/DXyj7pz/Bfwx6R/iE8Bzcvv+vP7DlNs3L5hz25+SbT+1yXk3A2dm99/U7Plzx38E+Ise/P3qE8nOBY5/NfAvucfbmZ9I/jG372TgR45Nx2YHf7c7svuLSLW91yz0WvW/H/MTyVfI/fMGXtnovcjtvxI4P/d3rE8k3wNW5x7/OrC90fHZZ+QNuccfBC4s8l5U5VpoP+wBlko6IiL2A0TELwNkDVSLSBl7EtiSa98U6YPw5PPUzs/sA44ueO7uiPjxkzulSdLlmdNJlxIAjpE0EREHGvwO08CLJT2c23YE6VLEsuz+3bl9Oxq/FS0dn90+mJXxjcB/IQU/pN91abOTJb0Y+G+kb51Hkr7h/m2TY68Bao3mvxMRc22UM/97IumngI9mz3cM6e/5UIvzf5C7vw94Sj42+syxWUzVYvNy4OOSXkJ6fyeBf2j3ter8NC3eJ0lnkJLzz5PiYhK4ZYHnyz/HjmxbM/Wfi1bHPmmcGtv/Gfg34MwWxzxA+lb3vIg4Nvt5Whxq7G2lyLlRd847gWcDL46InyR984T0IW90/N3AP+We/9hIDc+/S7qWvZ9UJa6ZKlDueq8hfZO9Q9I08DfAecCSiDgWuLVF+QAuIV1COTEinka6Ft+w11Gk3iq1xvN2kkij1/5Atu352Xv5hmavW0GOzWIqFZsRsQ/4O1Kj+znAZRHxeLuvVedemrxPkn4C+BypDeUZ2e98Na1/53tIST7/fPcUKEdbxiaRRMTDwJ+SvkG8TtLRkhZJegHpui0RcZAUnH+RfcNF0vGSfr3A83dy7jGkD/jDko4jfdPIu490Xbjm88DPSzpH0uLs55ckPTf7lng58CeSJiWdDKxZqNw1kp4h6bysDH+Q/T5HkYJzd3bMuaRvWPnynSAp36voGODBiPixpFXA64uWoUvHkDWsSjoeeFefXrdrjs3WKh6bm0iX9V7L/N5anb7WZ4Hfk3SCpKeTOhTU1Go2u4H9We3klbn99wFLJD0tt+1S4I8kLZO0FHgv8Jniv14xY5NIACLig6Sq8H8lfbO5j3Qd992ka9Jk9+8ErpP0KPCPpG9mRbR77kdIDZsPkBrEvlC3/wLgdZIekvTRiHiMFDhnkb5V/IBDDaSQvp0dnW2/GPjfBcr8sKQfkqrHq4HfioiLACLiO8D/JH1jvo/UePit3LlfIfV4+oGkB7JtbwPeJ+kxUtB+tkAZeuFPSQ28j5AuL1zep9ftCcdmQ8MQm18nxdz3I+LG3PZOX+tvgC8CW4GbyMVx9h7/XvZcD5GS01W5/d8lJY67JD0s6adJPRo3k3pi3ZI955+1/2u2pqxRxczMrCNjVSMxM7PeKy2RSHqKpBskbVWa5+hPGxwjSR+VdKekb0s6JbfvdEl3ZPveU3+u2aA4ts3mK7NG8m/Ar0TESlIf9NOzbnJ5ZwAnZT9rgb8GkDQBfCzbfzJwdtZAZ1YFjm2znNISSSR7s4eLs5/6BpkzgU9lx14HHCvpWcAq4M6IuCvrTncZrbtGmvWNY9tsvlIHJGbfvraQpq/4WERcX3fI8cwffLMr29Zo+4ubvMZa0jc+jjrqqBc95znP6U3hbeQ9+CB8//vw+ONw5JFw/PFw3HHNj9+yZcsDEbEMyo9tx7X1Sz6uO1VqIsn6j79A0rGkmTt/ISJuzR3SaIBOtNje6DU2AhsBZmZmYvPmtmZwtjE1Nwdr16YkAun2vvvg/e+H2dnG50h6coRw2bHtuLZ+ycd1p/rSaysbcPU10nQLebuYP4rzBFIf9GbbzXpi/XrYt2/+tn370vZ2OLbNyu21tSz7toakpwK/SprVMu8q4I1ZD5eXAI9ExL3AjcBJklZkI1PPIjfwxqxbO3e2tz3PsW02X5mXtp4FbMquJS8CPhsRn5e0DiAiLiTNE7OaNOJ2H2kKaiJifzYlwhdJE8tdFBG3lVhWGzNTU7CjQYV+qtgMUI5ts5yRGtnua8lWVK2NJH95a3ISNm5s2UayJSJm+lPCQxzXVqZexLVHtttYmp1NSWN6GqR02yqJmFlzTiQ21ObmYPlyWLQo3c61MRn97Cxs3w4HD6ZbJxGzzozTwlY2YuovT+3YkR6Dk4JZP7lGYkOr3S68c3OwdGm6lCWl++3UYMwqZ9scXLkcLlmUbrcNJqBdI7Gh1U4X3rk5ePObDw1ABNizB849N913DcaGzrY5uGEtHMi+Te3bkR4DrOhvQLtGYkOrWVfdRYsObzNZv35+Eql54gk4//zSimhWnq3rDyWRmgP70vY+cyKxobVhQ+qyW+/AAYg41GYyN9d6oOGePb7EZUNoX5Ogbra9RE4kNrTqu/BOTBx+TK3NZKGBhu1OjWI2cJNNgrrZ9hI5kdhQy3fhPXiw8TE7dsDevY331RSZGsWsUlZugIm6KvnEZNreZ04kNjKa1TqkdPmqlUWLfHnLhsyKWVi1ESanAaXbVRv73tAO7rVlI2TDhsOnPZFSe8lCDhzwGBQbQitmB5I46rlGYiOj0bQn7Uwl18k08mbmRGIjpn7ak+np9s53W4lZ+5xIbKQ16iI8OQlLljQ+vuA08maW40RiI63ZLL8XXNA4wWzof4cXs6HnxnYbebOzzRvQ169Pl7OmplIScUO7WfucSGxstUowZlacL22ZmVlXSquRSDoR+BTwTOAgsDEiLqg75l1A7TvhEcBzgWUR8aCk7cBjwAFg/yCWODVrxLFtNl+Zl7b2A++MiJskHQNskfTliPhO7YCI+BDwIQBJrwL+c0Q8mHuOV0TEAyWW0awTjm2znNIubUXEvRFxU3b/MeB24PgWp5wNXFpWeWw0FFlat5vld4twbNvQ6NPCV31pI5G0HHghcH2T/ZPA6cDncpsD+JKkLZLWll1Gq77a0ro7dhw+TXw7x/SSY9sqq7bw1b4dQBxa+KqEZFJ6IpF0NOlD9I6IeLTJYa8CvlVX9T81Ik4BzgDeLullTZ5/raTNkjbv3r27p2W3aimytG67y+92o8zYdlxb1/q48FWpiUTSYtIHbS4iLm9x6FnUVf0j4p7s9n7gCmBVoxMjYmNEzETEzLJly3pTcKukIkvrtrP8bjfKjm3HtXWtjwtflZZIJAn4JHB7RHy4xXFPA14O/H1u21FZIyaSjgJeCdxaVlltODSbviS/vcgx3XJs21Do48JXZdZITgXOAX5F0s3Zz2pJ6yStyx33GuBLEfHD3LZnAN+UtBW4AfiHiPhCiWW1IdBs3qz8tCZFjukBx7ZVXx8Xviqt+29EfBNQgeMuBi6u23YXsLKUgtnQqo1Cr5/WBFLvrNq2NWvg6qvLm/rEsW1DobZOydb16XLW5FRKIiWsX+IpUmyo1E9rUuulVWtg37EDNm1KEzN6+hMbe31a+MpTpNhQaDY2pJ+9tMysMddIrPIa1Tpqy+L2q5eWmTXnGolVXqtaRz96aZlZa04kVnmtah196qVlZi04kVilzc2ldpFGpqaar4Dohnaz/nEisb5pdzLFWtvIgQOH78vXOmZnYft2OHgw3TqJmPWXG9utL1o1mLdaBre+baQm3zPLicNssFwjsb7opJvuQj2vyp7Z18yKcSKxvuikm26RnlceM2I2eE4k1heddNPdsCE1oC/EY0bMBsuJxPqiUTddKV2eatbwPjsL69YtnEw8ZsRssJxIrC/y3XQhJYeIdL9VW8fHPw6f/vT88/I8ZsRs8JxIrG9q3XSnpw8lkZpWbR218yIOJRWPGTGrDnf/tb7rZn6s+tl/zWzwXCOxvvP8WGajxYnE+m6h+bHaHQFvZoPlRGJ912p+rNoI+B07UpuIBx2aVZ8TiQ1Es/mxvFCV2fApLZFIOlHSVyXdLuk2Sec3OOY0SY9Iujn7eW9u3+mS7pB0p6T3lFVOq4ba5awdOxrvr9KgQ8e22Xxl9traD7wzIm6SdAywRdKXI+I7dcd9IyJ+M79B0gTwMeDXgF3AjZKuanCuDaG5uVTD2LkzNbCvXp3WWW82QSNUriHesW2WU1qNJCLujYibsvuPAbcDxxc8fRVwZ0TcFRGPA5cBZ5ZTUuunRm0gF17YOolI1Rp06Ng2m68vbSSSlgMvBK5vsPulkrZKukbS87JtxwN3547ZRZMPqqS1kjZL2rx79+4eltrK0KgNpH5wYr2I6o4dKSu2Hdc2TEpPJJKOBj4HvCMiHq3bfRMwHRErgb8Erqyd1uCpGv67iYiNETETETPLli3rVbGtJJ20ddSmR6maMmPbcW3DpNREImkx6YM2FxGX1++PiEcjYm92/2pgsaSlpG9pJ+YOPQG4p8yyWn900tZRpctaNY5ta9u2ObhyOVyyKN1uG50+7WX22hLwSeD2iPhwk2OemR2HpFVZefYANwInSVoh6UjgLOCqsspq5VuoV1YzS5ZU77KWY9vatm0OblgL+3YAkW5vWDsyyaTMXlunAucAt0i6Odv2h8AUQERcCLwO+F1J+4EfAWdFRAD7JZ0HfBGYAC6KiNtKLKuVqH6Z3WbyMwJDGu1+wQXllq1Djm1rz9b1cKDuA3BgX9q+omLflDpQWiKJiG/S+Hpw/pi/Av6qyb6rgatLKJr1Wau112smJ2HNGvjsZ2HPnrTtqU8tv2ydcGxb2/Y1aRxstn3IeGS7la5VA3t+ipRTT4Uf/ejQvj17PD2KjYjJJo2DzbYPGScSa0snEyo2a2Cfnp4/RYqnR7GRtXIDTNTNVDoxmbaPACcSK6zTCRUXmu23ppt1SswqbcUsrNoIk9OA0u2qjSPRPgJOJNaGTmsMrWb7zfM6JTbSVszCq7fD6w+m2xFJIuBEYm3odmXDRrP95hWtuZhZtTiRWGHd1hgWal8pWnMxs2pxIrHCuqkxFG1fKVJzMbNqcSKxwtqtMeRrIGvWuEeW2agqc2S7jaDZ2WK1hPrR7AcOND7OPbLMhp9rJFaKIqPZwT2yzEaBE4mVokhNwz2yzEaDE4mVollNY2LCPbLMRo0TiZWiWQ+vTZvcI8ts1DiRWMdajQvxmBCz8eFeW9aR+l5ZtXEhcChZFO3hZWbDzTUS64hn6jWzGicS64hn6jWzGicS64hn6jWzmtISiaQTJX1V0u2SbpN0foNjZiV9O/u5VtLK3L7tkm6RdLOkzWWV0zozzjP1OrZH1LY5uHI5XLIo3W7z0pxFldnYvh94Z0TcJOkYYIukL0fEd3LHbANeHhEPSToD2Ai8OLf/FRHxQIlltA7VGtHXr0+Xs6amUhIZk8Z1x/ao2TYHN6yFA1nD374d6TGM1LohZSmtRhIR90bETdn9x4DbgePrjrk2Ih7KHl4HnFBWeaz3xnWmXsf2CNq6/lASqTmwL22vcY2lqb60kUhaDrwQuL7FYW8Brsk9DuBLkrZIWtviuddK2ixp8+7du3tRXLPCyoptx3Wf7WvSS6S2vVZj2bcDiEM1FicToA+JRNLRwOeAd0TEo02OeQXpw/bu3OZTI+IU4Azg7ZJe1ujciNgYETMRMbNs2bIel368LLTwlM1XZmw7rvtsskkvkdr2IjWWMVZqIpG0mPRBm4uIy5sc83zgE8CZEbGntj0i7slu7weuAFaVWdZxV3ThKUsc2yNm5QaYqOs9MjGZtsPCNZYxV2avLQGfBG6PiA83OWYKuBw4JyL+Nbf9qKwRE0lHAa8Ebi2rrNbdAMNxq8k4tkfQillYtREmpwGl21UbDzW0L1RjGXNl9to6FTgHuEXSzdm2PwSmACLiQuC9wBLg4+mzyf6ImAGeAVyRbTsCuCQivlBiWcdepwMMi0yVMoIc26NoxWzzHlorN8zv1QXzayxjThEx6DL0zMzMTGze7G75nVi+PCWBetPTqUdWr88bRpK2ZMmgrxzXFbFtLrWJ7NuZaiIrN4xE1+BexLVHthuw8ADDZpevPFWKjY0Vs/Dq7fD6g+l2BJJIr3j2XwNaDzBsdflqaqpxjcRTpZiND9dI7EnNBhi2aogf56lSbEh4IGHpnEhsQa0uX3kBK6s0DyTsCycSW9BCM/2O61QpNgQ8kLAvnEhsQY0uX0mwevVgymNWmAcS9oUTiS1odhbWrEnJoyYCNm0a/cGHNuQ8kLAvnEiskKuvTskjz0vrWuUtNPWJ9YQTiRXi8SI2lBaa+sR6wuNIrBCPF7Gh1WrqE+sJ10isEI8XMbNmnEisEI8XMbNmfGnLCpuddeIws8O5RmINjdsaI2bWOScSO0yz1RLf9jYnFxsSnl+rr3xpyw7TbJLGCy88NJZkTBawsmFUm1+rNjVKbX4tcO+tkrhGYoddxmrUzRc8INGGhOfX6jvXSMZco7VGpMOTRjMekGiV4/m1+q60GomkEyV9VdLtkm6TdH6DYyTpo5LulPRtSafk9p0u6Y5s33vKKue4a3QZK2L+vFpw+OOacRyQ6NiuuHbn13J7StfKvLS1H3hnRDwXeAnwdkkn1x1zBnBS9rMW+GsASRPAx7L9JwNnNzjXeqBZjSJi/piRdes8IDHHsV1l7cyv5fVKemLBRCLpPElPb/eJI+LeiLgpu/8YcDtwfN1hZwKfiuQ64FhJzwJWAXdGxF0R8ThwWXasdaBVV95mNYrp6flrjHz84x6QWOPYrrh25tdye0pPFGkjeSZwo6SbgIuAL0YUvYKeSFoOvBC4vm7X8cDduce7sm2Ntr+4yXOvJX3jY2ocr7MsoNV667OzqUaR3w/NaxoekHi4smLbcd2lovNruT2lJxaskUTEH5Gq558E3gT8P0l/Lulni7yApKOBzwHviIhH63c3eskW2xuVb2NEzETEzLJly4oUaay0Wm8dPPVJN8qMbcd1n3i9kp4o1EaS1UB+kP3sB54O/J2kD7Y6T9Ji0gdtLiIub3DILuDE3OMTgHtabLc2FZn+3Uvlts+xPSK8XklPFGkj+T1JW4APAt8CfjEifhd4EfDaFueJVIu5PSI+3OSwq4A3Zj1cXgI8EhH3AjcCJ0laIelI4KzsWGvTQuutW/sc2xXXTi8sr1fSE0XaSJYC/yEi5g1Ti4iDkn6zxXmnAucAt0i6Odv2h8BUdv6FwNXAauBOYB9wbrZvv6TzgC8CE8BFEXFb4d/KntROG4gV5tiuqk5GtXu9kq6pzXbzSpuZmYnNmzcPuhiVMzeX2kR27kw1kQ0bfPmqE5K2RMRMv1/Xcd2GK5dnXXnrTE7Dq7f3uzRDoRdx7ZHtY8C9rWxsuBfWQHiurRHUatyIp4e3keZeWAPhGsmIaTVuBFqPKTEbeis3wPVvhoOPH9q26Ej3wiqZayQjptW4kYXGlJhVSqdzYNW3+45QO3BVOZGMmFbjRoqMKTGrhE7nwNq6HuKJ+dviCU95UjInkhHTatxIs33HHVdeecw60ukcWG5sHwgnkhGzYUPzWXo3bIDFiw8/57HH3OhuFdNpQnBj+0A4kYyYVnNnzc7CT/7k4ec8/rjbSaxiOk0InvJkIJxIRlCrubMefLDxOW4nsUrpNCF4ypOBcPffMTM11XhNds+9ZZVS+8e/dX26nDU5lZJIkYTgKU/6zjWSMdOqDcWsUlbMpuQxOZWSydb1XrmwopxIxozXH7Gh4WVwh4YTSQWVPY2J1x+xoeBlcIeG20gqZqGlcc3GhseEDA3XSCrG05iYZTwmZGg4kVSMpzExy3hMyNBwIqmYXi6N6ynjbah5TMjQcCKpkLk52Lv38O357rlFk0OtrWXHjjT5aa2txcnEhsqK2bSy4esPplsnkUoqLZFIukjS/ZJubbL/XZJuzn5ulXRA0nHZvu2Sbsn2jcUao7V//Hv2zN++ZMmh7rntJAe3tZTHsW02X5k1kouB05vtjIgPRcQLIuIFwB8A/xQR+Qk8XpHt7/sa2YPQ6B8/wNFHp9vly+ENbyieHNzWUqqLcWybPam0RBIRXweazOx0mLOBS8sqyzBo9g++VutoNK1Jq3N72dZi8zm2zeYbeBuJpEnSt7vP5TYH8CVJWyStbXzmk+evlbRZ0ubdu3eXWdRStfoH36imstC5ngpl8LqJ7VGJaxsPA08kwKuAb9VV/U+NiFOAM4C3S3pZs5MjYmNEzETEzLJly8oua2ka/eMvolly8FQoldBxbI9KXNt4qEIiOYu6qn9E3JPd3g9cAawaQLn6qvaPf2Ki+DkLJQdPhTJwjm0bCwNNJJKeBrwc+PvctqMkHVO7D7wSaNg7ZtTMzqZ/+kUceWSqiTg5VJNju03b5uDK5XDJonTriRmHSmlzbUm6FDgNWCppF/DHwGKAiLgwO+w1wJci4oe5U58BXCGpVr5LIuILZZWzapqtF1KvtqqhE0n/ObZ7rDbLb22Cxtosv+BxI0NCETHoMvTMzMxMbN483F3z6ydtbEUqXoOx7knaMoguu6MQ1y1duTybKr7O5HQahGil6kVcV6GNxHIaNZIvWdL4WHfltZHgWX6HnhNJBdU3kl9wgbvy2gjzLL9Dz4lkCLgrr400z/I79Lyw1ZCYnXXisBFVa1Dfuj5dzpqcSknEDe1Dw4nEzAZvxawTxxDzpS0zM+uKE4mZ9ZYHF44dX9oys97x4MKx5BpJH3npWxt5W9cfSiI1B/al7TaynEh6oEiCaLS64TnnpO68RxyRbp1cbOgVGVzoS18jx5e2ulQ/pUltISqY31230QqItdlpDhxofa7Z0JicajLdSTa40Je+RpJrJF0qujZ60SVuva66DbWFBhf60tdIciLpUtG10duZF8vrqtvQWjELqzamCRdRul218VBtw/NqjSRf2upSs2nf6xPHhg3FZ/X1ZIw21FoNLlzo0pcNJddIulR0bfT8fFmQGtcbmZyE1avdu8uGWKvGdM+rNZKcSLrUzoSKtVl9I+DTnz6UVGrL605Pw5o1sGnT/N5da9c6mdiQqDWm79sBxKHG9FoyWejSlw0lL2xVIXNzKZHUenHlTU+nJGSD44WtCvAiVUPHC1sNqUbjTmrdiBslEXADvA0JN6aPpdISiaSLJN0v6dYm+0+T9Iikm7Of9+b2nS7pDkl3SnpPWWUchEYDE9euhfPPb90Q7wb46nBst+BFqsZSmTWSi4HTFzjmGxHxguznfQCSJoCPAWcAJwNnSzq5xHL2VbNxJ3v2ND/HqyFWzsU4thtzY/pYKi2RRMTXgQc7OHUVcGdE3BURjwOXAWf2tHAD1O4lqokJr4ZYNY7tFtyYPpYG3UbyUklbJV0j6XnZtuOBu3PH7Mq2NSRpraTNkjbv3r27zLIW1mrurWaXqJYsadyNeNMmJ5Eh1VVsVzGuC1sxmxrWX38w3TqJjLxBJpKbgOmIWAn8JXBltr3RCIumXcsiYmNEzETEzLJly0oo5sLyiWPpUjj33Obdd5uNO7ngAq/LPkK6ju0qxLVZUQNLJBHxaETsze5fDSyWtJT0Le3E3KEnAPcMoIiF1Dee79kDTzwx/5j8/Fmtxp3UxpkcPJhunUSG06jEtllRA5siRdIzgfsiIiStIiW1PcDDwEmSVgDfB84CXj+oci6kUeN5I/m2kVrSsNE0KrFtVlSZ3X8vBf4ZeLakXZLeImmdpHXZIa8DbpW0FfgocFYk+4HzgC8CtwOfjYjbyipnvXYXnyraeN6sbcSLXQ2fYY1ts9JExMj8vOhFL4pufOYzEZOTEekiVfqZnE3hN2oAAAtMSURBVEzbm5menn98o59mz9HJ69ngAJtjCOParJVexPWge21VStG1RfI2bGg+ASO0bjTv5PXMzKrGiSSn6NoiebOzh1Y6rCe1bjTv5PXMzKrGiSSnWTvGQtOT1Gbxbfe8Tl/PzKxKnEhyiq4tMujzzMyqxIkkp521RQZ5nplZlXip3TrtjvGYm0uN4zt3pktSn/50e+d7TImZDTvXSLrQaEr4c85JtQuPCTGzceFE0oVGa4jUenB5iVwzGxdOJB2am2u9hgh4TIhV3La5tDTuJYvS7TZ/67HOuI2kQ0UThMeEWCVtm4Mb1sKBrEq9b0d6DJ723drmGkmHup1jy2ygtq4/lERqDuxL283a5ESygGaTKhZJEB4TYpW1r8k3oWbbzVpwImmhUa+sWgN6o8GERx6ZVjr0mBCrvMkm34SabTdrwYmkhWaTKq5Zk+7XDya86CJ44AEvTGUV0aoxfeUGmKj7JjQxmbabtcmN7S00awc5cCCNF4lICaTdQYhmpVuoMb3WoL51fbqcNTmVkogb2q0DTiQtTE2ly1mN1I8XAScTq5BWjem1ZJFPKGZd8KWtFhq1gzTi8SJWOW5Mtz4a60Sy0DK3tUkVJyYWfq6dO71srlWIG9Otj8pcs/0iSfdLurXJ/llJ385+rpW0Mrdvu6RbJN0saXMZ5WvVIytvdhY2bVq4ZnLcccWez4Zf1WMbcGO69VWZNZKLgdNb7N8GvDwing+8H9hYt/8VEfGCiJgpo3DtLHObn+4dDl9at5ZkvGzu2LiYCsc2kNo+Vm2EyWlA6XbVRreJWClKSyQR8XXgwRb7r42Ih7KH1wEnlFWWRoouc1u7XHXOOenxZz6TemnVryHyYJPf1FOkjJ6qx/aTVszCq7fD6w+mWycRK0lV2kjeAlyTexzAlyRtkbS2jBcsssxts8tfkMaJ5MeLeNlca6LvsW3WbwNPJJJeQfqwvTu3+dSIOAU4A3i7pJe1OH+tpM2SNu/evbvw6xZZ5rady19eNtfqdRPbnca12SAMNJFIej7wCeDMiHhyUvaIuCe7vR+4AljV7DkiYmNEzETEzLJlywq/dpFlbote/ir6fDY+uo3tTuPabBAGNiBR0hRwOXBORPxrbvtRwKKIeCy7/0rgfWWUYaFlbpsNSGx2ucrL5hpUI7bN+qnM7r+XAv8MPFvSLklvkbRO0rrskPcCS4CP13WFfAbwTUlbgRuAf4iIL5RVzlZjP3y5yhoZltg265fSaiQRcfYC+98KvLXB9ruAlYef0Rtzc6mNY+fONPbj0UfhiSfSvvrpTmq1i9rxU1MpibjWMd6qGttmgzLwxvZ+qu+FtWfPoSRSs29fWou9Znb28B5aZpXjZXNtgMYqkTTqhdXInj0p6XjKExsKtZl+9+0A4tBMv04m1idjNftvO4MDzz8ffvSjQ4nHs/xaZRWZ6desRGNVI2lncOCePZ7yxIaEZ/q1ARubRDI3B3v3dv88nvLEKiHfJqImH2PP9Gt9MhaXtmqN7EXaR2qWLEm1knqe8sQGrn71wzhw+DGe6df6aCxqJM0a2ZutM7JkCVxwweFjSBYvTrUaN77bQDVqEwHQBJ7p1wZhLGokrdZen5ycn2QmJ1MSqR9Dctxx8Nhjh2opbny3gWnW9hEH00y/Zn02FjWSZpejavNhNZsfKz+G5Oij4fHH55/vxncbCK9+aBUzFomk1VQnRQcctjOBo1mpvPqhVcxYJJLZWViz5lCbyMREetzOJSmvN2KV4dUPrWLGIpHMzaV11w9knVsOHEiP6xvLPYGjDQ2vfmgVMtKJpJYY3vCGhQcXNlsNsZZMvN6ImVljiohBl6FnZmZmYvPmNGN3kbEjUmobgZRwGq09Mj2d2k7MJG2JiJl+v24+rs16rRdxPbI1kiITNObbN9yYbmbWmZFNJAslgPr2DTemm5l1ZmQTSasE0Kh9w43pZmadGdlE0iwxfOYzjceLuDHdzKwzIztFSifL5OaX1zUzs2JKq5FIukjS/ZJubbJfkj4q6U5J35Z0Sm7f6ZLuyPa9p9MyNBu17pUPrRtViG2zKinz0tbFwOkt9p8BnJT9rAX+GkDSBPCxbP/JwNmSTu5VoRYaL2JWwMVUMLbNBqW0RBIRXwcebHHImcCnIrkOOFbSs4BVwJ0RcVdEPA5clh3bE426BXvyRWtHVWPbbFAG2UZyPHB37vGubFuj7S9u9iSS1pK+9QHslXRHg8OWAg+kuy96UaPn2bEDpC1bCpa9G7myDJzL0lyj8kwXPLfr2C4Y14NWtb8ZVLNMUO1yFY3rpgaZSNRgW7TY3lBEbAQ2tnwhafMgRiQ34rI0VqWyQNfl6Tq2i8T1oFXtbwbVLBNUvlzLu32eQSaSXcCJuccnAPcARzbZbjYsHNs2VgY5juQq4I1ZD5eXAI9ExL3AjcBJklZIOhI4KzvWbFg4tm2slFYjkXQpcBqwVNIu4I+BxQARcSFwNbAauBPYB5yb7dsv6Tzgi8AEcFFE3NZlcap0icBlaaxKZYEW5alYbA9S1f5mUM0ywYiXa6Rm/zUzs/4b2SlSzMysP5xIzMysK0OdSBaabqKfU1UUKMtsVoZvS7pW0srcvu2SbpF0s6SerGBUoDynSXoke82bJb236LkllOVduXLcKumApOOyfT19bzy9ycKqFsttlKtvMd1GmfoW23Wv2984j4ih/CE1Vn4P+BlSt8qtwMl1x6wGriH1338JcH3Rc0soyy8DT8/un1ErS/Z4O7C0z+/NacDnOzm312WpO/5VwFdKfG9eBpwC3Npkf19ipqo/VYvlNsvVl5ju9HnLju261+prnA9zjaTIdBP9mqpiweeLiGsj4qHs4XWkMQRl6eb36/t7U+ds4NIuXq+l8PQmC6laLBcuV0nn9vJ5S43tvH7H+TAnkmbTUBQ5psi5vS5L3ltI3wZqAviSpC1KU2N0q2h5Xippq6RrJD2vzXN7XRYkTZImQ/xcbnOv35uF9CtmqqpqsdxuufoR0+2WqSqxndfTOB/m9UiKTDfRk2lYelSWdKD0CtKH79/lNp8aEfdI+ingy5K+m32jKLM8NwHTEbFX0mrgStJstQN7b0hV/29FRP6bVK/fm4X0K2aqqmqx3E65+hXT7ZSppgqxndfTOB/mGkmzaSiKHFPk3F6XBUnPBz4BnBkRe2rbI+Ke7PZ+4ApS9bIbC5YnIh6NiL3Z/auBxZKWFv1delmWnLOoq/qX8N4spF8xU1VVi+XC5epjTBcuU04VYjuvt3FeRkNPP35Itam7gBUcahR6Xt0xv8H8BqUbip5bQlmmSCOdf7lu+1HAMbn71wKn9+G9eSaHBqSuAnZm71Pf35vsuKeRrukeVeZ7kz3Xcpo3QvYlZqr6U7VYrmJMVzm2BxXnAw/cLt+o1cC/knoZrM+2rQPWZfdFWkjoe8AtwEyrc0suyyeAh4Cbs5/N2fafyf5YW4HbelGWguU5L3u9raQG019udW6ZZckevwm4rO68nr83pG+F9wJPkL59vWVQMVPVn6rFchVjuoqxPcg49xQpZmbWlWFuIzEzswpwIjEzs644kZiZWVecSMzMrCtOJGZm1hUnEjMz64oTiZmZdcWJZMxI+qVs/YGnSDpK0m2SfmHQ5TLrlmN7cDwgcQxJ+jPgKcBTgV0R8YEBF8msJxzbg+FEMoYkHQncCPyYNI3EgQEXyawnHNuD4Utb4+k44GjgGNK3N7NR4dgeANdIxpCkq0grn60AnhUR5w24SGY94dgejGFe2Mo6IOmNwP6IuETSBHCtpF+JiK8Mumxm3XBsD45rJGZm1hW3kZiZWVecSMzMrCtOJGZm1hUnEjMz64oTiZmZdcWJxMzMuuJEYmZmXfn/YaRUvP4e0+wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Plot the test and train datasets\n",
    "fig, ax = plt.subplots(1,2)\n",
    "\n",
    "ax[0].scatter(x_train,y_train, color='blue')\n",
    "ax[0].set_title('Generated Data - Train')\n",
    "ax[0].set_xlabel('x')\n",
    "ax[0].set_ylabel('y')\n",
    "ax[0].set_ylim(1,3) #May need to adjust if seed not set.\n",
    "\n",
    "ax[1].scatter(x_val,y_val, color='orange')\n",
    "ax[1].set_title('Generated Data - Validatoin')\n",
    "ax[1].set_xlabel('x')\n",
    "ax[1].set_ylim(1,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we make the data, we **know** that a=1 and b=2, but let's see how close we can get to the true values using **gradient descent** and the **training data** (80 randomly selected data values)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "\n",
    "In this section, Daniel outlines the **four basic staps** for **gradient descent**:\n",
    "1. Compute the loss\n",
    "1. Compute the gradients\n",
    "1. Update the parameters (also known as weights)\n",
    "1. Repeate the process\n",
    "\n",
    "### Step 1: Compute the loss\n",
    "\n",
    "For a regrssion problem, the **loss** function used is the **Mean square error (MSE)**. MSE is the average of all squared differences between **labels** (*y*) and **predictions** (*a + bx*).\n",
    "\n",
    "Daniel Gody notes that:\n",
    "> It is worth mentioning that, if we use **all points** in the training set (N) to compute the loss, we are performing a **batch** gradient descent. If we were to use a **single point** at each time, it would be a **stochastic** gradient descent. Anything else (n) **in-between 1 and N** characterizes a **mini-batch** gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formula for MSE is:\n",
    "\n",
    "\\begin{equation*}\n",
    "MSE = \\frac{1}{N}\\sum_{i=1}^N ( y_i - \\hat{y_i} )^2\n",
    "\\end{equation*}\n",
    "\n",
    "Or, the mean of the squared differences between the observed ($y_i$) and predicted ($\\hat{y_i}$) values. And since we made the data, we also know that the predicted values can be substituted with $a-bx_i$:\n",
    "\n",
    "\\begin{equation*}\n",
    "MSE = \\frac{1}{N}\\sum_{i=1}^N ( y_i - a - bx_i )^2\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Compute the gratients\n",
    "\n",
    "A **gradient** is a **partial derivative** because it is computed with respect to (w.r.t.) a **single parameter**. We have two parameters, **a** and **b**, so we must compute two partial derivatives.\n",
    "\n",
    "A **derivative** tells you *how much* **a given quantity changes** when you *slightly vary* some **other quantity**. In our case, how much does our **MSE loss** change when we vary **each one of our two parameters**?\n",
    "\n",
    "The *right-most* part of the equations below is what you usually see in implementations of gradient descent for a simple linear regression. In the **intermediate steps**, Daniel Gody shows you **all elements** that pop-up from the applications of the [chain rule](https://en.wikipedia.org/wiki/Chain_rule), so you can know how the final expression came to be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "\\frac{\\partial{MSE}}{\\partial{a}} =\n",
    "\\frac{\\partial{MSE}}{\\partial{\\hat{y_i}}} \n",
    "\\cdot \\frac{\\partial{\\hat{y_i}}} {\\partial{a}} =\n",
    "\\frac{1}{N}\\sum_{i=1}^N 2( y_i - a - bx_i ) \\cdot (-1) =\n",
    "-2\\frac{1}{N}\\sum_{i=1}^N(y_i - \\hat{y_i})\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{\\partial{MSE}}{\\partial{b}} =\n",
    "\\frac{\\partial{MSE}}{\\partial{\\hat{y_i}}} \n",
    "\\cdot \\frac{\\partial{\\hat{y_i}}} {\\partial{ab}} =\n",
    "\\frac{1}{N}\\sum_{i=1}^N 2( y_i - a - bx_i ) \\cdot (-x_i) =\n",
    "-2\\frac{1}{N}\\sum_{i=1}^Nx_i(y_i - \\hat{y_i})\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Update the parameters\n",
    "\n",
    "In the final step, we **use the gradients to update** the parameters. Since we are trying to **minimize** our **losses**, we **reverse the sign** of the gradient for the update.\n",
    "\n",
    "There is still another parameter to consider: the **learning rate**, denoted by the *Greek letter **eta** (**&eta;**)*, which is the **multiplicative factor** that we need to apply to the gradient for the parameter update."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation*}\n",
    "a = a - \\eta\\frac{\\partial{MSE}}{\\partial{a}}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "b = b-\\eta\\frac{\\partial{MSE}}{\\partial{b}}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choosing a learning rate is beyond the scope of this tutorial.\n",
    "\n",
    "### Step 4: Rinse and repeate!\n",
    "\n",
    "Now we use the **updated parameters** to go back to **step 1** and restart the process.\n",
    "\n",
    ">An **epoch is complete whenever every point has been already used for computing the loss**. For **batch** gradient descent, this is trivial, as it uses all points for computing the loss — **one epoch** is the same as **one update**. For **stochastic** gradient descent, **one epoch** means **N updates**, while for **mini-batch** (*of size n*), **one epoch** has **N/n updates**.\n",
    "\n",
    "Repeating this process over and over, for **many epochs**, is, in a nutshell, **training** a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression in Numpy\n",
    "\n",
    "Similar to the [Image Classification training module](https://github.com/UFResearchComputing/DL_pytorch_CNN) to understand what PyTorch is doing, we will first implement the linear regression model using gradient descent in Numpy only. Once that is understood, we can move on.\n",
    "\n",
    "For training a model, we have two steps:\n",
    "1. Random intialization of parameters, or **weights** (we have only two parameters, *a* and *b*).\n",
    "1. Intialization of **hyper-parameters** (in our case, only the *learning rate* and *number of epochs*)  \n",
    "\n",
    "In general, you always want to ensur randomization of the seed. The use of 42 for a seed in these examples ensures repeatability, but is *not* what you would want in real use.\n",
    "\n",
    "**For each epoch**, we conduct the 4 steps:\n",
    "1. Compute the model's predictions -- this is the **forward pass**\n",
    "1. Compute the loss, using *predictions*, *labels* and the **loss function** for the task at hand\n",
    "1. Compute the **gradients** for every parameter.\n",
    "1. Update the parameters using our **learning rate**.\n",
    "\n",
    "\n",
    "Keep in mind that, if you don’t use **batch gradient descent** (our example does),you’ll have to write an inner loop to perform the four training steps for either each individual point (stochastic) or n points (mini-batch). We’ll see a mini-batch example later down the line.\n",
    "\n",
    "### Initialize the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.49671415] [-0.1382643]\n"
     ]
    }
   ],
   "source": [
    "# Initializes parameters \"a\" and \"b\" randomly\n",
    "np.random.seed(42)\n",
    "a = np.random.randn(1)\n",
    "b = np.random.randn(1)\n",
    "\n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets hyper-parameters \n",
    "# learning rate\n",
    "lr = 1e-1\n",
    "# Defines number of epochs\n",
    "n_epochs = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For each epoch, loop through the 4 steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.02354094] [1.96896411]\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(n_epochs):\n",
    "    \n",
    "    # Step 1: Computes our model's predicted output\n",
    "    yhat = a + b * x_train\n",
    "    \n",
    "    # Step 2: Compue loss\n",
    "    # How wrong is our model? That's the error! \n",
    "    error = (y_train - yhat)\n",
    "    # It is a regression, so it computes mean squared error (MSE)\n",
    "    loss = (error ** 2).mean()\n",
    "    \n",
    "    # Step 3: Compute gradients\n",
    "    # Computes gradients for both \"a\" and \"b\" parameters\n",
    "    a_grad = -2 * error.mean()\n",
    "    b_grad = -2 * (x_train * error).mean()\n",
    "    \n",
    "    # Step 4: Update parameters\n",
    "    # Updates parameters using gradients and the learning rate\n",
    "    a = a - lr * a_grad\n",
    "    b = b - lr * b_grad\n",
    "    \n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to make sure we haven’t done any mistakes in our code, we can use Scikit-Learn’s Linear Regression to fit the model and compare the coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.02354075] [1.96896447]\n"
     ]
    }
   ],
   "source": [
    "# Sanity Check: do we get the same results as our gradient descent?\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "linr = LinearRegression()\n",
    "linr.fit(x_train, y_train)\n",
    "print(linr.intercept_, linr.coef_[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They match up to 6 decimal places — we have a fully working implementation of linear regression using Numpy.\n",
    "\n",
    "Time to **TORCH** it :-)\n",
    "\n",
    "**[Continue to part 2 of the tutorial: 02_PyTorch_Tensors_autograd.ipynb](02_PyTorch_Tensors_autograd.ipynb)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Tensorflow-2.1",
   "language": "python",
   "name": "tensorflow-2.1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}