week-1-simple-regression-assignment-blank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression Week 1: Simple Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will use data on house sales in King County to predict house prices using simple (one input) linear regression. You will:\n",
    "* Use graphlab SArray and SFrame functions to compute important summary statistics\n",
    "* Write a function to compute the Simple Linear Regression weights using the closed form solution\n",
    "* Write a function to make predictions of the output given the input feature\n",
    "* Turn the regression around to predict the input given the output\n",
    "* Compare two different models for predicting house prices\n",
    "\n",
    "In this notebook you will be provided with some already complete code as well as some code that you should complete yourself in order to answer quiz questions. The code we provide to complete is optional and is there to assist you with solving the problems but feel free to ignore the helper code and write your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fire up graphlab create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load house sales data\n",
    "\n",
    "Dataset is from house sales in King County, the region where the city of Seattle, WA is located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] GraphLab Create v1.8.3 started. Logging: /tmp/graphlab_server_1456673758.log\n"
     ]
    }
   ],
   "source": [
    "sales = graphlab.SFrame('kc_house_data.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split data into training and testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use seed=0 so that everyone running this notebook gets the same results.  In practice, you may set a random seed (or let GraphLab Create pick a random seed for you).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data,test_data = sales.random_split(0.8, seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Useful SFrame summary functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to make use of the closed form solution as well as take advantage of graphlab's built in functions we will review some important ones. In particular:\n",
    "* Computing the sum of an SArray\n",
    "* Computing the arithmetic average (mean) of an SArray\n",
    "* multiplying SArrays by constants\n",
    "* multiplying SArrays by other SArrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average price via method 1: 540088.141905\n",
      "average price via method 2: 540088.141905\n"
     ]
    }
   ],
   "source": [
    "# Let's compute the mean of the House Prices in King County in 2 different ways.\n",
    "prices = sales['price'] # Extract the price column of the sales SFrame -- this is now an SArray.\n",
    "\n",
    "# Recall that the arithmetic average (the mean) is the sum of the prices divided by the total number of houses:\n",
    "sum_prices = prices.sum()\n",
    "num_houses = prices.size() # When prices is an SArray, .size() returns its length.\n",
    "avg_price_1 = sum_prices / num_houses\n",
    "avg_price_2 = prices.mean() # If you just want the average, the .mean() function.\n",
    "print(\"Average price via method 1: \" + str(avg_price_1))\n",
    "print(\"Average price via method 2: \" + str(avg_price_2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see we get the same answer both ways"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sum of price squared is: 9.21732513355e+15\n"
     ]
    }
   ],
   "source": [
    "# If we want to multiply every price by 0.5 it's a simple as:\n",
    "half_prices = 0.5 * prices\n",
    "# Let's compute the sum of squares of price. We can multiply two SArrays of the same length elementwise also with \"*\".\n",
    "prices_squared = prices * prices\n",
    "sum_prices_squared = prices_squared.sum() # price_squared is an SArray of the squares and we want to add them up.\n",
    "print(\"The sum of price squared is: \" + str(sum_prices_squared))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aside: The python notation x.xxe+yy means $x.xx \\times 10^{yy} $ e.g 100 = 10^2 = 1*10^2 = 1e2 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a generic simple linear regression function "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Armed with these SArray functions we can use the closed form solution found from the lecture to compute the slope and intercept for a simple linear regression on observations stored as SArrays: input_feature, output.\n",
    "\n",
    "Complete the following function (or write your own) to compute the simple linear regression slope and intercept:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def simple_linear_regression(input_feature, output):\n",
    "    # Compute the sum of input_feature and output.\n",
    "    sum_input = input_feature.sum()\n",
    "    sum_output = output.sum()\n",
    "    # Feature size, N.\n",
    "    num_inputs = input_feature.size()\n",
    "    # Compute the product of the output and the input_feature and its sum.\n",
    "    input_output = (input_feature * output).sum()\n",
    "    # Compute the squared value of the input_feature and its sum.\n",
    "    input_squared = input_feature * input_feature\n",
    "    sum_input_squared = input_squared.sum() \n",
    "    # Use the formula for the slope.\n",
    "    slope = ((input_output - (sum_input * sum_output / num_inputs)) /\n",
    "            (sum_input_squared - (sum_input * sum_input / float(num_inputs))))\n",
    "    # Use the formula for the intercept.\n",
    "    intercept = output.mean() - (slope * input_feature.mean())\n",
    "    \n",
    "    return (intercept, slope)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test that our function works by passing it something where we know the answer. In particular we can generate a feature and then put the output exactly on a line: output = 1 + 1\\*input_feature then we know both our slope and intercept should be 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: 1.0\n",
      "Slope: 1.0\n"
     ]
    }
   ],
   "source": [
    "test_feature = graphlab.SArray(range(5))\n",
    "test_output = graphlab.SArray(1 + 1*test_feature)\n",
    "(test_intercept, test_slope) =  simple_linear_regression(test_feature, test_output)\n",
    "print \"Intercept: \" + str(test_intercept)\n",
    "print \"Slope: \" + str(test_slope)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know it works let's build a regression model for predicting price based on sqft_living. Rembember that we train on train_data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: -47116.0765749\n",
      "Slope: 281.958838568\n"
     ]
    }
   ],
   "source": [
    "sqft_intercept, sqft_slope = simple_linear_regression(train_data['sqft_living'], train_data['price'])\n",
    "\n",
    "print \"Intercept: \" + str(sqft_intercept)\n",
    "print \"Slope: \" + str(sqft_slope)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the model parameters: intercept & slope we can make predictions. Using SArrays it's easy to multiply an SArray by a constant and add a constant value. Complete the following function to return the predicted output given the input_feature, slope and intercept:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_regression_predictions(input_feature, intercept, slope):\n",
    "    # calculate the predicted values:\n",
    "    predicted_values = input_feature * slope + intercept\n",
    "    return predicted_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we can calculate a prediction given the slope and intercept let's make a prediction. Use (or alter) the following to find out the estimated price for a house with 2650 squarefeet according to the squarefeet model we estimated above.\n",
    "\n",
    "**Quiz Question: Using your Slope and Intercept from (4), What is the predicted price for a house with 2650 sqft?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated price for a house with 2650 squarefeet is $700074.85\n"
     ]
    }
   ],
   "source": [
    "my_house_sqft = 2650\n",
    "estimated_price = get_regression_predictions(my_house_sqft, sqft_intercept, sqft_slope)\n",
    "print(\"The estimated price for a house with {:d} squarefeet is ${:.2f}\".format(my_house_sqft, estimated_price))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Residual Sum of Squares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a model and can make predictions let's evaluate our model using Residual Sum of Squares (RSS). Recall that RSS is the sum of the squares of the residuals and the residuals is just a fancy word for the difference between the predicted output and the true output. \n",
    "\n",
    "Complete the following (or write your own) function to compute the RSS of a simple linear regression model given the input_feature, output, intercept and slope:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_residual_sum_of_squares(input_feature, output, intercept, slope):\n",
    "    # First get the predictions\n",
    "    predicted_values = get_regression_predictions(input_feature, intercept, slope)\n",
    "    # then compute the residuals (since we are squaring it doesn't matter which order you subtract)\n",
    "    residuals = predicted_values - output\n",
    "    # square the residuals and add them up\n",
    "    RSS = (residuals * residuals).sum()\n",
    "    return(RSS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test our get_residual_sum_of_squares function by applying it to the test model where the data lie exactly on a line. Since they lie exactly on a line the residual sum of squares should be zero!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n"
     ]
    }
   ],
   "source": [
    "print(get_residual_sum_of_squares(test_feature, test_output, test_intercept, test_slope))  # should be 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use your function to calculate the RSS on training data from the squarefeet model calculated above.\n",
    "\n",
    "**Quiz Question: According to this function and the slope and intercept from the squarefeet model What is the RSS for the simple linear regression using squarefeet to predict prices on TRAINING data?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The RSS of predicting Prices based on Square Feet is : 1.20191835632e+15\n"
     ]
    }
   ],
   "source": [
    "rss_prices_on_sqft = get_residual_sum_of_squares(train_data['sqft_living'], train_data['price'], sqft_intercept, sqft_slope)\n",
    "print('The RSS of predicting Prices based on Square Feet is : ' + str(rss_prices_on_sqft))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict the squarefeet given price"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we want to predict the squarefoot given the price? Since we have an equation y = a + b\\*x we can solve the function for x. So that if we have the intercept (a) and the slope (b) and the price (y) we can solve for the estimated squarefeet (x).\n",
    "\n",
    "Comlplete the following function to compute the inverse regression estimate, i.e. predict the input_feature given the output!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inverse_regression_predictions(output, intercept, slope):\n",
    "    # solve output = intercept + slope*input_feature for input_feature. Use this equation to compute the inverse predictions:\n",
    "    estimated_feature = (output - intercept) / slope\n",
    "    return estimated_feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a function to compute the squarefeet given the price from our simple regression model let's see how big we might expect a house that costs $800,000 to be.\n",
    "\n",
    "**Quiz Question: According to this function and the regression slope and intercept from (3) what is the estimated square-feet for a house costing $800,000?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The estimated squarefeet for a house worth $800000.00 is 3004\n"
     ]
    }
   ],
   "source": [
    "my_house_price = 800000\n",
    "estimated_squarefeet = inverse_regression_predictions(my_house_price, sqft_intercept, sqft_slope)\n",
    "print(\"The estimated squarefeet for a house worth ${:.2f} is {:.0f}\".format(my_house_price, estimated_squarefeet))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Model: estimate prices from bedrooms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have made one model for predicting house prices using squarefeet, but there are many other features in the sales SFrame. \n",
    "Use your simple linear regression function to estimate the regression parameters from predicting Prices based on number of bedrooms. Use the training data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept: 109473.180469\n",
      "Slope: 127588.952175\n"
     ]
    }
   ],
   "source": [
    "# Estimate the slope and intercept for predicting 'price' based on 'bedrooms'\n",
    "bedrooms_intercept, bedrooms_slope = simple_linear_regression(train_data[\"bedrooms\"], train_data[\"price\"])\n",
    "\n",
    "print(\"Intercept: \" + str(bedrooms_intercept))\n",
    "print(\"Slope: \" + str(bedrooms_slope))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test your Linear Regression Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have two models for predicting the price of a house. How do we know which one is better? Calculate the RSS on the TEST data (remember this data wasn't involved in learning the model). Compute the RSS from predicting prices using bedrooms and from predicting prices using squarefeet.\n",
    "\n",
    "**Quiz Question: Which model (square feet or bedrooms) has lowest RSS on TEST data? Think about why this might be the case.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The RSS of predicting Prices based on No. of bedrooms is: 3.52859832694e+20\n"
     ]
    }
   ],
   "source": [
    "# Compute RSS when using bedrooms on TEST data:\n",
    "rss_prices_on_bedrooms = get_residual_sum_of_squares(test_data['sqft_living'], test_data['price'], \n",
    "                                                     bedrooms_intercept, bedrooms_slope)\n",
    "print('The RSS of predicting Prices based on No. of bedrooms is: ' + str(rss_prices_on_bedrooms))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The RSS of predicting Prices based on Square Feet is : 2.75402936247e+14\n"
     ]
    }
   ],
   "source": [
    "# Compute RSS when using squarefeet on TEST data:\n",
    "rss_prices_on_sqft = get_residual_sum_of_squares(test_data['sqft_living'], test_data['price'], \n",
    "                                                 sqft_intercept, sqft_slope)\n",
    "print('The RSS of predicting Prices based on Square Feet is : ' + str(rss_prices_on_sqft))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}