3_EADH_intermediate.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "EADH-intermediate.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyOyfrRx+85BTgnFT12mpYhj",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/esohman/EADH/blob/main/3_EADH_intermediate.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r9vJqtdexPnP"
      },
      "source": [
        "#The Intermediate notebook\n",
        "Welcome to the intermediate notebook.\n",
        "By now you should have a solid grasp of how different data types work and how they can be used and manipulated in different situations.\n",
        "\n",
        "You should also understand for loops and list comprehension and be able to do small scale NLP projects using NLTK and/or spaCy as well as understand how to create and use regex patterns.\n",
        "\n",
        "If you feel that this is not the case, I recommend that you go back and revise the sections that you feel you do not yet quite understand. Look at the notebooks themselves and follow the links to additional resources to better understand all the different aspects of Python programming at the beginner level.\n",
        "\n",
        "Welcome to the intermediate level!\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykhy95fyHN3g"
      },
      "source": [
        "#Functions\n",
        "\n",
        "The first thing we are going to discuss is functions. Functions are a way to reuse code. Instead of writing the same or similar piece of code multiple times, you can create a function and call it every time you need it.\n",
        "\n",
        "Say you have a list of tuples that contain the heights and bases of triangles. You can create a for loop to go through that list and in the loop pass those values to your function that calculates the area of the triangle.\n",
        "\n",
        "Check out the excellent [video](https://pythonhumanities.com/lesson-10-python-functions/) on functions on Python for DH.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "We create functions with the keyword **def**, this is followed by the name of the function and then parentheses that can be left empty, but typically contain the names of the variables you want to use inside your function that have been passed to the function when the function was first called. In the code below, you pass the arguments i,j to the function tri_area. Inside that function i,j are known as h,b. As a sidenote, i and j are known as arguments when they are passed to the function, h and b are known as parameters. Some people use these interchangeably, but this is their correct meaning and there is a distinction.\n",
        "\n",
        "Functions have a return function. the return function states what it is that the function outputs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cfn7t5RdxRVc"
      },
      "source": [
        "def tri_area(h,b):\n",
        "  return (h*b)/2\n",
        "\n",
        "lst = [(2.45,4),(3.3,5.4),(4.2,2.7),(4,4),(68,45)]\n",
        "for i,j in lst:\n",
        "  print(tri_area(i,j))\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0120XpAvW5Z"
      },
      "source": [
        "A more complex example where we call another function from within a function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9g2GXQIwxflv"
      },
      "source": [
        "from math import pi\n",
        "\n",
        "def area_circle(r):\n",
        "  return pi * r ** 2\n",
        "\n",
        "def vol_cylinder(r,h):\n",
        "  return area_circle(r)*h\n",
        "\n",
        "#we can use the same list of values from the previous example\n",
        "lst = [(2.45,4),(3.3,5.4),(4.2,2.7),(4,4),(68,45)]\n",
        "\n",
        "for i,j in lst:\n",
        "  print(f'The area of the circle is {\"{:.2f}\".format(area_circle(i))} and the volume of the cylinder is {\"{:.2f}\".format(vol_cylinder(i,j))}')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "daIP02zVzRFf"
      },
      "source": [
        "###format()\n",
        "You might have noticed that we used .format() when printing. You do not have to use format here, but you can use it to specify the number of decimal places you want to show.\n",
        "\n",
        "Learn more about format() [here](https://www.w3schools.com/python/ref_string_format.asp)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ClJuZ6n3mTi"
      },
      "source": [
        "###input\n",
        "Sometimes you want to get user input. Now, there are ways of creating websites and graphical user interfaces in Python, but what if we just want a value or two for interactiveness in our script?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RVTy-RlV3zkf"
      },
      "source": [
        "def monthly_pay(hrs,hpay,extra=0): #we can set a default value to a parameter\n",
        "  return (hrs*hpay)*0.8+extra #let's assume a tax rate of 20%, this could be a parameter too\n",
        "  \n",
        "  \n",
        "hours = float(input(\"How many hours did you work this month: \"))\n",
        "hourly = float(input(\"What is your hourly pay: \"))\n",
        "\n",
        "\n",
        "e = input(\"Are you getting a bonus or similar?(y/n) \")\n",
        "\n",
        "if e == \"y\":\n",
        "   extra = float(input(\"How much: \"))\n",
        "elif e == \"n\":\n",
        "   extra = 0\n",
        "else:\n",
        "   print(\"Error\")\n",
        "\n",
        "print(f'Your monthly take-home pay is {monthly_pay(hours,hourly,extra)}')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_xB9ycV058p"
      },
      "source": [
        "##Modules, libraries, and documentation\n",
        "More and more we have started using external modules and libraries. These libraries have functions that are not built into your base Python installation. Sometimes you need to use pip install to install them (on colab you can sue pip with \"!pip install pandas\" where pandas is the name of the library you want to install. On Colab, many of the most commonly used libraries are already installed, so you rarely have to do this.)\n",
        "\n",
        "We import these libraries so that we can use them in our code. We do this by typing import and the name of the library we want. We can also import only certain parts of a library. In the functions example we could have written#\n",
        "\n",
        "```\n",
        "import math\n",
        "\n",
        "print(math.pi*r**2)\n",
        "```\n",
        "or\n",
        "```\n",
        "from math import pi\n",
        "print(pi*r**2)\n",
        "```\n",
        "we can also rename the libraries we are importing\n",
        "```\n",
        "import pandas as pd\n",
        "```\n",
        "All decent libraries come with documentation. Documentation is very important in learning to understand how to use new libraries, or how to get the most out of familiar libraries.\n",
        "\n",
        "If you are ever stumped on how to do something, the documentation of the library you are using should be one of the first places you look for more information. Stackoverflow, is another top two contender.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vOU_9DsuVv4j"
      },
      "source": [
        "#Pandas & numpy\n",
        "\n",
        "pandas is a highly useful Python library for data analysis.\n",
        "Really, anytime you are dealing with csv files, you should consider if pandas might be the best option for the task at hand.\n",
        "\n",
        "pandas is built-on numpy and using numpy mathematical functions with pandas is quick, easy, and stress-free.\n",
        "\n",
        "With pandas we can create dataframes, which are kind of like spreadsheets in that we have rows and columns of data. With pandas it is very easy to manipulate this data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yteFFSdo1qUc"
      },
      "source": [
        "## Series\n",
        "Series is like a one-column dataframe. It cannot have a column name, but it can have a series name and you can name the rows.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fB01o-So1n6h"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "my_list = [123,2134,123] # a list of integers\n",
        "\n",
        "#let's make this list into a pandas series\n",
        "my_series = pd.Series(my_list)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PPg8povcXVzu"
      },
      "source": [
        "We can test the difference between a list and that same list having been converted to a series."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XBBKEYnMV06R"
      },
      "source": [
        "print(f'My list: {my_list}\\nMy Series: \\n{my_series}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qGDzePz9Xa0r"
      },
      "source": [
        "Just like lists, we can use square brackets to access elements in the series. In this case, you can think of the index as the rown number. The default numbering starts from 0."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qslyTjBrWIIg"
      },
      "source": [
        "#accessing individual elements\n",
        "print(my_series[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0e5WK9bXnVS"
      },
      "source": [
        "We can set the index names when we create the series, or we can rename the index after creation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "elQ3jkP7WUQ7"
      },
      "source": [
        "#renaming the index\n",
        "my_series1 = pd.Series(my_list,index = [\"first\", \"second\", \"third\"])\n",
        "my_series2 = my_series.rename(index = {0:\"first\",1:\"second\",2:\"third\"}) #there is also the \"inplace\" option. Remember what it does?"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R0ox9zsWXFBo"
      },
      "source": [
        "print(f'1:\\n{my_series1}\\n2: \\n{my_series2}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-wWD3xECXzSE"
      },
      "source": [
        "Series can also be created from dictionaries in addition to lists. In this case the dictionary key becomes the index name and the value becomes the value of that index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H1ZAntD6YguC"
      },
      "source": [
        "#Creating a series from a dictionary\n",
        "dicty = {\"first\":\"lalala\",\"second\":123,\"third\":99.4}\n",
        "my_dictseries = pd.Series(dicty) #if you specify the index here, the series will only consist of the specified indexes e.g.: my_dictseries = pd.Series(dicty, index = [\"first\",\"third\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QDz2OwI-Y0Z7"
      },
      "source": [
        "print(f'Series from dict:\\n{my_dictseries}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7w188dRZsbe"
      },
      "source": [
        "##Dataframes\n",
        "Dataframes are the essence of pandas and most data analysis with Python relies on pandas dataframes. If series are like columns in a spreadsheet, dataframes are like individual sheets ro tables. We can combines series to make dataframes, we can read in csv or Excel files to create dataframes, or we can create them from dictionaries for example. We can also create dataframes from a variety of other sources such as json files.\n",
        "\n",
        "Kaggle has a [great tutorial on pandas](https://www.kaggle.com/learn/pandas) if you want to dive in deeper into basic data analysis. \n",
        "\n",
        "Kaggle also offers additional pandas tutorials of varying quality on many different topics. [This link](https://www.kaggle.com/search?q=pandas) will take you to up-to-date search results of pandas tutorials on Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-tgPDR5uZt-U"
      },
      "source": [
        "#we can merge series to create a dataframe\n",
        "df1 = pd.concat([my_series1, my_series2,my_dictseries], axis=1)\n",
        "\n",
        "#or we can create one from a dictionary (typically a dictionary of lists)\n",
        "d = {\"ex1\":[89,8,6,1,2,7,6],\"ex2\":[7,5,1,66,8,74,1]}\n",
        "df2 = pd.DataFrame(d)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "PYN-A9rkasEV"
      },
      "source": [
        "print(f'Df1:\\n{df1}\\nDf2:\\n{df2}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KxB-HRRbhFl"
      },
      "source": [
        "###loc and iloc\n",
        "We can use loc and iloc to access specific information in a dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "1O3crlN0hDf8"
      },
      "source": [
        "#access specific row using loc\n",
        "df2.iloc[3] # we are accessing index 3, which gives us the values for all columns"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c98_HFOHhiBv"
      },
      "source": [
        "df2.iloc[:,1] # we can also use iloc to access a single column\n",
        "#the first part is to say that we want all the indexes, the second number is the column index (which also starts from 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9i9GCZ8VpY77"
      },
      "source": [
        "iloc slices work just like list slices.\n",
        "\n",
        "Although iloc an loc are virtually identical, there is one huge difference: iloc is index-based and loc is name-based.\n",
        "\n",
        "Compare:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MUrLnFx1psGE"
      },
      "source": [
        "df2.iloc[3] #accessing index 3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liM631tEpwea"
      },
      "source": [
        "df2.loc[3] #accessing the index named 3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fvxGQ9_kp4zH"
      },
      "source": [
        "df1.iloc[1] #accessing the second index, i.e. index 1 for columns 0,1, and 2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NhnwUZlWqPDQ"
      },
      "source": [
        "df1.loc[1]#gets you an error"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s6jw-J98qSbu"
      },
      "source": [
        "df1.loc[\"second\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gggcxo2CrHwJ"
      },
      "source": [
        "we can also access values based on conditions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOBbf8oXqm9N"
      },
      "source": [
        "print(df2.ex2 > 10) #this gets us a series of booleans\n",
        "print(df2.loc[df2.ex2 > 10]) #this shows the values that are True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YJwRHLeqh3Gi"
      },
      "source": [
        "### Reading in a csv files\n",
        "The most common file type you will ikely be reading in is a csv file. We can simply use read_csv to create a dataframe from the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nDFXS3RdhFV0"
      },
      "source": [
        "df = pd.read_csv('https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv') \n",
        "#you can replace the path with any csv file, such as a dataset from Kaggle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "fc3Peojoi5Wm"
      },
      "source": [
        "#Head gets us the first few rows of data in our dataframe. The default number of rows is 5, but we can specify a number too.\n",
        "df.head(12)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "arnCdoQIjQY7"
      },
      "source": [
        "#Tail works the same as head but gets us the last rows\n",
        "df.tail(7)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "LnaDsocYjThU"
      },
      "source": [
        "#we can also use info and shape to get an overview of the data\n",
        "df.info()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kvHziVvC814Z"
      },
      "source": [
        "df.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oHymgT49knSg"
      },
      "source": [
        "#and you can access specific column simply by using their name as a list\n",
        "print(df[\"Country\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aL39NljzkwEf"
      },
      "source": [
        "print(df.Country) #or if only one, just like this:"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SctOuR5ipEN-"
      },
      "source": [
        "#how about a specific cell\n",
        "df.Country[121]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RwbR-vEBpN7D"
      },
      "source": [
        "#or\n",
        "df[\"Country\"][122]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8w7d9NJVlL75"
      },
      "source": [
        "## numpy\n",
        "\n",
        "When we use numpy with pandas, we have access to powerful math tools. numpy arrays are also very efficient, much more so than Python lists, so if you have a lot of data that you need to loop thorugh, it might be worth converting your lists to numpy arrays first.\n",
        "\n",
        "We are not exploring numpy in depth in this notebook, but let's look at the basics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MHP_ukedjNmZ"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "arr = np.array([1, 2, 3, 4, 5])\n",
        "\n",
        "print(arr)\n",
        "\n",
        "print(type(arr))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MKL7xIA_unXk"
      },
      "source": [
        "mylist = [5,3,4,7,8]\n",
        "\n",
        "arr2 = np.array(mylist) #this doesn't have to be a list. It can also be a tuple or other sequence\n",
        "arr2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ngLTHA7Gu0Ri"
      },
      "source": [
        "arr"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59s6IifIyFmY"
      },
      "source": [
        "These numpy arrays are 1-dimensional. Numpy arrays can have multiple dimensions though."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ju2emq6CyLZE"
      },
      "source": [
        "np0 = np.array(3) #0D\n",
        "np1 = np.array([1,2,3,4]) #1D\n",
        "np2 = np.array([[1,2,3,4],[5,6,7,8]]) #2D etc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68afkl80zdq8"
      },
      "source": [
        "#access elements\n",
        "np1[2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KIwkgjJBziFD"
      },
      "source": [
        "np2[1,3]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "quvH6vtQoyj6"
      },
      "source": [
        "The beauty of numpy is not only in its arrays, but in the copious mathematical fucntions it offers. You will use them with data analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hLYfK3FV0Ul"
      },
      "source": [
        "#Data Analysis\n",
        "For more basic pandas I recommend the Kaggle notebook linked to above, but if you need something even more basic, [W3Schools also offer a pandas tutorial](https://www.w3schools.com/python/pandas/default.asp) that deals with the very basics.\n",
        "\n",
        "\n",
        "I whole-heartedly recommend [this notebook](https://colab.research.google.com/github/Tanu-N-Prabhu/Python/blob/master/Exploratory_data_Analysis.ipynb#scrollTo=0oVZnezwQ159) for getting started with Exploratory Data Analysis. It covers all the basic functions you will need to examine a data set (like a csv file).\n",
        "\n",
        "Kaggle also has a good exploratory [data analysis tutorial](https://www.kaggle.com/kashnitsky/topic-1-exploratory-data-analysis-with-pandas) with links to further resources.\n",
        "\n",
        "Another great resource is the Kaggle pandas tutorial linked to above, and also [this pandas cheatsheet with excercises](https://www.kaggle.com/rajacsp/pandas-cheatsheet-125-exercises) from Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EBgLgLF2V0N6"
      },
      "source": [
        "#Let's check how many null values we have (empty cells or cells with a value of NaN)\n",
        "df.isnull().sum()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9OO4Q1h5rgQp"
      },
      "source": [
        "#We can also check unique values\n",
        "df.Region.unique()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X9PVx12wrrrs"
      },
      "source": [
        "#or just the number of unique values\n",
        "df.Region.nunique()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVeeDsMqrwM9"
      },
      "source": [
        "#Another useful function is groupby\n",
        "df.groupby(\"Region\").Country.count()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mP9Z9gfotDWq"
      },
      "source": [
        "###numpy with pandas\n",
        "Let's load a different dataset with numbers so that we can check out some math functions\n",
        "\n",
        "This is a dataset that lists eurovision song contest votes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JyC0QBE1subj"
      },
      "source": [
        "df = pd.read_csv('https://github.com/Spijkervet/eurovision-dataset/releases/download/2020.0/votes.csv') \n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PFU-xiHgtPAB"
      },
      "source": [
        "df1980 = df.loc[df.year == 1980] #let's create a subset of the dataset for just the year 1980\n",
        "df1980"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rmIEkAcwvkkx"
      },
      "source": [
        "df1980.total_points.mean() #we can calculate the mean of an entire column"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3PfQmR1AvvDE"
      },
      "source": [
        "df1980.total_points.median() #or the median"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fDQYYRAnVypA"
      },
      "source": [
        "#Visualization\n",
        "\n",
        "Kaggle comes thourgh again with a [great tutorial on visualizations](https://www.kaggle.com/learn/data-visualization).\n",
        "\n",
        "We will be working on many similar things in the next section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nJzJQVssV74Z"
      },
      "source": [
        "import seaborn as sns                       #visualization\n",
        "import matplotlib.pyplot as plt             #visualization\n",
        "%matplotlib inline     \n",
        "sns.set(color_codes=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kOQzh5T4uosv"
      },
      "source": [
        "###correlation matrix\n",
        "In this matrix we are looking at how well the numerical column values correlate with each other throughout all the years."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cN7to_DEtuUE"
      },
      "source": [
        "plt.figure(figsize=(10,5))\n",
        "cm = df.corr()\n",
        "sns.heatmap(cm,cmap=\"OrRd\",annot=True)\n",
        "cm #why are we only getting a correlation matrix for 3 columns?"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aNUlpa2DwKGF"
      },
      "source": [
        "plt.figure(figsize=(10,8),dpi =150)\n",
        "sns.scatterplot(data = df , x = 'year',y= 'total_points',hue = 'to_country')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B3_3JlLrwu_B"
      },
      "source": [
        "## bad visualization\n",
        "This is a great example of a terrible visualization.\n",
        "1. It is a technically a bad visualization because it is hard to see anything and the legend covers part of the plot and is very long\n",
        "2. It is also practically bad because we do not gain much insights from it\n",
        "3. It is also ugly because there are so many countries that the difference in color is almost impossible to differentiate in the graph.\n",
        "\n",
        "Perhaps we can fix it.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aXyFZzoeyLkS"
      },
      "source": [
        "temp = df.groupby(\"year\").total_points.mean()\n",
        "temp = pd.DataFrame(temp)\n",
        "temp"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c9k_xo3QxvtO"
      },
      "source": [
        "plt.figure(figsize=(10,8),dpi =150)\n",
        "sns.scatterplot(data = temp , x = 'year',y='total_points')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I07t1G8k3Dph"
      },
      "source": [
        "That's better! We can see that it is quite likely there have been four different voting systems in use.\n",
        "\n",
        "For more visualization with seaborn and matplotlib + pandas, take a look at [this tutorial](https://www.kaggle.com/learn/data-visualization) on Kaggle."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SxoJ_CeB-glg"
      },
      "source": [
        "# ALL DONE!\n",
        "You have now completed the intermediate notebook and can move onto the advanced one!"
      ]
    }
  ]
}