W3D1-Exercise3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3: Functions and Data\n",
    "\n",
    "Builds on exercises provided by programminghistorian.org. \n",
    "\n",
    "## 3.1 Functions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3.1.1**: Write a function `is_number` that uses the `try` and `except` structure to return `True` of the input is a number, or print a relevant error statement and returns `None` if the input is not a number. Test your function on several cases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 3.1.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3.1.2**: Write some test cases for your function. Did you have to change your function based on your test cases? If so, explain. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 3.1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3.1.3**: Write a function `word_freq` that takes in a list of words and counts the frequency of each word in the list. Your function should return a dictionary with the word as the key and the count as the value. \n",
    "\n",
    "Hint: Useful string methods to look at is `count()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 3.1.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3.1.4**: Write a function  `sort_freq_dict` that takes a dictionary of key value pairs that are words and their frequencies. The function should turn the dictionary into a list of (freq, word) tuples, sorted highest to lowest by frequency. The function should return this list. \n",
    "\n",
    "Hint: list methods `sort()` and `reverse()` will be useful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 3.1.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 3.1.5**: Write a function called `remove_stop_words` that takes two lists of words as input (a word list and a stop word list). Your function should return a list of words with all the words in the word list except those found in the stop word list. \n",
    "\n",
    "Note: A stop word list is a list of common words (e.g. the, a, and, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 3.1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Text Files\n",
    "\n",
    "**Question 2.2.1**: Read in the text files `stop_words.txt` and turn it into a list of words. \n",
    "\n",
    "Note: when you look at the file you see that each word is on a different line. Remember to remove the newline characters for each line/word.\n",
    "\n",
    "Stopwords from: http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 2.2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 2.2.2**: Write a function `make_wordlist` that takes a filename as the argument. The function reads the file, turning the text to lower case and removes all non alpha-numeric characters. The function returns a list of all the words.  \n",
    "\n",
    "Hint: You will need to use a regular expression for only alphanumeric characters. Use `search(pattern, text)` and `group()` methods following the example in class.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to Question 2.2.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 2.2.3**: Putting the above together allows you to read a file, clean up the words to remove unwanted characters, and count the frequency of words in the text from the file. This results in a sorted list of words. \n",
    "\n",
    "Run the code below to count the frequency of words in `Speech_2019.txt`. This is the speech given by DK Prime Minister Mette Frederiksen at the opening of parliament. Original found [here](http://www.stm.dk/_p_14878.html).\n",
    "\n",
    "Write the resulting list of words frequencies to a new text file with an appropriate name. Have a look at the file, does it seem like all your functions are working as they should? \n",
    "\n",
    "Note: it is ok if you have words that are single letters or numbers. We'll work on improving our data cleaning techniques in later classes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'make_word_list' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-10-f0f1a0ae3c3b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m#Read the file and turn it into a list of words\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mword_list\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmake_word_list\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Speech_2019.txt'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;31m#remove stop words from the list\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mword_list\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mremove_stop_words\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mword_list\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstop_words\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;31m#count the frequency of different words\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'make_word_list' is not defined"
     ]
    }
   ],
   "source": [
    "#Read the file and turn it into a list of words\n",
    "word_list = make_wordlist('Speech_2019.txt')\n",
    "#remove stop words from the list\n",
    "word_list = remove_stop_words(word_list, stop_words)\n",
    "#count the frequency of different words\n",
    "word_dict = word_freq(word_list)\n",
    "#sort the list\n",
    "sorted_words = sort_freq_dict(word_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 2.2.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 (More) Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 2.3.1**: Write a regular expression for a telephone number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Answer to Question 2.3.1* \n",
    "\n",
    "Note this cell is in markdown mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question 2.3.2**: Write a regular expression for a URL. \n",
    "\n",
    "Note: For this exercise you can use a simple solution that does not account for all possible conditions, but will capture most URLs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Answer to Question 2.3.2* \n",
    "\n",
    "Note this cell is in markdown mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Reflection\n",
    "**Question 2.4.1**: Write a brief paragraph reflecting on your experience learning programming today. What did you struggle with? What did you enjoy? What surprised you?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Answer to question 2.4.1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}