IntroductionToSelenium.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Introduction to Selenium"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Background"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<a href=\"docs.seleniumhq.org\">Selenium</a> is a free and open-source testing framework for web browser automation. Because Selenium provides a programmatic way of controlling web browser interactions, it can also be used for simple web scraping or crawling of webpages. \n",
      "\n",
      "Implemented in Java, Selenium WebDriver controls browsers by injecting Javascript calls. Selenium come with built-in support for Firefox with supplemental drivers to support other browsers. Selenium offers multiple programming language bindings including Python.\n",
      "\n",
      "This tutorial is designed to get you familiarized with Selenium and assumes you have basic working knowledge of the <a href=\"en.wikipedia.org/wiki/Document_Object_Model\">Document Object Model (DOM)</a>, which Selenium uses to identify elements on a page."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Make sure you have selenium module installed:<br><br> \n",
      "<code>pip install selenium</code><br><br>\n",
      "and <a href=\"www.mozilla.org/en-US/firefox/new/\">Firefox browser</a> installed."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Launching the Browser"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import time\n",
      "from selenium import webdriver\n",
      "from selenium.common.exceptions import NoSuchElementException"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 95
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now open a new Firefox browser."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser = webdriver.Firefox()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 81
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Navigate to Yelp's search page using <code>get()</code>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser.get(\"http://www.yelp.com\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 82
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Locating and Interacting with Elements"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we going to interact with some of the elements on that page. Selenium is able to locate elements very quickly, but if the page is loading, the element could be unavailable. Selenium will raise an Exception when you try to interact with objects that are unavailable. \n",
      "\n",
      "The default maximum timeout period to locate an element in Selenium is 0 seconds. This means that Selenium will only try once to locate an element. You can change Selenium's default timeout period so that Selenium will attempt to locate the element until it is found or the maximum timeout period is reached."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser.implicitly_wait(20)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 83
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are several different ways to locate an element on a page:\n",
      "\n",
      "* id: locate the element with the specificed @id attribute.\n",
      "\n",
      "* name: locate the element with the specificed @name attribute.\n",
      "\n",
      "* class: locate the element with the specificed @class attribute.\n",
      "\n",
      "* css: locate an element using CSS selectors.\n",
      "\n",
      "* xpath: locate an element using an XPath expression.\n",
      "\n",
      "* link: locate an element using the link name."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Looking for Pizza in Berkeley"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's try to locate the element Find search bar by id:\n",
      "<br>\n",
      "<input autocomplete=\"off\" id=\"find_desc\" maxlength=\"64\" name=\"find_desc\" placeholder=\"tacos, cheap dinner, Max\u2019s\" tabindex=\"1\" value=\"\">\n",
      "<br><br>\n",
      "Right-click the element in the browser and select inspect element. You should be taken to the source for the Find search bar. You'll discover that this search bar has <code>id=\"find_desc\"</code>."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "find = browser.find_element_by_id(\"find_desc\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 84
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Then type \"pizza\" into the search bar."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "find.send_keys(\"pizza\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 85
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's try locating the element Location search bar by name:<br>\n",
      "<input autocomplete=\"off\" id=\"dropperText_Mast\" maxlength=\"64\" name=\"find_loc\" placeholder=\"address, neighborhood, city, state or zip\" tabindex=\"2\" value=\"Berkeley, CA\">"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "loc = browser.find_element_by_name(\"find_loc\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 86
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Clear the existing text before typing in \"Berkeley, CA\"."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "loc.clear()\n",
      "loc.send_keys(\"Berkeley, CA\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 87
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Then find and click the search button."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser.find_element_by_id(\"header-search-submit\").click()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 88
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now that you have all of the search results for pizza in Berkeley, CA in the browser, how we do interact with each result? Well-designed interfaces with static content should label each element with an id or name value. With dynamic content and poorly implemented interfaces, there are times where you have to locate one element in reference to another element, or anchor element."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "items = browser.find_elements_by_xpath(\"//div[@id='super-container']/div[3]/div[3]/div/div/div/ul/li\")\n",
      "biz_names = []\n",
      "for item in items:\n",
      "    biz_names.append(item.find_element_by_class_name(\"biz-name\").text)\n",
      "    print biz_names[-1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Cheese Board Pizza\n",
        "Emilia\u2019s Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Gioia Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Sliver"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Zachary\u2019s Chicago Pizza"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Pizza Moda"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Jupiter"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Pizzahhh!"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Pieology Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Arinell Pizza"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 89
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Click on the link for the first business."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser.find_element_by_link_text(biz_names[0]).click()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 90
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's access some basic information about this business:\n",
      "\n",
      "* address\n",
      "\n",
      "* phone number\n",
      "\n",
      "* website\n",
      "\n",
      "* number of reviews\n",
      "\n",
      "* average star rating"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print browser.find_element_by_css_selector(\"address\").text\n",
      "print browser.find_element_by_class_name(\"biz-phone\").text\n",
      "print browser.find_element_by_class_name(\"biz-website\").text\n",
      "print browser.find_element_by_xpath(\"//span[@itemprop='reviewCount']\").text + \" reviews\"\n",
      "print browser.find_element_by_class_name(\"rating-very-large\").find_element_by_xpath(\"./i\").get_attribute(\"title\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1512 Shattuck Ave\n",
        "Berkeley, CA 94709\n",
        "(510) 549-3183\n",
        "cheeseboardcollective.coop"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "3238 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 91
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Go back to the previous page with all search results."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "browser.back()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 92
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's combine all of the previous code and visit each business' website to scrape some information."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for biz_name in biz_names:\n",
      "    browser.find_element_by_link_text(biz_name).click()\n",
      "    #business name\n",
      "    print biz_name\n",
      "    #business address\n",
      "    print browser.find_element_by_css_selector(\"address\").text\n",
      "    #business phone number\n",
      "    print browser.find_element_by_class_name(\"biz-phone\").text\n",
      "    #business website if it exists.\n",
      "    try:\n",
      "        print browser.find_element_by_class_name(\"biz-website\").text\n",
      "    except NoSuchElementException:\n",
      "        pass\n",
      "    #number of reviews\n",
      "    print browser.find_element_by_xpath(\"//span[@itemprop='reviewCount']\").text + \" reviews\"\n",
      "    #average business rating\n",
      "    print browser.find_element_by_class_name(\"rating-very-large\").find_element_by_xpath(\"./i\").get_attribute(\"title\")\n",
      "    print\n",
      "    browser.back()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Cheese Board Pizza\n",
        "1512 Shattuck Ave\n",
        "Berkeley, CA 94709"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 549-3183"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "cheeseboardcollective.coop"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "3238 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Emilia\u2019s Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2995 Shattuck Ave\n",
        "Berkeley, CA 94705"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 704-1794"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "emiliaspizzeria.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "289 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Gioia Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "1586 Hopkins St\n",
        "Berkeley, CA 94707"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 528-4692"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "gioiapizzeria.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "555 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Sliver"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2132 Center St\n",
        "Berkeley, CA 94704"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 356-4044"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "sliverpizzeria.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "425 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.0 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Zachary\u2019s Chicago Pizza"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "1853 Solano Ave\n",
        "Berkeley, CA 94707"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 525-5950"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "zacharys.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "1405 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Pizza Moda"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "1401 University Ave\n",
        "Berkeley, CA 94702"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 841-5200"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "pizzamoda.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "120 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Jupiter"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2181 Shattuck Ave\n",
        "Berkeley, CA 94704"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 843-8277"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "jupiterbeer.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "1953 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.0 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Pizzahhh!"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2503 Hearst Ave\n",
        "Berkeley, CA 94709"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 843-1241"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "55 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.5 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Pieology Pizzeria"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2468 Telegraph Ave\n",
        "Berkeley, CA 94704"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 649-0600"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "pieology.com"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "100 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.0 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Arinell Pizza"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "2119 Shattuck Ave\n",
        "Berkeley, CA 94704"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "(510) 841-4035"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "388 reviews"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "4.0 star rating"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 96
    }
   ],
   "metadata": {}
  }
 ]
}