WQP_AOI_service_tests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assessing WQP API's via AOI's\n",
    "- Emilio Mayorga, UW-APL. 2018-4-5\n",
    "- Run with Python 2.7 using [`odm2client` conda environment](https://github.com/BiG-CZ/wshp2017_tutorial_content/blob/master/clientenvironment.yml), the one created for the [BiGCZ-ODM2 workshop in Nov. 2017](https://github.com/BiG-CZ/bigcz_wshp2017)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code used to calculate the area in $km^2$ of an AOI bounding box\n",
    "This AOI is specified in lat-lon coordinates as `bbox = (ymin, xmin, ymax, xmax)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from shapely.geometry import Polygon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From https://stackoverflow.com/questions/45733838/how-to-calculate-area-of-a-polygon-with-latitude-and-longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_polygon_vertices(ymin, xmin, ymax, xmax):\n",
    "    # list of bounding box vertices, to form a closed polygon\n",
    "    box_points = [(ymin, xmin), (ymax, xmin), (ymax, xmax), (ymin, xmax), (ymin, xmin)]\n",
    "    return box_points\n",
    "\n",
    "def switch_to_xy_coordinates(coordinates):\n",
    "    # returns x,y coordinates in km\n",
    "    earth_radius = 6371.0  # in km\n",
    "    lat_dist = np.pi * earth_radius / 180.0\n",
    "\n",
    "    latitudes, longitudes = zip(*coordinates)\n",
    "    y = (lat * lat_dist for lat in latitudes)\n",
    "    x = (lon * lat_dist * np.cos(np.radians(lat)) for lat, lon in zip(latitudes, longitudes))\n",
    "    return list(zip(x, y))\n",
    "\n",
    "def compute_polygon_area(coordinates):\n",
    "    # returns area in km2\n",
    "    xy_coordinates = switch_to_xy_coordinates(coordinates)\n",
    "    return Polygon(xy_coordinates).area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WQP first tests\n",
    "[Water Quality Portal Web Services Guide](https://www.waterqualitydata.us/webservices_documentation/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict\n",
    "from StringIO import StringIO\n",
    "import requests as r\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# url = \"https://www.waterqualitydata.us/data/Station/search?characteristicName=Caffeine&bBox=-92.8,44.2,-88.9,46.0&mimeType=tsv&sorted=no\"\n",
    "wsurl = \"https://www.waterqualitydata.us/data/Station/search\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set AOI request parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use this scheme to generate a square bounding box centered on the specified lat-lon point\n",
    "# and with the specified width (in degrees)\n",
    "width_deg = 1.0\n",
    "\n",
    "latctr, lonctr = 40.1, -75.5  # just north of the Schuykil river near Philly\n",
    "# latctr, lonctr = 41.1, -75.5  # 1 deg north of the above PA/DRB center point\n",
    "# latctr, lonctr = 46.5, -123.0  # About halfway between Olympia, WA and Portland, OR\n",
    "# latctr, lonctr = 42.0, -93.0  # Iowa\n",
    "# latctr, lonctr = 30.0, -97.5  # Texas, south of Austin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = (latctr-0.5*width_deg, lonctr-0.5*width_deg, latctr+0.5*width_deg, lonctr+0.5*width_deg)\n",
    "ymin,xmin,ymax,xmax = bbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BBOX: (39.6, -76.0, 40.6, -75.0).  AOI area: 9457 km2\n"
     ]
    }
   ],
   "source": [
    "print(\"BBOX: {0}.  AOI area: {1:.0f} km2\".format(bbox, compute_polygon_area(to_polygon_vertices(*bbox))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run WQP API calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#characteristicName=\"Caffeine\",\n",
    "params = dict(\n",
    "    bBox=\"{0:.3f},{1:.3f},{2:.3f},{3:.3f}\",\n",
    "    mimeType=\"tsv\",\n",
    "    sorted=\"no\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "params['bBox'] = params['bBox'].format(xmin, ymin, xmax, ymax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bBox': '-76.000,39.600,-75.000,40.600', 'mimeType': 'tsv', 'sorted': 'no'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.09 s, sys: 140 ms, total: 1.23 s\n",
      "Wall time: 17.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "resp = r.get(wsurl, params=params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mayorga/miniconda/envs/odm2client/lib/python2.7/site-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (5,10) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(StringIO(resp.content), sep='\\t')  # coerce_float=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19528, 36)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OrganizationIdentifier</th>\n",
       "      <th>OrganizationFormalName</th>\n",
       "      <th>MonitoringLocationIdentifier</th>\n",
       "      <th>MonitoringLocationName</th>\n",
       "      <th>MonitoringLocationTypeName</th>\n",
       "      <th>MonitoringLocationDescriptionText</th>\n",
       "      <th>HUCEightDigitCode</th>\n",
       "      <th>DrainageAreaMeasure/MeasureValue</th>\n",
       "      <th>DrainageAreaMeasure/MeasureUnitCode</th>\n",
       "      <th>ContributingDrainageAreaMeasure/MeasureValue</th>\n",
       "      <th>...</th>\n",
       "      <th>CountyCode</th>\n",
       "      <th>AquiferName</th>\n",
       "      <th>FormationTypeText</th>\n",
       "      <th>AquiferTypeName</th>\n",
       "      <th>ConstructionDateText</th>\n",
       "      <th>WellDepthMeasure/MeasureValue</th>\n",
       "      <th>WellDepthMeasure/MeasureUnitCode</th>\n",
       "      <th>WellHoleDepthMeasure/MeasureValue</th>\n",
       "      <th>WellHoleDepthMeasure/MeasureUnitCode</th>\n",
       "      <th>ProviderName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>USGS-MD</td>\n",
       "      <td>USGS Maryland Water Science Center</td>\n",
       "      <td>MD006-394205075514101</td>\n",
       "      <td>CE Ae  75</td>\n",
       "      <td>Well</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2060002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Piedmont and Blue Ridge crystalline-rock aquifers</td>\n",
       "      <td>Gneisses, On Garret Island, Near Elk Milles, &amp;...</td>\n",
       "      <td>Unconfined single aquifer</td>\n",
       "      <td>20100609.0</td>\n",
       "      <td>240.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>240.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>NWIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>USGS-MD</td>\n",
       "      <td>USGS Maryland Water Science Center</td>\n",
       "      <td>MD006-394205075514301</td>\n",
       "      <td>CE Ae  74</td>\n",
       "      <td>Well</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2060002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Piedmont and Blue Ridge crystalline-rock aquifers</td>\n",
       "      <td>Gneisses, On Garret Island, Near Elk Milles, &amp;...</td>\n",
       "      <td>Unconfined single aquifer</td>\n",
       "      <td>20100608.0</td>\n",
       "      <td>340.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>340.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>NWIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>USGS-MD</td>\n",
       "      <td>USGS Maryland Water Science Center</td>\n",
       "      <td>MD006-394207075514501</td>\n",
       "      <td>CE Ae  73</td>\n",
       "      <td>Well</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2060002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Piedmont and Blue Ridge crystalline-rock aquifers</td>\n",
       "      <td>Gneisses, On Garret Island, Near Elk Milles, &amp;...</td>\n",
       "      <td>Unconfined single aquifer</td>\n",
       "      <td>20100604.0</td>\n",
       "      <td>380.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>380.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>NWIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>USGS-MD</td>\n",
       "      <td>USGS Maryland Water Science Center</td>\n",
       "      <td>MD006-394208075514901</td>\n",
       "      <td>CE Ae  78</td>\n",
       "      <td>Well</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2060002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Piedmont and Blue Ridge crystalline-rock aquifers</td>\n",
       "      <td>Gneisses, On Garret Island, Near Elk Milles, &amp;...</td>\n",
       "      <td>Unconfined single aquifer</td>\n",
       "      <td>19920425.0</td>\n",
       "      <td>450.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>450.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>NWIS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>USGS-MD</td>\n",
       "      <td>USGS Maryland Water Science Center</td>\n",
       "      <td>MD006-394208075514902</td>\n",
       "      <td>CE Ae  79</td>\n",
       "      <td>Well</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2060002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Piedmont and Blue Ridge crystalline-rock aquifers</td>\n",
       "      <td>Gneisses, On Garret Island, Near Elk Milles, &amp;...</td>\n",
       "      <td>Unconfined single aquifer</td>\n",
       "      <td>19920426.0</td>\n",
       "      <td>450.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>450.0</td>\n",
       "      <td>ft</td>\n",
       "      <td>NWIS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  OrganizationIdentifier              OrganizationFormalName  \\\n",
       "0                USGS-MD  USGS Maryland Water Science Center   \n",
       "1                USGS-MD  USGS Maryland Water Science Center   \n",
       "2                USGS-MD  USGS Maryland Water Science Center   \n",
       "3                USGS-MD  USGS Maryland Water Science Center   \n",
       "4                USGS-MD  USGS Maryland Water Science Center   \n",
       "\n",
       "  MonitoringLocationIdentifier MonitoringLocationName  \\\n",
       "0        MD006-394205075514101              CE Ae  75   \n",
       "1        MD006-394205075514301              CE Ae  74   \n",
       "2        MD006-394207075514501              CE Ae  73   \n",
       "3        MD006-394208075514901              CE Ae  78   \n",
       "4        MD006-394208075514902              CE Ae  79   \n",
       "\n",
       "  MonitoringLocationTypeName MonitoringLocationDescriptionText  \\\n",
       "0                       Well                               NaN   \n",
       "1                       Well                               NaN   \n",
       "2                       Well                               NaN   \n",
       "3                       Well                               NaN   \n",
       "4                       Well                               NaN   \n",
       "\n",
       "   HUCEightDigitCode  DrainageAreaMeasure/MeasureValue  \\\n",
       "0            2060002                               NaN   \n",
       "1            2060002                               NaN   \n",
       "2            2060002                               NaN   \n",
       "3            2060002                               NaN   \n",
       "4            2060002                               NaN   \n",
       "\n",
       "  DrainageAreaMeasure/MeasureUnitCode  \\\n",
       "0                                 NaN   \n",
       "1                                 NaN   \n",
       "2                                 NaN   \n",
       "3                                 NaN   \n",
       "4                                 NaN   \n",
       "\n",
       "   ContributingDrainageAreaMeasure/MeasureValue     ...      CountyCode  \\\n",
       "0                                           NaN     ...            15.0   \n",
       "1                                           NaN     ...            15.0   \n",
       "2                                           NaN     ...            15.0   \n",
       "3                                           NaN     ...            15.0   \n",
       "4                                           NaN     ...            15.0   \n",
       "\n",
       "                                         AquiferName  \\\n",
       "0  Piedmont and Blue Ridge crystalline-rock aquifers   \n",
       "1  Piedmont and Blue Ridge crystalline-rock aquifers   \n",
       "2  Piedmont and Blue Ridge crystalline-rock aquifers   \n",
       "3  Piedmont and Blue Ridge crystalline-rock aquifers   \n",
       "4  Piedmont and Blue Ridge crystalline-rock aquifers   \n",
       "\n",
       "                                   FormationTypeText  \\\n",
       "0  Gneisses, On Garret Island, Near Elk Milles, &...   \n",
       "1  Gneisses, On Garret Island, Near Elk Milles, &...   \n",
       "2  Gneisses, On Garret Island, Near Elk Milles, &...   \n",
       "3  Gneisses, On Garret Island, Near Elk Milles, &...   \n",
       "4  Gneisses, On Garret Island, Near Elk Milles, &...   \n",
       "\n",
       "             AquiferTypeName ConstructionDateText  \\\n",
       "0  Unconfined single aquifer           20100609.0   \n",
       "1  Unconfined single aquifer           20100608.0   \n",
       "2  Unconfined single aquifer           20100604.0   \n",
       "3  Unconfined single aquifer           19920425.0   \n",
       "4  Unconfined single aquifer           19920426.0   \n",
       "\n",
       "  WellDepthMeasure/MeasureValue WellDepthMeasure/MeasureUnitCode  \\\n",
       "0                         240.0                               ft   \n",
       "1                         340.0                               ft   \n",
       "2                         380.0                               ft   \n",
       "3                         450.0                               ft   \n",
       "4                         450.0                               ft   \n",
       "\n",
       "  WellHoleDepthMeasure/MeasureValue  WellHoleDepthMeasure/MeasureUnitCode  \\\n",
       "0                             240.0                                    ft   \n",
       "1                             340.0                                    ft   \n",
       "2                             380.0                                    ft   \n",
       "3                             450.0                                    ft   \n",
       "4                             450.0                                    ft   \n",
       "\n",
       "  ProviderName  \n",
       "0         NWIS  \n",
       "1         NWIS  \n",
       "2         NWIS  \n",
       "3         NWIS  \n",
       "4         NWIS  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrganizationIdentifier                                                                       USGS-MD\n",
       "OrganizationFormalName                                            USGS Maryland Water Science Center\n",
       "MonitoringLocationIdentifier                                                   MD006-394205075514101\n",
       "MonitoringLocationName                                                                     CE Ae  75\n",
       "MonitoringLocationTypeName                                                                      Well\n",
       "MonitoringLocationDescriptionText                                                                NaN\n",
       "HUCEightDigitCode                                                                            2060002\n",
       "DrainageAreaMeasure/MeasureValue                                                                 NaN\n",
       "DrainageAreaMeasure/MeasureUnitCode                                                              NaN\n",
       "ContributingDrainageAreaMeasure/MeasureValue                                                     NaN\n",
       "ContributingDrainageAreaMeasure/MeasureUnitCode                                                  NaN\n",
       "LatitudeMeasure                                                                              39.7013\n",
       "LongitudeMeasure                                                                            -75.8615\n",
       "SourceMapScaleNumeric                                                                          24000\n",
       "HorizontalAccuracyMeasure/MeasureValue                                                            .1\n",
       "HorizontalAccuracyMeasure/MeasureUnitCode                                                    seconds\n",
       "HorizontalCollectionMethodName                     Mapping grade GPS unit (handheld accuracy rang...\n",
       "HorizontalCoordinateReferenceSystemDatumName                                                   NAD83\n",
       "VerticalMeasure/MeasureValue                                                                     391\n",
       "VerticalMeasure/MeasureUnitCode                                                                 feet\n",
       "VerticalAccuracyMeasure/MeasureValue                                                              10\n",
       "VerticalAccuracyMeasure/MeasureUnitCode                                                         feet\n",
       "VerticalCollectionMethodName                                      Interpolated from topographic map.\n",
       "VerticalCoordinateReferenceSystemDatumName                                                    NGVD29\n",
       "CountryCode                                                                                       US\n",
       "StateCode                                                                                         24\n",
       "CountyCode                                                                                        15\n",
       "AquiferName                                        Piedmont and Blue Ridge crystalline-rock aquifers\n",
       "FormationTypeText                                  Gneisses, On Garret Island, Near Elk Milles, &...\n",
       "AquiferTypeName                                                            Unconfined single aquifer\n",
       "ConstructionDateText                                                                     2.01006e+07\n",
       "WellDepthMeasure/MeasureValue                                                                    240\n",
       "WellDepthMeasure/MeasureUnitCode                                                                  ft\n",
       "WellHoleDepthMeasure/MeasureValue                                                                240\n",
       "WellHoleDepthMeasure/MeasureUnitCode                                                              ft\n",
       "ProviderName                                                                                    NWIS\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MonitoringLocationIdentifier</th>\n",
       "      <th>MonitoringLocationName</th>\n",
       "      <th>OrganizationIdentifier</th>\n",
       "      <th>OrganizationFormalName</th>\n",
       "      <th>LongitudeMeasure</th>\n",
       "      <th>LatitudeMeasure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19518</th>\n",
       "      <td>STROUD-WELL7</td>\n",
       "      <td>Stroud Preserve: Well7</td>\n",
       "      <td>STROUD</td>\n",
       "      <td>Stroud Water Research Center (Pennsylvania)</td>\n",
       "      <td>-75.653158</td>\n",
       "      <td>39.944822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19519</th>\n",
       "      <td>STROUD-WELL8</td>\n",
       "      <td>Stroud Preserve: Well8</td>\n",
       "      <td>STROUD</td>\n",
       "      <td>Stroud Water Research Center (Pennsylvania)</td>\n",
       "      <td>-75.653400</td>\n",
       "      <td>39.945586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19520</th>\n",
       "      <td>STROUD-WELL9</td>\n",
       "      <td>Stroud Preserve: Well9</td>\n",
       "      <td>STROUD</td>\n",
       "      <td>Stroud Water Research Center (Pennsylvania)</td>\n",
       "      <td>-75.653531</td>\n",
       "      <td>39.945444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19521</th>\n",
       "      <td>WWMD_VA-USGS01458500</td>\n",
       "      <td>USGS 1458500  Delaware River at Frenchtown NJ</td>\n",
       "      <td>WWMD_VA</td>\n",
       "      <td>Virginia - World Water Monitoring Day</td>\n",
       "      <td>-75.065000</td>\n",
       "      <td>40.526111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19522</th>\n",
       "      <td>WWMD_VA-USGS01467200</td>\n",
       "      <td>USGS 1467200  Delaware R at Ben Franklin Bridg...</td>\n",
       "      <td>WWMD_VA</td>\n",
       "      <td>Virginia - World Water Monitoring Day</td>\n",
       "      <td>-75.137399</td>\n",
       "      <td>39.954002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19523</th>\n",
       "      <td>WWMD_VA-USGS01477050</td>\n",
       "      <td>USGS 1477050  Delaware River at Chester, PA</td>\n",
       "      <td>WWMD_VA</td>\n",
       "      <td>Virginia - World Water Monitoring Day</td>\n",
       "      <td>-75.366302</td>\n",
       "      <td>39.836779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19524</th>\n",
       "      <td>USGS-01467150</td>\n",
       "      <td>Cooper River at Haddonfield NJ</td>\n",
       "      <td>USGS-NJ</td>\n",
       "      <td>USGS New Jersey Water Science Center</td>\n",
       "      <td>-75.021389</td>\n",
       "      <td>39.903056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19525</th>\n",
       "      <td>USGS-01477120</td>\n",
       "      <td>Raccoon Creek near Swedesboro NJ</td>\n",
       "      <td>USGS-NJ</td>\n",
       "      <td>USGS New Jersey Water Science Center</td>\n",
       "      <td>-75.259167</td>\n",
       "      <td>39.740556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19526</th>\n",
       "      <td>USGS-01464907</td>\n",
       "      <td>Little Neshaminy C at Valley Road nr Neshaminy PA</td>\n",
       "      <td>USGS-PA</td>\n",
       "      <td>USGS Pennsylvania Water Science Center</td>\n",
       "      <td>-75.119616</td>\n",
       "      <td>40.229275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19527</th>\n",
       "      <td>USGS-01472157</td>\n",
       "      <td>French Creek near Phoenixville, PA</td>\n",
       "      <td>USGS-PA</td>\n",
       "      <td>USGS Pennsylvania Water Science Center</td>\n",
       "      <td>-75.601305</td>\n",
       "      <td>40.151491</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      MonitoringLocationIdentifier  \\\n",
       "19518                 STROUD-WELL7   \n",
       "19519                 STROUD-WELL8   \n",
       "19520                 STROUD-WELL9   \n",
       "19521         WWMD_VA-USGS01458500   \n",
       "19522         WWMD_VA-USGS01467200   \n",
       "19523         WWMD_VA-USGS01477050   \n",
       "19524                USGS-01467150   \n",
       "19525                USGS-01477120   \n",
       "19526                USGS-01464907   \n",
       "19527                USGS-01472157   \n",
       "\n",
       "                                  MonitoringLocationName  \\\n",
       "19518                             Stroud Preserve: Well7   \n",
       "19519                             Stroud Preserve: Well8   \n",
       "19520                             Stroud Preserve: Well9   \n",
       "19521      USGS 1458500  Delaware River at Frenchtown NJ   \n",
       "19522  USGS 1467200  Delaware R at Ben Franklin Bridg...   \n",
       "19523        USGS 1477050  Delaware River at Chester, PA   \n",
       "19524                     Cooper River at Haddonfield NJ   \n",
       "19525                   Raccoon Creek near Swedesboro NJ   \n",
       "19526  Little Neshaminy C at Valley Road nr Neshaminy PA   \n",
       "19527                 French Creek near Phoenixville, PA   \n",
       "\n",
       "      OrganizationIdentifier                       OrganizationFormalName  \\\n",
       "19518                 STROUD  Stroud Water Research Center (Pennsylvania)   \n",
       "19519                 STROUD  Stroud Water Research Center (Pennsylvania)   \n",
       "19520                 STROUD  Stroud Water Research Center (Pennsylvania)   \n",
       "19521                WWMD_VA        Virginia - World Water Monitoring Day   \n",
       "19522                WWMD_VA        Virginia - World Water Monitoring Day   \n",
       "19523                WWMD_VA        Virginia - World Water Monitoring Day   \n",
       "19524                USGS-NJ         USGS New Jersey Water Science Center   \n",
       "19525                USGS-NJ         USGS New Jersey Water Science Center   \n",
       "19526                USGS-PA       USGS Pennsylvania Water Science Center   \n",
       "19527                USGS-PA       USGS Pennsylvania Water Science Center   \n",
       "\n",
       "       LongitudeMeasure  LatitudeMeasure  \n",
       "19518        -75.653158        39.944822  \n",
       "19519        -75.653400        39.945586  \n",
       "19520        -75.653531        39.945444  \n",
       "19521        -75.065000        40.526111  \n",
       "19522        -75.137399        39.954002  \n",
       "19523        -75.366302        39.836779  \n",
       "19524        -75.021389        39.903056  \n",
       "19525        -75.259167        39.740556  \n",
       "19526        -75.119616        40.229275  \n",
       "19527        -75.601305        40.151491  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['MonitoringLocationIdentifier', 'MonitoringLocationName', 'OrganizationIdentifier', 'OrganizationFormalName', \n",
    "    'LongitudeMeasure', 'LatitudeMeasure']].tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Well                                               16023\n",
       "River/Stream                                        1509\n",
       "Stream                                              1203\n",
       "Estuary                                              207\n",
       "Spring                                               177\n",
       "Lake                                                 105\n",
       "Atmosphere                                            67\n",
       "Facility: Outfall                                     44\n",
       "Lake, Reservoir, Impoundment                          32\n",
       "River/Stream Perennial                                30\n",
       "Well: Collector or Ranney type well                   18\n",
       "Facility Other                                        14\n",
       "Stream: Tidal stream                                  14\n",
       "Well: Test hole not completed as a well               12\n",
       "Facility Municipal Sewage (POTW)                      10\n",
       "Subsurface: Unsaturated zone                           9\n",
       "Facility: Water-distribution system                    7\n",
       "Reservoir                                              7\n",
       "Facility: Diversion                                    6\n",
       "Land Runoff                                            5\n",
       "Land: Outcrop                                          4\n",
       "Facility Privately Owned Non-industrial                4\n",
       "Land: Excavation                                       2\n",
       "Channelized Stream                                     2\n",
       "Facility Industrial                                    2\n",
       "Stream: Canal                                          2\n",
       "Stream: Ditch                                          2\n",
       "Land                                                   2\n",
       "Land: Sinkhole                                         2\n",
       "Other-Surface Water                                    1\n",
       "Land: Soil hole                                        1\n",
       "Storm Sewer                                            1\n",
       "Well: Multiple wells                                   1\n",
       "Riverine Impoundment                                   1\n",
       "Facility: Wastewater land application                  1\n",
       "Facility: Laboratory or sample-preparation area        1\n",
       "Name: MonitoringLocationTypeName, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['MonitoringLocationTypeName'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NWIS       17542\n",
       "STORET      1982\n",
       "BIODATA        4\n",
       "Name: ProviderName, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ProviderName'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "USGS-PA                  15162\n",
       "USGS-NJ                   1547\n",
       "USGS-MD                    837\n",
       "11NPSWRD                   419\n",
       "NJDEP_BFBM                 253\n",
       "21NJDEP1                   199\n",
       "DRBC                       152\n",
       "NJDEP_AMERICORPS           124\n",
       "21DELAWQ_WQX               121\n",
       "NJDEPNJWAP                  89\n",
       "KWMNDATA                    79\n",
       "NJDEP_DSREH                 67\n",
       "MDE_FIELDSERVICES_WQX       60\n",
       "SAN                         43\n",
       "21PA_WQX                    39\n",
       "NJDEP_BEARS                 32\n",
       "STROUD                      31\n",
       "31DRBCSP                    29\n",
       "NARS_WQX                    27\n",
       "EMAP_CS                     25\n",
       "EMAP_CS_WQX                 23\n",
       "31DELRBC_WQX                20\n",
       "31DELRBC                    17\n",
       "MDEDAT05_WQX                15\n",
       "42SRBCWQ_WQX                12\n",
       "NALMS                       11\n",
       "OST_SHPD_TEST               10\n",
       "NARSTEST                    10\n",
       "RCE WRP                     10\n",
       "NARS                         8\n",
       "MDE_TMDL                     8\n",
       "OST_SHPD                     7\n",
       "MDEDAT06_WQX                 7\n",
       "MDPOINTSOURCE                7\n",
       "DEMOTEST                     6\n",
       "MDEDAT03_WQX                 6\n",
       "EPA_REGION_2                 4\n",
       "CBP_WQX                      4\n",
       "WWMD_VA                      3\n",
       "DEMOTEST_WQX                 3\n",
       "NJWSA                        1\n",
       "MUSCONETCONGORG_WQX          1\n",
       "Name: OrganizationIdentifier, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['OrganizationIdentifier'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Initially, let's target these attributes only.** (Note: the Provider is the data system or database that the data are extracted from; the Organization is the entity that actually collected or is ultimately responsible for the data). The values shown are the ones from the example listed in cell 15, above.\n",
    "```\n",
    "ProviderName                                                     NWIS\n",
    "OrganizationIdentifier                                        USGS-MD\n",
    "OrganizationFormalName             USGS Maryland Water Science Center\n",
    "MonitoringLocationIdentifier                    MD006-394205075514101\n",
    "MonitoringLocationName                                      CE Ae  75\n",
    "MonitoringLocationTypeName                                       Well\n",
    "LatitudeMeasure                                               39.7013\n",
    "LongitudeMeasure                                             -75.8615\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:odm2client]",
   "language": "python",
   "name": "conda-env-odm2client-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}