MicaSense Image Processing Setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MicaSense Image Processing Setup\n",
    "\n",
    "## Overview\n",
    "\n",
    "This series of tutorials will be a walk through on how to process RedEdge data from raw images through conversion to reflectance. In this first tutorial, we will cover the tools required to do this, get them installed, and verify that the installation works. \n",
    "\n",
    "## System Requirements\n",
    "\n",
    "Our tutorials are written using Python3.  Python has great library support for image processing through libraries such as OpenCV, SciKit Image, and others.  In this tutorial, we'll use python, OpenCV, numpy, and matplotlib, as well as the standalone exiftool and it's python wrapper to open and manipulate RedEdge images to transform raw digital number values into quantitative reflectance. Python 2.7 can work for this tutorial, but we only test with Python3 and later tutorials use some extra libraries that are best supported in Python3 (specifically pysolar), so we recommend that if you're starting with python from scratch to install Python3.\n",
    "\n",
    "This tutorial has been tested on Windows, MacOS, and Linux. It is likely to work on other platforms, especially unix-based platforms like OSX, but you will have to do the legwork to get the required software installed and working.\n",
    "\n",
    "### Software/Libraries Overview\n",
    "\n",
    "The following softare and libraries are required for this tutorial:\n",
    "\n",
    "* [python3](https://www.python.org/downloads/release/latest)\n",
    "* [numpy](https://www.numpy.org/)\n",
    "* [openCV](https://opencv.org/releases.html)\n",
    "* [matplotlib](https://matplotlib.org/users/installing.html)\n",
    "* [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) + [pyexiftool](https://github.com/smarnach/pyexiftool)\n",
    "* [scikit-image](https://scikit-image.org/)\n",
    "* [zbar](https://zbar.sourceforge.net/) + [pyzbar](https://github.com/NaturalHistoryMuseum/pyzbar)\n",
    "* [pysolar](http://pysolar.org/)\n",
    "* [pandas](https://pandas.pydata.org/)\n",
    "* [mapboxgl](https://github.com/mapbox/mapboxgl-jupyter)\n",
    "\n",
    "Below, we go through the options to download and install a full working python environment with these tools (and their dependencies).  We're using the [Anaconda](https://www.anaconda.com/download/) or [miniconda](https://conda.io/miniconda.html) environments where possible to ease installation, but if you're already a python package management guru, you can use `git` to checkout this code repository and look at the `micasense_conda_env.yml` file for the dependencies you'll need in your virtual environment.\n",
    "\n",
    "### Linux (Debian/Ubuntu)\n",
    "\n",
    "For linux (and Mac, to some extent) you can either install the libraries directly using `pip` or install `miniconda` or `anaconda` to create completely separate environments.  We have had success installing `miniconda` locally -- it's a smaller install than `anaconda` and can be installed without using `sudo` and doesn't impact the system-installed python or python libraries.  You will likely still need to use `sudo` to install \n",
    "\n",
    "The following is what we had to do on a fresh Ubuntu 18.04 image to install the library. First we installed some system tools and libraries:\n",
    "\n",
    "    sudo apt install git\n",
    "    sudo apt install libzbar0\n",
    "    sudo apt install make\n",
    "    \n",
    "Next we installed [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/):\n",
    "\n",
    "    wget https://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-10.98.tar.gz\n",
    "    tar -xvzf Image-ExifTool-10.98.tar.gz \n",
    "    cd Image-ExifTool-10.98/\n",
    "    perl Makefile.PL \n",
    "    make test\n",
    "    sudo make install\n",
    "\n",
    "Then we installed miniconda.  Navigate to the [miniconda download page](https://conda.io/miniconda.html) and download the installer for your system and follow the [installation instructions](https://conda.io/docs/user-guide/install/index.html)\n",
    "\n",
    "Once these tools are installed, you can check out this repository and create the `micasense conda` environment:\n",
    "\n",
    "    git clone https://github.com/micasense/imageprocessing.git\n",
    "    cd imageprocessing\n",
    "    conda env create -f micasense_conda_env.yml\n",
    "\n",
    "Finally, one way to verify our install by running the built in tests:\n",
    "\n",
    "    cd imageprocessing\n",
    "    conda activate micasense\n",
    "    pytest .\n",
    "\n",
    "Or, to start working with the notebooks (including running the test code below):\n",
    "\n",
    "    cd imageprocessing\n",
    "    conda activate micasense\n",
    "    jupyter notebook .\n",
    "\n",
    "\n",
    "### Windows setup\n",
    "\n",
    "When installing on Windows we rely on the [Anaconda](https://www.anaconda.com/download/) python environment to do most of the heavy lifting for us.\n",
    "\n",
    "* Install [Anaconda](https://www.anaconda.com/download/) for your system by downloading the **Python 3.6** version\n",
    "\n",
    " * When installing Anaconda, choose **\"install for only me\"** instead of \"install for all users,\" as this simplifies installation of other packages\n",
    "\n",
    "* Download the [exiftool windows package](https://www.sno.phy.queensu.ca/~phil/exiftool/) and unzip it to a permanent location such as `c:\\exiftool\\`.  Now we need to tell the python code where to find exiftool (so we don't have to set it up in every script we write), and we do that by adding the path to exiftool as an environment variable.\n",
    " * Create an environment variable called `exiftoolpath` with a value of the full path to exiftool.  For example, `c:\\exiftool\\exiftool.exe`\n",
    " * To do this on Windows 10, press Start button or the Windows key, then type `Path` and click `Edit Environment Variables for Your Account`\n",
    " * Click `New`\n",
    " * In Variable Name type `exiftoolpath`\n",
    " * In Variable Value type `c:\\exiftool\\exiftool.exe`\n",
    "\n",
    "* Open an Anaconda console from the start menu as an administrator by clicking `Start->Anaconda`, right-click `Anaconda Console`, choose `Run as Administrator`. Execute the following commands in the anaconda console:\n",
    "\n",
    " * `cd` to the directory you git cloned this repository to\n",
    " * `conda env create -f micasense_conda_env.yml`\n",
    " * This will configure an anaconda environment with all of the required tools and libraries This will take a while depending on your computer and internet speeds (5-10 minutes is not uncommon)\n",
    " * When it's done, run `activate micasense` to activate the environment configured\n",
    " * Each time you run start a new anaconda prompt, you'll need to run `activate micasense`\n",
    " \n",
    "### MacOS setup\n",
    "\n",
    "The following steps to get going on MacOS worked for us. \n",
    "\n",
    "First we installed `git` by installing the `xcode` developer tools, or you can follow the instructions at the [git site](https://git-scm.com/downloads). \n",
    "\n",
    "Next, we [downloaded and installed exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) using the MacOS installer.\n",
    "\n",
    "Third we installed [Homebrew](https://brew.sh/) and used it to install `zbar`:\n",
    "\n",
    "    /usr/bin/ruby -e \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\"\n",
    "    brew install zbar\n",
    "    \n",
    "Then we installed miniconda.  If you're comfortable on the command line, navigate to the [miniconda download page](https://conda.io/miniconda.html) and download the installer for your system. Open an iTerm and follow the [installation instructions](https://conda.io/docs/user-guide/install/index.html).\n",
    "\n",
    "If instead you're more comfortable with graphical installers, the [Anaconda](https://www.anaconda.com/download/) version for **Python 3.6** may be right for you.\n",
    "\n",
    "Once these tools are installed, you can check out this repository and create the `micasense conda` environment by opening an iTerm and running the following commands:\n",
    "    \n",
    "    git clone https://github.com/micasense/imageprocessing.git\n",
    "    cd imageprocessing\n",
    "    conda env create -f micasense_conda_env.yml\n",
    "    \n",
    "This will take a while (5-10 minutes isn't uncommon).  Once it's done, one way to verify our install by running the built-in tests:\n",
    "\n",
    "    conda activate micasense\n",
    "    pytest .\n",
    "\n",
    "Or, to start working with the notebooks (including running the test code below):\n",
    "\n",
    "    cd imageprocessing\n",
    "    conda activate micasense\n",
    "    jupyter notebook .\n",
    "\n",
    "\n",
    "## Running the notebooks\n",
    "\n",
    "* If running on Windows, run the `Anaconda Prompt` from the Start menu and type `activate micasense`\n",
    "* `cd` to the imageprocessing checkout directory\n",
    "* Run `jupyter notebook .`\n",
    "\n",
    "## Testing Installation\n",
    "\n",
    "The following python snippet can be run from a jupyter notebook, inside iPython, or by saving to a script and running from the command line.  If you're on windows, make sure you have set the location of exiftool in the `exiftoolpath` environment variable.  If this script succeeds, your system is ready to go! If not, check the installation documentation for the module import that is having issues.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import cv2 #openCV\n",
    "import exiftool\n",
    "import os, glob\n",
    "import numpy as np\n",
    "import pyzbar.pyzbar as pyzbar\n",
    "import matplotlib.pyplot as plt\n",
    "import mapboxgl\n",
    "\n",
    "print()\n",
    "print(\"Successfully imported all required libraries.\")\n",
    "print()\n",
    "\n",
    "if os.name == 'nt':\n",
    "    if os.environ.get('exiftoolpath') is None:\n",
    "        print(\"Set the `exiftoolpath` environment variable as described above\")\n",
    "    else:\n",
    "        if not os.path.isfile(os.environ.get('exiftoolpath')):\n",
    "            print(\"The provided exiftoolpath isn't a file, check the settings\")\n",
    "\n",
    "try:\n",
    "    with exiftool.ExifTool(os.environ.get('exiftoolpath')) as exift:\n",
    "        print('Successfully executed exiftool.')\n",
    "except Exception as e:\n",
    "    print(\"Exiftool isn't working. Double check that you've followed the instructions above.\")\n",
    "    print(\"The execption text below may help to find the source of the problem:\")\n",
    "    print()\n",
    "    print(e)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing image reading and panel detection\n",
    "\n",
    "The above code checks for the proper libraries to be installed and verifies it can execute `exiftool`. This code opens an example image, reads the metadata, and then uses the `pyzbar` library to find a MicaSense panel in the image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from micasense.image import Image\n",
    "imagePath = os.path.join('.','data','0000SET','000')\n",
    "imageName = glob.glob(os.path.join(imagePath,'IMG_0000_1.tif'))[0]\n",
    "\n",
    "img = Image(imageName)\n",
    "img.plot_raw(figsize=(8.73,8.73));\n",
    "\n",
    "from micasense.panel import Panel\n",
    "panel = Panel(img)\n",
    "if not panel.panel_detected():\n",
    "    raise IOError(\"Panel Not Detected! Check your installation of pyzbar\")\n",
    "else:\n",
    "    panel.plot(figsize=(8,8));\n",
    "\n",
    "print('Success! Now you are ready for Part 1 of the tutorial.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Copyright (c) 2017-2019 MicaSense, Inc.  For licensing information see the [project git repository](https://github.com/micasense/imageprocessing)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}