1_explore_CoolStore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Science\n",
    "\n",
    "\n",
    "To begin, we import a range of python packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For running inference on the TF-Hub module.\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from PIL import ImageColor\n",
    "from PIL import ImageDraw\n",
    "from PIL import ImageFont\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "print('Imported libraries')\n",
    "print(tf.version.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import our image\n",
    "\n",
    "In the next cell we import the image we want to test our model on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = 'images/RHODS_cool_store.png'\n",
    "img2 = Image.open(img)\n",
    "img2.resize((500, 300))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see some items form the CoolStuff Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_img(path):\n",
    "    img = tf.io.read_file(path)\n",
    "    img = tf.image.decode_jpeg(img, channels=3)\n",
    "    return img\n",
    "\n",
    "print('Defined load_img function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_coolstore = load_img(img)\n",
    "tf_coolstore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load in a model\n",
    "\n",
    "The model we are going to use today is the ssd Mobilenet v2/1 model, which you can download from TensorFlow Hub [here](https://tfhub.dev/google/openimages_v4/ssd/mobilenet_v2/1). The model has been trained on Google's [Open Images](https://storage.googleapis.com/openimages/web/index.html) v6 data set, and can recognise 600 types of objects. We begin by loading in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dir = 'models/openimages_v4_ssd_mobilenet_v2_1'\n",
    "saved_model = tf.saved_model.load(model_dir)\n",
    "detector = saved_model.signatures['default']\n",
    "\n",
    "print('Loaded model into the detector variable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "converted_img = tf.image.convert_image_dtype(tf_coolstore, tf.float32)[tf.newaxis, ...]\n",
    "converted_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = detector(converted_img)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has returned a dictionary, containing arrays, each of which holds information about the detected objects. The information includes identifiers for the types of objects, coordinates locating the objects within the image, and detection scores, corresponding to how certain the model is about its prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use afew functions to help us to superimpose the information in this dictionary onto the origional image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_image(image):\n",
    "    fig = plt.figure(figsize=(10, 10))\n",
    "    plt.grid(False)\n",
    "    plt.imshow(image)\n",
    "\n",
    "\n",
    "def draw_bounding_box_on_image(image,\n",
    "                               ymin,\n",
    "                               xmin,\n",
    "                               ymax,\n",
    "                               xmax,\n",
    "                               color,\n",
    "                               font,\n",
    "                               thickness=4,\n",
    "                               display_str_list=()):\n",
    "    \"\"\"Adds a bounding box to an image.\"\"\"\n",
    "    draw = ImageDraw.Draw(image)\n",
    "    im_width, im_height = image.size\n",
    "    (left, right, top, bottom) = (xmin * im_width, xmax * im_width,\n",
    "                                  ymin * im_height, ymax * im_height)\n",
    "    draw.line([(left, top), (left, bottom), (right, bottom), (right, top),\n",
    "               (left, top)],\n",
    "              width=thickness,\n",
    "              fill=color)\n",
    "\n",
    "    # If the total height of the display strings added to the top of the bounding\n",
    "    # box exceeds the top of the image, stack the strings below the bounding box\n",
    "    # instead of above.\n",
    "    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]\n",
    "    # Each display_str has a top and bottom margin of 0.05x.\n",
    "    total_display_str_height = (1 + 2 * 0.05) * sum(display_str_heights)\n",
    "\n",
    "    if top > total_display_str_height:\n",
    "        text_bottom = top\n",
    "    else:\n",
    "        text_bottom = top + total_display_str_height\n",
    "    # Reverse list and print from bottom to top.\n",
    "    for display_str in display_str_list[::-1]:\n",
    "        text_width, text_height = font.getsize(display_str)\n",
    "        margin = np.ceil(0.05 * text_height)\n",
    "        draw.rectangle([(left, text_bottom - text_height - 2 * margin),\n",
    "                        (left + text_width, text_bottom)],\n",
    "                       fill=color)\n",
    "        draw.text((left + margin, text_bottom - text_height - margin),\n",
    "                  display_str,\n",
    "                  fill=\"black\",\n",
    "                  font=font)\n",
    "        text_bottom -= text_height - 2 * margin\n",
    "\n",
    "\n",
    "def draw_boxes(image, boxes, class_names, scores, max_boxes=10, min_score=0.1):\n",
    "    \"\"\"Overlay labeled boxes on an image with formatted scores and label names.\"\"\"\n",
    "    colors = list(ImageColor.colormap.values())\n",
    "\n",
    "    font = ImageFont.load_default()\n",
    "\n",
    "    for i in range(min(boxes.shape[0], max_boxes)):\n",
    "        if scores[i] >= min_score:\n",
    "            ymin, xmin, ymax, xmax = tuple(boxes[i])\n",
    "            display_str = \"{}: {}%\".format(class_names[i].decode(\"ascii\"),\n",
    "                                           int(100 * scores[i]))\n",
    "            color = colors[hash(class_names[i]) % len(colors)]\n",
    "            image_pil = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n",
    "            draw_bounding_box_on_image(\n",
    "                image_pil,\n",
    "                ymin,\n",
    "                xmin,\n",
    "                ymax,\n",
    "                xmax,\n",
    "                color,\n",
    "                font,\n",
    "                display_str_list=[display_str])\n",
    "            np.copyto(image, np.array(image_pil))\n",
    "    return image\n",
    "\n",
    "\n",
    "print('Defined image display functions')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = {key: value.numpy() for key, value in result.items()}\n",
    "\n",
    "image_with_boxes = draw_boxes(\n",
    "    tf_coolstore.numpy(), result[\"detection_boxes\"],\n",
    "    result[\"detection_class_entities\"], result[\"detection_scores\"])\n",
    "\n",
    "display_image(image_with_boxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, you can see we've got some predictions for 'clothing', and we can now see what parts of the image have been identified as such.\n",
    "\n",
    "However, if we look at the detection scores, some of the predictions are not made with high confidence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['detection_scores']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's only draw boxes if the model was over 30% confident that the object was correctly classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "per = 0.3\n",
    "image_with_boxes_per = draw_boxes(\n",
    "    tf_coolstore.numpy(), result[\"detection_boxes\"][result[\"detection_scores\"] > per],\n",
    "    result[\"detection_class_entities\"][result[\"detection_scores\"] > per],\n",
    "    result[\"detection_scores\"][result[\"detection_scores\"] > per])\n",
    "\n",
    "display_image(image_with_boxes_per)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! So you've seen how we can use a pre-trained model to identify objects in images. In the next notebooks, we will wrap this model in a flask app, which we can use as part of a larger application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}