notebook_examples/audi-autonomous_driving-oracle_open_data.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{audi-autonomous_driving-oracle_open_data.ipynb,\n",
    "    title: Audi Autonomous Driving Dataset Repository,\n",
    "    summary: Download, process and display autonomous driving data, and map LiDAR data onto images.,\n",
    "    developed_on: computervision_p37_cpu_v1,\n",
    "    keywords: autonomous driving, oracle open data,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc. All rights reserved. Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
    "\n",
    "***\n",
    "\n",
    "# <font color=red>Audi Autonomous Driving Dataset Repository</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by <font color=\"teal\">Oracle for Research</font></p>\n",
    "\n",
    "***\n",
    "\n",
    "## Overview:\n",
    "\n",
    "Audi Autonomous Driving Dataset (A2D2) is an open multi-sensor dataset for autonomous driving research. This dataset consists of simultaneously recorded images and 3D point clouds, together with 3D bounding boxes, semantic segmentation, instance segmentation, and data extracted from the automotive bus. The sensor suite consists of six cameras and five LiDAR units, providing 360-degree coverage.\n",
    "\n",
    "This notebook demonstrates how to download the dataset from Oracle Cloud Infrastructure (OCI) Object Storage and work with the JSON configuration file. It also demonstrates how to process image and LiDAR data, and display them.\n",
    "\n",
    "Compatible conda pack: [Computer Vision](https://docs.oracle.com/en-us/iaas/data-science/using/conda-com-vision-fam.htm) for CPU on Python 3.7 (version 1.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href=\"#intro\">Introduction</a>\n",
    "    - <a href='#data'>Dataset</a>\n",
    "    - <a href=\"#open_data\">What is Oracle Open Data?</a>\n",
    "- <a href=\"#understand\">Download and Understand the Dataset</a>\n",
    "- <a href='#download'>Download</a>\n",
    "    - <a href='#config'>Understanding the Configuration File\n",
    "    - <a href='#lidar'> Understanding the LiDAR Data\n",
    "- <a href=\"#images\"> Processing Images\n",
    "    - <a href='#load'> Loading Images\n",
    "    - <a href='#map'> Mapping LiDAR Data onto Images\n",
    "- <a href=\"#cleanup\">Clean Up</a>\n",
    "- <a href='#ref'>References</a>\n",
    "    \n",
    "---\n",
    "\n",
    "\n",
    "Datasets are provided as a convenience. Datasets are considered third-party content and are not considered materials under your agreement with Oracle.\n",
    "\n",
    "You can access the `Audi Autonomous Driving Dataset (A2D2)` dataset license [here](https://aev-autonomous-driving-dataset.s3.eu-central-1.amazonaws.com/LICENSE.txt).\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import glob\n",
    "import json\n",
    "import matplotlib.pylab as pt\n",
    "import numpy as np\n",
    "import os\n",
    "import pprint\n",
    "import requests\n",
    "import tarfile\n",
    "\n",
    "from os.path import join\n",
    "from shutil import rmtree\n",
    "from tempfile import mkdtemp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"intro\"></a>\n",
    "# Introduction\n",
    "\n",
    "<a id='data'></a>\n",
    "## Dataset\n",
    "\n",
    "**Data**: A2D2 includes data recorded on highways, country roads, and cities in the south of Germany. The data are recorded under cloudy, rainy, and sunny weather conditions. The dataset provides semantic segmentation labels, instance segmentation labels, and 3D bounding boxes for non-sequential frames. 41,277 images have semantic and instance segmentation labels for 38 categories. All images have corresponding LiDAR point clouds, of which 12,497 are annotated with 3D bounding boxes within the field of view of the front-center camera. The [A2D2: Audi Autonomous Driving Dataset](https://arxiv.org/pdf/2004.06320.pdf) paper provides more details about how the data are collected and structured.\n",
    "\n",
    "**Directory Structure**: There are multiple tar files for the sections, such as _camera_ and _lidar_ with the released timestamp in 'YYYYMMDDHHMMSS' format. Each tar file contains images, annotated labels, and point clouds.\n",
    "\n",
    "**Template**: `https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/a2d2/o/<tar file name>`. For example, \n",
    "* The data for camera 'frontcenter' with timestamp '20180810150607': https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/a2d2/o/camera_lidar-20180810150607_camera_frontcenter.tar\n",
    "\n",
    "**Data Availability**: All data is available from the [Audi Autonomous Driving Dataset (A2D2) repository](https://opendata.oraclecloud.com/ords/r/opendata/opendata/details?data_set_id=6), which is part of [Oracle Open Data](https://opendata.oraclecloud.com/ords/r/opendata/opendata/home).\n",
    "\n",
    "<a id=\"open_data\"></a>\n",
    "## What is Oracle Open Data?\n",
    "\n",
    "Oracle Open Data is a free service that curates spatial images, protein sequences, and annotated text files from the world's leading scientific databases. The repository connects researchers, developers, students, and educators with petabytes of open data from trusted resources. You can use Oracle Open Data to view important metadata and sample code for each data set, which simplifies technical complexities and makes it easy for researchers to use.\n",
    "\n",
    "<a id=\"understand\"></a>\n",
    "# Download and Understand the Dataset\n",
    "\n",
    "<a id=\"download\"></a>\n",
    "## Download\n",
    "OCI Object Storage enables you to securely store any type of data in its native format. With built-in redundancy, Object Storage is ideal for building modern applications that require scale and flexibility because you can use it to consolidate multiple data sources for analytics, backup, or archive purposes.\n",
    "\n",
    "The Caltech pedestrian detection benchmark for video data are stored in Object Storage in the SEQ format. You can run the next cell to download the `a2d2-preview.tar` and extract all files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2d2_preview_url = \"https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/a2d2/o/a2d2-preview.tar\"\n",
    "print(\"Downloading A2D2 Preview Dataset...\")\n",
    "response = requests.get(a2d2_preview_url, stream=True)\n",
    "file = tarfile.open(fileobj=response.raw, mode=\"r|tar\")\n",
    "data_path = mkdtemp()\n",
    "print(f\"Extracting data to {data_path}\")\n",
    "file.extractall(path=data_path)\n",
    "print(\"Download completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"config\"></a>\n",
    "## Understanding the Configuration File\n",
    "\n",
    "Each `.tar` file includes a `cams_lidars.json` file that plays an important role as the configuration file in processing the A2D2 dataset. This JSON file contains three main items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(join(data_path, \"cams_lidars.json\"), \"r\") as config:\n",
    "    config = json.load(config)\n",
    "config.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sensor in the keys `lidars`, `cameras`, and `vehicle` have an associated `view`. A view is a sensor coordinate system, defined by an origin, and an x and y-axis. These are specified in cartesian coordinates (in m) relative to an external coordinate system. Unless otherwise stated the external coordinate system is the car's frame of reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config[\"vehicle\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `vehicle` key contains a `view` object specifying the frame of reference of the car. It also contains an `ego-dimensions` object, which specifies the extension of the vehicle in the frame of reference of the car."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config[\"lidars\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `lidars` key contains objects specifying the extrinsic calibration parameters for each LiDAR sensor. The car has five LiDAR sensors `front_left`, `front_center`, `front_right`, `rear_right`, and `rear_left`'. Each LiDAR has a `view` in the frame of reference of the car."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config[\"cameras\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `cameras` key contains camera objects which specify their calibration parameters. The car has six cameras `front_left`, `front_center`, `front_right`, `side_right`, `rear_center`, and `side_left`.\n",
    "\n",
    "<a id=\"lidar\"></a>\n",
    "## Understanding the LiDAR Data\n",
    "\n",
    "This section demonstrates how to read point clouds corresponding to the camera. The `front_center` camera is used as an example in the next cell. The LiDAR data are saved in compressed numpy format and can be read using `np.load`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_path = join(data_path, \"camera_lidar_semantic_bboxes\")\n",
    "file_names = sorted(glob.glob(join(root_path, \"*/lidar/cam_front_center/*.npz\")))\n",
    "\n",
    "file_name_lidar = file_names[0]  # select the lidar point cloud. You are able to\n",
    "# change this to view different images.\n",
    "\n",
    "lidar_front_center = np.load(file_name_lidar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore the LiDAR data using the LiDAR points within the field of view of the front center camera."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(lidar_front_center.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, you can access the coordinates of LiDAR points `row` and `col` in image space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "row = lidar_front_center[\"row\"]\n",
    "col = lidar_front_center[\"col\"]\n",
    "(row, col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"images\"></a>\n",
    "# Processing Images\n",
    "<a id=\"load\"></a>\n",
    "## Loading Images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell parses a LiDAR filename, `file_name_lidar`. This information is used to create an image filename for the front, center camera image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_parse = file_name_lidar.split(\"/\")[-1].split(\".\")[0].split(\"_\")\n",
    "file_name_image = join(\n",
    "    root_path,\n",
    "    f\"{file_parse[0][0:8]}_{file_parse[0][8:]}\",\n",
    "    \"camera\",\n",
    "    \"cam_front_center\",\n",
    "    f\"{file_parse[0]}_camera_frontcenter_{file_parse[3]}.png\",\n",
    ")\n",
    "file_name_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OpenCV-Python is a library of Python bindings designed to solve computer vision problems. In the next cell, `cv2.imread()` loads an image with the color channel order of the BGR (blue, green, red). The `cv2.cvtColor()` method is used to convert the image to the RGB (red, green, blue) color space and the image is saved.\n",
    "\n",
    "Then you can visualize the selected image captured by the front center camera."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_front_center = cv2.imread(file_name_image)\n",
    "image_front_center = cv2.cvtColor(image_front_center, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "# display image from front center camera\n",
    "pt.fig = pt.figure(figsize=(15, 15))\n",
    "pt.imshow(image_front_center)\n",
    "pt.axis(\"off\")\n",
    "pt.title(\"front center\")\n",
    "pt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"map\"></a>\n",
    "## Mapping LiDAR Data onto Images\n",
    "\n",
    "This section demonstrates how to map LiDAR data onto images. The next cell defines the function `hsv_to_rgb()` to convert a hue, saturation, value (HSV) image into RGB format. The function accepts three parameters, `h`, `s`, `v`, which represent hue, saturation, value respectively. The [Wikipedia page](https://en.wikipedia.org/wiki/HSL_and_HSV) details how this function is defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hsv_to_rgb(h, s, v):\n",
    "    if s == 0.0:\n",
    "        return v, v, v\n",
    "\n",
    "    i = int(h * 6.0)\n",
    "    f = (h * 6.0) - i\n",
    "    p = v * (1.0 - s)\n",
    "    q = v * (1.0 - s * f)\n",
    "    t = v * (1.0 - s * (1.0 - f))\n",
    "    i = i % 6\n",
    "\n",
    "    if i == 0:\n",
    "        return v, t, p\n",
    "    if i == 1:\n",
    "        return q, v, p\n",
    "    if i == 2:\n",
    "        return p, v, t\n",
    "    if i == 3:\n",
    "        return p, q, v\n",
    "    if i == 4:\n",
    "        return t, p, v\n",
    "    if i == 5:\n",
    "        return v, p, q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell defines the function `map_lidar_points_onto_image`. It maps LiDAR points onto the image. This function accepts four parameters, where `image_orig` is, for example, `image_front_center`, `lidar` represents the data you loaded in the section of <a href='#lidar'>Understanding the LiDAR Data<a>, `pixel_size` and `pixel_opacity`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def map_lidar_points_onto_image(image_orig, lidar, pixel_size=3, pixel_opacity=1):\n",
    "    image = np.copy(image_orig)\n",
    "\n",
    "    # get rows and cols\n",
    "    rows = (lidar[\"row\"] + 0.5).astype(np.int32)\n",
    "    cols = (lidar[\"col\"] + 0.5).astype(np.int32)\n",
    "\n",
    "    # lowest distance values to be accounted for in colour code\n",
    "    MIN_DISTANCE = np.min(lidar[\"distance\"])\n",
    "    # largest distance values to be accounted for in colour code\n",
    "    MAX_DISTANCE = np.max(lidar[\"distance\"])\n",
    "\n",
    "    # get distances\n",
    "    distances = lidar[\"distance\"]\n",
    "    # determine point colours from distance\n",
    "    colours = (distances - MIN_DISTANCE) / (MAX_DISTANCE - MIN_DISTANCE)\n",
    "    colours = np.asarray(\n",
    "        [np.asarray(hsv_to_rgb(0.75 * c, np.sqrt(pixel_opacity), 1.0)) for c in colours]\n",
    "    )\n",
    "    pixel_rowoffs = np.indices([pixel_size, pixel_size])[0] - pixel_size // 2\n",
    "    pixel_coloffs = np.indices([pixel_size, pixel_size])[1] - pixel_size // 2\n",
    "    canvas_rows = image.shape[0]\n",
    "    canvas_cols = image.shape[1]\n",
    "    for i in range(len(rows)):\n",
    "        pixel_rows = np.clip(rows[i] + pixel_rowoffs, 0, canvas_rows - 1)\n",
    "        pixel_cols = np.clip(cols[i] + pixel_coloffs, 0, canvas_cols - 1)\n",
    "        image[pixel_rows, pixel_cols, :] = (1.0 - pixel_opacity) * np.multiply(\n",
    "            image[pixel_rows, pixel_cols, :], colours[i]\n",
    "        ) + pixel_opacity * 255 * colours[i]\n",
    "    return image.astype(np.uint8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can visualize the mapping of the LiDAR point clouds onto the front center image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = map_lidar_points_onto_image(image_front_center, lidar_front_center)\n",
    "pt.fig = pt.figure(figsize=(15, 15))\n",
    "pt.imshow(image)\n",
    "pt.axis(\"off\")\n",
    "pt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "# Clean Up\n",
    "\n",
    "This notebook downloaded several gigabytes of data. This section will delete those files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmtree(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# References\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- [Understanding Conda Environments](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/use-notebook-sessions.htm#conda_understand_environments)\n",
    "- [Use Resource Manager to Configure Your Tenancy for Data Science](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/orm-configure-tenancy.htm)\n",
    "- [A2D2 at Oracle Open Data](https://opendata.oraclecloud.com/ords/r/opendata/opendata/details?data_set_id=6)\n",
    "- [A2D2: Audi Autonomous Driving Dataset](https://arxiv.org/pdf/2004.06320.pdf)\n",
    "- [A2D2 Tutorial](https://www.a2d2.audi/a2d2/en/tutorial.html)\n",
    "- [Convert BGR and RGB with Python, OpenCV](https://note.nkmk.me/en/python-opencv-bgr-rgb-cvtcolor/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}