notebooks/tutorials/00_Calibration/3_calib_arucos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sandbox Markers calibration\n",
    "In this tutorial we will learn the basic steps to set up the ArUco markers and how to use them for the _open AR Sandbox_. \n",
    "This not only involves tweaking of calibration parameters, but also how to use this information.\n",
    "\n",
    "Let's start with importing the main module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Only use if sandbox is not installed\n",
    "import os,sys\n",
    "sys.path.append('../../')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Before starting:\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b> Projector calibration file: </b> We will use the calibration file generated from the previous tutorial (Sandbox Projector calibration). If you have not calibrated yet the projector, please stop here and go back to the 1_calib_projector.ipynb jupyter notebook to calibrate the projector. If this is already done you can continue.\n",
    "    \n",
    "    \n",
    "<b> Sensor calibration file: </b> The same applies for the sensor calibration file from the previous tutorial (Sanbox Sensor calibration). If you have not calibrated yet the sensor, please stop here and go back to the 2_calib_sensor.ipynb jupyter notebook to calibrate the Sensor. If this is already done you can continue. \n",
    "</div>\n",
    "\n",
    "* **Marker**: (This module is optional) This will allow to use the the ArUco markers in the AR-Sandbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Marker Detection **(optional)** \n",
    "\n",
    "The aruco markers is a class from the OpenCv package, for pose estimation with many computer vision applications. (Follow the link to read more about Aruco marker detection https://docs.opencv.org/trunk/d5/dae/tutorial_aruco_detection.html)\n",
    "\n",
    "The basics of the Aruco marker class for sandbox purposes are described below:\n",
    "* Each marker represents a point in space captured by the camera of the kinect and projected in real time.\n",
    "* This position can be used to introduce point information to the model.\n",
    "* This position may represent, e.g. wellbore location in the model, the extremes to create a 2d cross section of the 3d model, seed point for simulations etc...  \n",
    "\n",
    "_Moreover, the aruco markers are used to perform the automatic calibration of the sandbox by fitting the projected image inside the sandbox and by croping the depth frame to the information inside the sandbox._ (Feature disabled for the moment) \n",
    "\n",
    "So first, we start the class that will receive the information of the sensor kinect. For this, start the Sensor class. \n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b> Only suport KinectV2 or LiDAR L515</b> If you are using KinectV1 or DummySensor, you can use the Dummy aruco marker class. Go to the bottom of this notebook \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save the paths to the calibration files \n",
    "from sandbox import _calibration_dir\n",
    "_calibprojector = _calibration_dir + \"my_projector_calibration.json\"\n",
    "_calibsensor = _calibration_dir + \"my_sensor_calibration.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import Sensor\n",
    "from sandbox.sensor import Sensor\n",
    "sensor = Sensor(calibsensor=_calibsensor, name=\"lidar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize the aruco detection\n",
    "from sandbox.markers import ArucoMarkers\n",
    "aruco = ArucoMarkers(sensor=sensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### * Place as many Arucos you want inside the sandbox, the more the better for the calibration - Skip this step for the moment, future "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Detect arucos\n",
    "#--- Show the pandas dataframe\n",
    "# "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see the the DataFrame containing all the relevant information of the aruco. Its coordinates in the depth space (pixel precise), color space (pixel precise), and camera space (milimeter precise). If not all (or None) of the arucos are detected, consider improving the light setting so the arucos can be detected by the camera. If this does not fix the problem, then put the Sensor closer to the sandbox and run the calibration of the Sensor again.  \n",
    "\n",
    "Now we can plot the coordinates in both spaces.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show plots\n",
    "#---- both depth space and color space\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are happy with the mapping of the markers from the color space to the depth space you can skip the calibration. If you see that the markers are displaced we can change that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here put the calibration of the markers \n",
    "# is just changing the constant and then save this constant to the calibration file for future occations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the new figure. If you are happy with the result you can continue, otherwise repeat the previous until you are happy with the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Sensor and Projector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save the paths to the calibration files \n",
    "from sandbox import _calibration_dir\n",
    "_calibprojector = _calibration_dir + \"my_projector_calibration.json\"\n",
    "_calibsensor = _calibration_dir + \"my_sensor_calibration.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import Sensor\n",
    "from sandbox.sensor import Sensor\n",
    "sensor = Sensor(calibsensor=_calibsensor, name=\"lidar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import projector\n",
    "from sandbox.projector import Projector\n",
    "projector = Projector(calibprojector=_calibprojector)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize marker detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize the aruco detection\n",
    "from sandbox.markers import MarkerDetection\n",
    "aruco = MarkerDetection(sensor=sensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize main thread for live update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sandbox.main_thread import MainThread\n",
    "main = MainThread(sensor=sensor, projector=projector, aruco=aruco)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start the thread \n",
    "main.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Control the features of the main thread"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "main.widget_plot_module()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stop thread of sandbox\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "main.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use ArUco markers without ArUcos - Dummy markers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add manually arucos to your Sandbox by creating a dummy directory with id and coordinates in the coordinates of the sensor. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#{id:[coordinate in x, coordinate in y]}\n",
    "aruco_dict = {1:[50,  30 ],\n",
    "              2:[100, 150],\n",
    "             29:[40,  100],\n",
    "             13:[120, 80 ],\n",
    "             }\n",
    "# This corresponds to 4 dummy markers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aruco.set_aruco_position(dict_position=aruco_dict, frame=sensor.get_frame())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now you can check if they were succesfullly added. \n",
    "aruco.df_aruco_position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use ArUco dummy Class - For KinectV1 and DummySensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save the paths to the calibration files \n",
    "from sandbox import _calibration_dir\n",
    "_calibprojector = _calibration_dir + \"my_projector_calibration.json\"\n",
    "_calibsensor = _calibration_dir + \"my_sensor_calibration.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import Sensor - Any sensor [\"kinect_v1\",\"kinect_v2\", \"dummy\"]\n",
    "from sandbox.sensor import Sensor\n",
    "sensor = Sensor(calibsensor=_calibsensor, name=\"dummy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import projector\n",
    "from sandbox.projector import Projector\n",
    "projector = Projector(calibprojector=_calibprojector)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize the aruco detection\n",
    "from sandbox.markers import MarkerDetection # Fix here!!!!\n",
    "dummy_sensor = Sensor(calibsensor=_calibsensor, name=\"dummy\")\n",
    "aruco = MarkerDetection(sensor=dummy_sensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#{id:[coordinate in x, coordinate in y]}\n",
    "aruco_dict = {1:[50,  30 ],\n",
    "              2:[100, 150],\n",
    "             29:[40,  100],\n",
    "             13:[120, 80 ],\n",
    "             }\n",
    "# This corresponds to 4 dummy markers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aruco.set_aruco_position(dict_position=aruco_dict, frame=sensor.get_frame())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now you can check if they were succesfullly added. \n",
    "aruco.df_aruco_position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}