This repo contains tools for synchronized RGB+D data recording, streaming and extraction from multiple Azure Kinect DKs.
The project supports distributed setup: cameras can be connected to separate computers during recording.
Complete example of installation and recording is prepared for Jetson Nano platform that is listed in docs readme.
To start usage of the code, clone the repo with already modified Azure SDK:
git clone --recurse-submodules https://github.com/MobileRoboticsSkoltech/multi-azure-recorder
And follow the building process described in the official Azure Kinect DK API building instructions (the root path there is Azure-Kinect-Sensor-SDK path): link. In order to use the Azure Kinect SDK with the device and without being 'root', this also must be done. No additional actions are needed since our modifications are already implemented to the SDK.
The following source paths and files are created in addition to original Azure code:
Azure-Kinect-Sensor-SDK/tools:
mrob_recorder # modified k4arecorder as a core executable for recorder.py
mrob_images_extractor # images extractor + undistorter + depth-to-color projector from MKV files as a backbone for extractor.sh
mrob_imu_data_extractor # IMU data extractor from MKV files as a backbone for extractor.sh
mrob_timestamps_extractor # timestamps data extractor from MKV files as a backbone for extractor.sh
mrob_calibration_params_extractor # camera color and depth camera intrinsic and color-to-depth
# camera extrinsic calib params extractor as a backbone for extractor.sh
recorder.py # multi- mrob_recorder launcher for multiple cam recording
streamer.py # online multi- RGB+D data stream visualizer
extractor.sh # MKV data extractor based on ffmpeg, mrob_imu_data_extractor, mrob_timestamps_extractor
server.py # NEW! server for distributed recording from independent computers
We use RAM folder for temporary data sharing instead of hard drive frequent read-write stress. For that, three steps must be performed (according to askubuntu).
- Make the folder you want to mount the ram disk to (must be
/mnt/mrob_tmpfseverywhere):
sudo mkdir /mnt/mrob_tmpfs
- Create a ram disk. The size chosen to be 512MB, however can be decreased just in case.
sudo mount -t tmpfs -o size=512m tmpfs /mnt/mrob_tmpfs
- Make mount permanent.
- open
/etc/fstabin nano:
sudo nano /etc/fstab
- add the following line to that file and save the file:
tmpfs /mnt/mrob_tmpfs tmpfs nodev,nosuid,noexec,nodiratime,size=512M 0 0
The ffmpeg is also required for running extractor if mrob_images_extractor is not used as a backbone:
sudo apt-get install ffmpegFor streamer.py numpy, python3-tk, python3-pil, python3-pil.imagetk packages are needed
sudo apt-get install python3-tk python3-pil python3-pil.imagetk python3-pip
pip3 install numpy
OpenCV is utilized for mrob_images_extractor backbine of extractor.sh.
Testing needed while building on a machine. Update this README too.
It is also can be needed to increase USB memory buffer. For that, use this instruction.
We also found an issue with old OpenGL version when using Azure SDK (to be more precise, this problem comes from depth engine) on Ubuntu 18.04 with Intel integrated graphics. One of the solutions is installing open source Mesa drivers. But still, we are not sure if it is the best solution. With up-to-date Nvidia GPU drivers there should not be problems.
Recording process include synchronized data gathering from multiple Azure cameras. To start recording from locally attached cameras, launch
./recoder.py
with no arguments. In this case, camera parameters are predifined by dict in params.py.
This is an example of the dict with camera params:
cams = {#keys '1', '2', etc. correspond to the written numbers sticked to camera bodies
'1' : {'ser_num' : '000583592412', 'master' : True , 'index' : None, 'sync_delay' : None, 'depth_mode' : 'NFOV_UNBINNED', 'color_mode' : '720p', 'frame_rate' : '5', 'exposure' : '8000', 'output_name' : None, 'timestamps_table_filename' : None},
'2' : {'ser_num' : '000905794612', 'master' : False, 'index' : None, 'sync_delay' : 360 , 'depth_mode' : 'NFOV_UNBINNED', 'color_mode' : '720p', 'frame_rate' : '5', 'exposure' : '8000', 'output_name' : None, 'timestamps_table_filename' : None}
}
In case of any specific parameters, params.py can be modified or command-line arguments can be put.
Files belonging to a single recording launch are stored in records/ path. Their names contain date and time of the recrording start:
records/
├── 2022-02-10-08-36-51/
│ ├── 1m.mkv
│ ├── 2s.mkv
│ ├── 3s.mkv
│ ├── ...
│ ├── 1m.csv
│ ├── 2s.csv
│ ├── 3s.csv
│ ├── ...
│ └── recording_params.json
├── 2022-02-10-08-53-13/
│ ├── 1m.mkv
│ ├── 2s.mkv
│ ├── 3s.mkv
│ ├── ...
│ ├── 1m.csv
│ ├── 2s.csv
│ ├── 3s.csv
│ ├── ...
│ └── recording_params.json
...
Every path contains:
- multiple MKV files (every file correspond to a single cam,
1min a file name means "1st camera, Master",2smeans "2nd camera, Subordinate"), - multiple CSV files that stores matching of global and local image timestamps, and
- json dictionary with parameters of cameras (the majority of params are equal to python dict; however, has some updates for some values).
Every recorded MKV file contains (if turned on in params)
- RGB video stream,
- D video stream,
- IMU data stream.
streamer.py is an additional tool for online image streams visualization. During recording, it can pool temporaly image files (jpg for RGB and binary for D image) created by mrob_recorder executable instance. For now, doesn't support streaming in distributed setup.
Extraction is aimed to
- extract RGB+D images from video streams,
- extract IMU data to a CSV file from IMU data stream,
- extract timestamps and name extracted images by timestamps,
- extract camera color and depth camera intrinsic and color-to-depth camera extrinsic calib params
- undistort + and project depth to color when using
mrob_images_extractorbackbone inextractor.shfrom every MKV file.
To extract the data, launch the following script with the <input path> argument:
extractor.sh <input path> # For instance, 'extractor.sh records/2022-02-10-08-36-51'
To use mrob_images_extractor backbone, change use_cpp_extractor=false to use_cpp_extractor=true in extractor.sh file. For choosing extraction+undistortion+matching option, launch mrob_images_extractor with no arguments to get info.
If recording was performed in distributed setup, all data should be moved to a single path for correct extraction.
extracted-data/2022-03-03-17-14-36/
├── 1m
│ ├── color
│ │ ├── 000000391122.png
│ │ ├── 000000591066.png
│ │ ...
│ ├── depth
│ │ ├── 000000391066.png
│ │ ├── 000000591066.png
│ │ ...
│ ├── calib_params.json
│ ├── color_timestamps.csv
│ ├── depth_timestamps.csv
│ ├── ir_timestamps.csv
│ └── imu.csv
├── 2s
│ ├── color
│ │ ├── 000000189833.png
│ │ ├── 000000389800.png
│ │ ...
│ ├── depth
│ │ ├── 000000389755.png
│ │ ├── 000000589755.png
│ │ ...
│ ├── calib_params.json
│ ├── color_timestamps.csv
│ ├── depth_timestamps.csv
│ ├── ir_timestamps.csv
│ └── imu.csv
...
Every image name represents internal camera timestamp in microseconds from the start of every camera capturing process. Although, the timestamps do not belong to a common clock source, they are synchronized with sub-millisecond precision by Azure hardware by default. Leading zeros are used in names for better visibility and sorting.
If cameras are connect over distributed independent devices with known IP-addresses, the recording can be launched on a client PC and command-line may look like
./recoder.py --distributed true
On the server, to which camera is connected, server.py must be launched by uvicorn:
uvicorn server:app --host 0.0.0.0
While debugging, uvicorn may be launched with --reload option to re-launch server when server.py is updated:
uvicorn server:app --reload --host 0.0.0.0
--reload option shouldn't be set in normal working conditions since it consumes CPU power.