+ | Name | Type, possible values | Description |
|---|
+
+ | room_size_multiplier_min |
+ Float > 1.0, < room_size_multiplier_max |
+
+ Minimum room size as a factor of the biggest sampled target object. The room is cubic.
+ The size of the biggest object is the length of the largest diagonal of its axis-aligned bounding box. This tends to overestimate the size of the object.
+ If the size of the biggest object is 0.5m, room_size_multiplier_max = 2 and room_size_multiplier_max = 4, then the room's size will be randomly sampled to be between 1m and 2m.
+ |
+
+
+ | room_size_multiplier_max |
+ Float > room_size_multiplier_min |
+
+ Minimum room size as a factor of the biggest sampled target object. The room is cubic.
+ The size of the biggest object is the length of the largest diagonal of its axis-aligned bounding box. This tends to overestimate the size of the object.
+ If the size of the biggest object is 0.5m, room_size_multiplier_max = 2 and room_size_multiplier_max = 4, then the room's size will be randomly sampled to be between 1m and 2m.
+ |
+
+
+
+ | simulate_physics |
+ Boolean |
+ Whether to simulate physics. If false, then objects will be floating across the room. If true, then objects will fall to the ground. |
+
+
+ | max_num_textures |
+ Int > 0 |
+ Max number of textures per blenderproc run. If scenes_per_run is 1, max_num_textures = 50 and the number of distractors is more than 50, then the 50 textures will be used across all distractors (and walls). In this case, new materials will be sampled for each scene. |
+
+
+ | distractors |
+ Dictionary |
+ See below |
+
+
+ | lights |
+ Dictionary |
+ See below |
+
+
+ | objects |
+ Dictionary |
+ See below |
+
+
+
+Distractors are small, simple objects that are added along with the target objects to create some variations and occlusions. You can also load custom objects as distractors.
+To modify their properties, you can change the "distractors" field of the scene
+
+ | Name | Type, possible values | Description |
|---|
+
+ | min_count |
+ Int >= 0, < max_count |
+
+ Minimum number of distractors to place in the room.
+ |
+
+
+ | max_count |
+ Int > min_count |
+
+ Maximum number of distractors to place in the room.
+ |
+
+
+ | custom_distractors |
+ string, or null |
+
+ If not null, path to a folder containing custom distractor objects, in .obj or .ply format.
+ |
+
+
+ | custom_distractor_proba |
+ float, >= 0.0, <= 1.0 |
+
+ If custom_distractors is not null, probability that a distractor is sampled from the user specified distractors.
+ |
+
+
+ | min_size_rel_scene |
+ Float > 0.0 |
+
+ Minimum size of the distractors, relative to the room size.
+ If the room size is 0.5m and min_size_rel_scene = 0.1, then the minimum size of distractor will be 0.05m.
+ Scale is applied independently on each axis.
+ |
+
+
+ | max_size_rel_scene |
+ Float < 1.0, > min_size_rel_scene |
+
+ maximum size of the distractors, relative to the room size.
+ If the room size is 0.5m and max_size_rel_scene = 0.2, then the maximum size of distractor will be 0.1m.
+ Scale is applied independently on each axis.
+ |
+
+
+ | displacement_max_amount |
+ Float >= 0.0 |
+
+ Amount of displacement to apply to distractors.
+ Displacement subdivides the mesh and displaces each of the distractor's vertices according to a random noise pattern.
+ This option greatly slows down rendering: set it to 0 if needed.
+ |
+
+
+ | pbr_noise |
+ Float >= 0.0 |
+
+ Amount of noise to add to the material properties of the distractors.
+ These properties include the specularity, the "metallicness" and the roughness of the material, according to Blender's principled BSDF.
+ |
+
+
+ | emissive_prob |
+ Float >= 0.0 , <= 1.0 |
+
+ Probability that a distractor becomes a light source: its surface emits light. Set to more than 0 to add more light variations and shadows.
+ |
+
+
+ | emissive_min_strength |
+ Float >= 0.0, < emissive_max_strength |
+
+ Minimum emission strength for a distractor that emits lights. In Watts/m².
+ |
+
+
+ | emissive_max_strength |
+ Float > emissive_min_strength |
+
+ Maxmimum emission strength for a distractor that emits lights. In Watts/m².
+ |
+
+
+
+To change the lighting behaviour, see the options below:
+
+ | Name | Type, possible values | Description |
|---|
+
+ | min_count |
+ Int >= 0, < max_count |
+
+ Minimum number of target objects in the scene.
+ |
+
+
+ | max_count |
+ Int > min_count |
+
+ Maximum number of target objects in the scene.
+ |
+
+
+ | multiple_occurences |
+ Boolean |
+
+ Whether a single object can appear multiple times in the same scene (sampling with replacement).
+ |
+
+
+ | scale_noise |
+ Float >= 0.0 |
+
+ Object size noise. if scale_noise > 0.0, the object is scaled uniformly on all axes (it does not appear stretched)
+ |
+
+
+ | displacement_max_amount |
+ Float >= 0.0 |
+
+ Amount of displacement to apply to target objects.
+ Displacement subdivides the mesh and displaces each of the distractor's vertices according to a random noise pattern.
+ This option greatly slows down rendering: set it to 0 if needed.
+ Note that this is in absolute units: and does not vary depending on the size of the object.
+ |
+
+
+ | pbr_noise |
+ Float >= 0.0 |
+
+ Amount of noise to add to the material properties of the target objects.
+ These properties include the specularity, the "metallicness" and the roughness of the material, according to Blender's principled BSDF.
+ |
+
+
+ | cam_min_dist_rel |
+ Float >= 0.0, < cam_max_dist_rel |
+
+ Minimum distance of the camera to the point of interest of the object when sampling camera poses. This is expressed in terms of the size of the target object.
+ If the target object has a size of 0.5m and cam_min_dist_rel = 1.5, then the closest possible camera will be at 0.75m away from the point of interest.
+ |
+
+
+ | cam_max_dist_rel |
+ Float >= cam_min_dist_rel |
+
+ Maximum distance of the camera to the point of interest of the object when sampling camera poses. This is expressed in terms of the size of the target object.
+ If the target object has a size of 0.5m and cam_max_dist_rel = 2.0, then the farthest possible camera will be 1m away from the point of interest.
+ |
+
+
+
+Finally, we can customize the dataset that we will generate from the given scenes.
+This includes the number of scenes, images, and what information to save.
+
+All the data will be stored in HDF5 format, which can then be unpacked later.
+
+To customize the dataset, modify the options in the "dataset" field:
+
+ | Name | Type, possible values | Description |
|---|
+
+ | save_path |
+ String |
+ Path to the folder that will contain the final dataset. This folder will contain one folder per scene, and each sample of a scene will be its own HDF5 file. |
+
+
+ | scenes_per_run |
+ Int > 0 |
+
+ Number of scenes to generate per blenderproc run. Between blenderproc runs, Blender is restarted in order to avoid memory issues.
+ |
+
+
+ | num_scenes |
+ Int > 0 |
+
+ Total number of scenes to generate. Generating many scenes will add more diversity to the dataset as object placement, materials and lighting are randomized once per scene.
+ |
+
+
+ | images_per_scene |
+ Int > 0 |
+
+ Number of images to generate per scene. The total number of samples in the dataset will be num_scenes * (images_per_scene + empty_images_per_scene).
+ |
+
+
+ | empty_images_per_scene |
+ Int >= 0, <= images_per_scene |
+
+ Number of images without target objects to generate per scene. The camera poses for these images are sampled from the poses used to generate images with target objects. Thus, the only difference will be that the objects are not present, the rest of the scene is left untouched.
+ |
+
+
+ | pose |
+ Boolean |
+
+ Whether to save the pose of target objects that are visible in the camera. The pose of the objects are expressed in the camera frame as an homogeneous matrix \f$^{c}\mathbf{T}_{o}\f$
+ |
+
+
+ | depth |
+ Boolean |
+
+ Whether to save the depth buffer associated to the RGB image. Same size as the RGB image.
+ |
+
+
+ | normals |
+ Boolean |
+
+ Whether to save the normal map associated to the RGB image. Same size as the RGB image.
+ The normals are 3D unit vectors, expressed in the camera frame.
+ |
+
+
+ | segmentation |
+ Boolean |
+
+ Whether to save the segmentation maps (by class and by instance).
+ Segmentation by class only contains the target objects (class >= 1).
+ Segmentation by instance includes every visible object.
+ |
+
+
+ | detection |
+ Boolean |
+
+ Whether to save the bounding box detections. In this case, bounding boxes are not computed from the segmentation map (also possible with Blenderproc), but rather in way such that occlusion does not influence the final bounding box.
+ The detections can be filtered with the parameters in "detection_params".
+ |
+
+
+ | detection_params:min_size_size_px |
+ Int >= 0 |
+
+ Minimum side length of a detection for it to be considered as valid. Used to filter really far or small objects, for which detection would be hard.
+ |
+
+
+ | detection_params:min_visibility |
+ Float [0.0, 1.0] |
+
+ Percentage of the object that must be visible for a detection to be considered as valid. The visibility score is computed as such:
+ First, the vertices of the mesh that are behind the camera are filtered. Then, the vertices that are outside of the camera's field of view are filtered. Then, we randomly sample "detection_params:points_sampling_occlusion" points to test whether the object is occluded (test done through ray casting).
+ If too many points are filtered, then the object is considered as not visible and detection is invalid.
+ |
+
+
+
+
+\section dnn_run_script Running the script to generate data
+
+Once you have configured the generation to your liking, navigate to the `script/dataset_generator` located in your ViSP source directory.
+
+You can then run the `generate_dataset.py` script as such
+\code{.sh}
+(blenderproc) user@machine:~/visp/script/dataset_generator $ python generate_dataset.py --config path/to/config.json
+\endcode
+
+If all is well setup, then the dataset generation should start and run.
+
+
+\warning If during generation, you encounter a message about invalid camera placement, try to make room_size_multiplier_min and room_size_multiplier_max larger, so that more space is available for object placement.
+
+To give an idea of generation time, generating 1000 images (with a resolution of 640 x 480) and detections of a single object, with a few added distractors, takes around 30mins on a Quadro RTX 6000.
+
+Once generation is finished, you are ready to leverage the data to train your neural network.
+
+\section dnn_output Using and parsing the generation output
+
+The dataset generated by Blender is located in the "dataset:save_path" path that you specified in your JSON configuration file.
+
+The dataset has the following structure
+\verbatim
+- dataset
+ - 0
+ - 0.hdf5
+ - 1.hdf5
+ - ...
+ - 1
+ - 0.hdf5
+ - 1.hdf5
+ - ...
+ - ...
+ - classes.txt
+\endverbatim
+
+There is one subfolder per scene, and each HDF5 file is a single sample that may contain RGB, depth, normals, etc.
+
+You can visualise the generated images (RGB, depth, normals, segmentation maps) by running
+\code{.sh}
+(blenderproc) $ blenderproc vis hdf5 path/to/your/output/0/*.hdf5
+\endcode
+
+You will then see something that looks like this, depending on the outputs that you chose in the configuration file:
+\image html blenderproc_viz.png
+
+where you can replace the "0" in the path with another number, which corresponds to the generated scene index.
+You can also specify the path to a single HDF5 file to view one sample at a time.
+
+\subsection dnn_output_yolov7 Using detections to finetune a YoloV7
+
+To help, we provide a script that reformats a blenderproc dataset to the format expected by YoloV7.
+This script can be run like this:
+\code{.sh}
+(blenderproc) $ python export_for_yolov7.py --input path/to/dataset --output path/to/yolodataset --train-split 0.8
+\endcode
+
+here "--input" indicates the path to the location of the blenderproc dataset, while "--output" points to the folder where the dataset in the format that YoloV7 expects will be saved. "--train-split" is an argument that indicates how much of the dataset is kept for training. A value of 0.8 indicates that 80% of the dataset is used for training, while 20% is used for validation. The split is performed randomly across all scenes (a scene may be visible in both train and validation sets).
+
+Once the script has run, the folder "path/to/yolodataset" should be created and contain the dataset as expected by YoloV7.
+This folder contains a "dataset.yml" file, which will be used when training a YoloV7. It contains:
+the following:
+
+```
+names:
+- esa
+nc: 1
+train: /local/sfelton/yolov7_esa_dataset/images/train
+val: /local/sfelton/yolov7_esa_dataset/images/val
+```
+where nc is the number of class, "names" are the class names, and "train" and "val" are the paths to the dataset splits.
+
+To start training a YoloV7, you should download the