|
| 1 | +# Augmentations |
| 2 | + |
| 3 | +## `AlbumentationsEngine` |
| 4 | + |
| 5 | +The default engine used with `LuxonisLoader`. It is powered by the [Albumentations](https://albumentations.ai/) library and should be satisfactory for most use cases. Apart from the albumentations transformations, it also supports custom transformations registered in the `TRANSFORMATIONS` registry. |
| 6 | + |
| 7 | +### Configuration Format |
| 8 | + |
| 9 | +The configuration format for the `AlbumentationsEngine` consists of a list of records, where each record contain 2 fields; `name` and `params`: |
| 10 | + |
| 11 | +- `name`: The name of the transformation class to be applied (e.g., `HorizontalFlip`, `RandomCrop`, etc.). The name must be either a valid name of an Albumentations transformation (accessible under the `albumentations` namespace), or a name of a custom transformation registered in the `TRANSFORMATIONS` registry. |
| 12 | +- `params`: A dictionary of parameters to be passed to the transformation. |
| 13 | +- `use_for_resizing`: An optional boolean flag that indicates whether the transformation should be used for resizing. If no resizing augmentation is provided, the engine will use either `A.Resize` or `LetterboxResize` depending on the `keep_aspect_ratio` parameter (provided through the `LuxonisLoader`). |
| 14 | + |
| 15 | +**Example:** |
| 16 | + |
| 17 | +```yaml |
| 18 | +- name: Defocus |
| 19 | + params: |
| 20 | + p: 1 |
| 21 | +- name: Sharpen |
| 22 | + params: |
| 23 | + p: 1 |
| 24 | +- name: Affine |
| 25 | + params: |
| 26 | + p: 1 |
| 27 | +- name: RandomCrop |
| 28 | + params: |
| 29 | + p: 1 |
| 30 | + width: 512 |
| 31 | + height: 512 |
| 32 | +- name: Mosaic4 |
| 33 | + params: |
| 34 | + p: 1. |
| 35 | + out_width: 256 |
| 36 | + out_height: 256 |
| 37 | +``` |
| 38 | +
|
| 39 | +### Order of Transformations |
| 40 | +
|
| 41 | +The order of transformations provided in the configuration is not |
| 42 | +guaranteed to be preserved. The transformations are divided into |
| 43 | +the following groups and are applied in the same order: |
| 44 | +
|
| 45 | +1. Batch transformations - Subclasses of our custom `BatchTransform` |
| 46 | + |
| 47 | +1. Spatial transformations - Subclasses of `A.DualTransform` |
| 48 | + |
| 49 | +1. Custom transformations - Subclasses of `A.BasicTransform`, |
| 50 | + but not subclasses of more specific base classes above |
| 51 | + |
| 52 | +1. Pixel transformations: Subclasses of `A.ImageOnlyTransform`. |
| 53 | + These transformations act only on the image |
| 54 | + |
| 55 | +The resize transformation is applied either before or after the pixel transformations, depending on desired output size. If the output size is smaller than the initial image size, the resize transformation is applied before the pixel transformations to save compute. In the other case it is applied last. |
| 56 | + |
| 57 | +## Extensibility |
| 58 | + |
| 59 | +`LuxonisLoader` can work with any subclass of `BaseEngine` that is registered in the `AUGMENTATION_ENGINES` registry. |
| 60 | + |
| 61 | +To implement a custom augmentation engine, you need to create a new class that inherits from `BaseEngine` and implements the required methods. Any subclass of `BaseEngine` is automatically registered in the aforementioned registry. |
| 62 | + |
| 63 | +**Required Methods:** |
| 64 | + |
| 65 | +- `__init__`: The constructor method that initializes the engine with the provided configuration. It needs to create a new instance of the augmentation engine from the following arguments: |
| 66 | + - `height`: The output height of the images |
| 67 | + - `width`: The output width of the images |
| 68 | + - `n_classes`: The number of classes in the dataset |
| 69 | + - `config`: The configuration for the augmentation engine as an iterable of dictionaries. Interpretation of the configuration is left to the engine (it doesn't need to follow the format used in `AlbumentationsEngine`). |
| 70 | + - `keep_aspect_ratio`: A boolean flag that indicates whether to keep the aspect ratio of the images during resizing. The engine should respect this flag when applying the resizing transformation. |
| 71 | + - `is_validation_pipeline`: A boolean flag that indicates whether the engine is being used for validation. Typically, the applied transformations differ between training and validation (for example validation pipeline would only use resizing and normalization). |
| 72 | + - `targets`: A dictionary mapping names of individual labels (given in `apply`) to their respective label types. Possible values of the label types are `"boundingbox"`, `"segmentation"`, `"instance_segmentation"`, `"keypoints"`, `"array"`, `"classification"`, and `"metadata"`. Interpretation of the targets is left to the engine. |
| 73 | +- `apply`: This method applies the augmentation engine to the provided batch of images and labels. It should return a tuple containing the augmented images and labels. The method should also handle the resizing of images and targets according to the specified output size and aspect ratio. |
| 74 | +- `batch_size`: An abstract property that returns the expected batch size of the inputs. This is required for the `LuxonisLoader` to properly handle the input data for batched augmentations. For example if the pipeline contains `MixUp` augmentation (which requires 2 images) and `Mosaic4` (requiring 4 images), the batch size should be $$2 * 4 = 8$$. |
0 commit comments