Skip to content

Conversation

@dtronmans
Copy link
Contributor

Purpose

Go from only being able to export with DatasetType.NATIVE to all DatasetType entries.

Specification

  • Every current implemented parser now has a matching exporter.
  • The logic for preparing the LDF (shared across all exporters), transforming it and saving are all separated
  • BaseExporter defined to be implemented by different exporters, in a very similar way that BaseParsers defines an interface for different parsers

Dependencies & Potential Impact

Deployment Plan

Testing & Validation

The first way I tested the correctness of the functionality is to parse datasets, export them using the exporters then open them with different tools like Roboflow and FiftyOne:

parsed_fiftyone parsed_dataset

The second and more precise way is by writing tests in tests/test_data/test_parse_export_equivalence.py, in which the following is tested:

  1. Parse dataset collected from Roboflow, Google Cloud, etc... and save as prev_dataset
  2. Export the dataset using defined exporters
  3. Parse this exported dataset again and save as new_dataset
  4. Assert that previous and new dataset are exactly the same. For bounding boxes, even though I try to save with as much precision as possible (.12f), we sometimes lose precision with repeated parsing and exporting so I assert that the normalized boxes are almost equal by rounding to 2 decimal places.

@dtronmans dtronmans self-assigned this Nov 10, 2025
@dtronmans dtronmans requested a review from a team as a code owner November 10, 2025 13:38
@dtronmans dtronmans requested review from conorsim, klemen1999, kozlov721 and tersekmatija and removed request for a team November 10, 2025 13:38
@dtronmans dtronmans added the data Changes affecting luxonis_ml.data subpackage label Nov 10, 2025
@github-actions github-actions bot added the enhancement New feature or request label Nov 10, 2025
Copy link
Collaborator

@kozlov721 kozlov721 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thamks. Left some few comments. Otherwise I think it looks good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Changes affecting luxonis_ml.data subpackage enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants