Thoughts on Export functionality & interface

[bradyzp]

Export Functionality

I've begun work on a prototype model and interface for exporting data from the DGP interface. Here I want to layout the basic structure of the export functionality, and solicit ideas as far as functionality that the user may need/desire from the exporter.

Structure

The current prototype export library is organized into Exporter's, ExportProfile's, and ColumnProfile's.

The Exporter class is responsible for transforming input data (DataFrame/Series) and exporting to some particular format. The intent is for the base Exporter class to be sub-classed and its abstract methods implemented to provide exporters for different data export formats.

ExportProfiles are instances of an ExportProfile class which is simply used to define common parameters for an export, most significantly the ExportProfile's are used to determine which columns (and in which order) of data are exported. Profiles are registered upon creation for later enumeration and use.

ColumnProfile's are used to define data columns that may be exported, all columns may not be available to export in any one context, but this allows the user to create ExportProfiles without regard for what is currently being exported. ColumnProfiles may also include some limited meta-data such as a description, unit-type, user-friendly name, and category/group. ColumnProfile instances are also registered for use in the Exporter framework.

ToDo

• Determine the best way for exporters to expose and accept parameters specific to their implementation, e.g. allowing a user to specify separator value for the CSV exporter but not for HDF5 where it makes no sense.
• Does the ColumnProfile framework make sense, is it extensible when new columns are introduced (e.g. new transformations)? My thinking is that any new transform graphs for example would define column profiles in their source/package for any new column types introduced (these would be automatically registered and thus available for export).

Questions/Unknowns

• Should exporters be filtered for user based on the export context (i.e. is the user exporting the DataSet, or from the transform interface, or from plotted data etc.)
• Likewise should ExportProfiles be tied to the export context and/or only available for specific Exporters?
• What options/parameters might a user need to adjust during export (both in general, and specific parameters tied to an export type e.g. HDF5 or CSV)
• How to handle user defined profiles, how/where to store user ExportProfiles (within project scope/directory, or use QSettings).
• To allow users to edit/overwrite builtin ExportProfiles, or mark builtin profiles as 'copy-only', allowing users to make a copy and then modify the profile.

User Interface

Here is a look at the working dialog implementation which simply exposes the available exporters and profiles, and allows the user to edit/modify the export profile. The exporter package remains entirely decoupled from the UI implementation, and has no dependencies on any Qt libraries.

[cbertinato]

I like the class design. Well thought out.

• Determine the best way for exporters to expose and accept parameters specific to their implementation, e.g. allowing a user to specify separator value for the CSV exporter but not for HDF5 where it makes no sense.

We could defined attributes as simple methods that raise a NotImplementedError. If they are relevant, then they will be overridden, and if not, then then won't be and will raise.

• Does the ColumnProfile framework make sense, is it extensible when new columns are introduced (e.g. new transformations)? My thinking is that any new transform graphs for example would define column profiles in their source/package for any new column types introduced (these would be automatically registered and thus available for export).

Your thinking sounds good here. We can certainly build in the functionality for the transform graph to define column profiles. In fact, I think that this is probably the best route to go because when users do eventually get the ability to write their own graphs, then it would be good to provide an easy interface with the export functionality. And it would make the export framework able to adapt to changing graph definitions without having to manually make the changes.

• Should exporters be filtered for user based on the export context (i.e. is the user exporting the DataSet, or from the transform interface, or from plotted data etc.)

This question, and the next one, are related to another concern that I had, namely: how do we handle exporting by line, DataSet, flight, project, etc.? I see a couple of possibilities:

1. Tie ExportProfiles to the context, which in this case I take to be DataSet, plot, flight, etc. The profile would dictate how exactly a flight gets exported, for example, and then defers to profiles of the things contained within a flight to determine how to export themselves. So we would have a flight export profile, line export profile, dataset export profile, etc.
2. Define a single ExportProfile and require that it define how to handle all of those cases.

• Likewise should ExportProfiles be tied to the export context and/or only available for specific Exporters?

My thought here is that ExportProfiles should not be dependent on the export file format. Of course, there are differences that we would have to account for. For example, if we were to export a project with several flights, then for a CSV we would need to create an extra column for flight number, whereas and HDF5 is structured, so we could create separate tables (or it could be flat if the user so desired). So we would need to have some awareness of the context in the Exporter as well.

• What options/parameters might a user need to adjust during export (both in general, and specific parameters tied to an export type e.g. HDF5 or CSV)

With regard to general options, I know that a diversity of time/date formats and references would be desired here. Perhaps that's something that would be covered in the column profiles. I'm sure there are more that are not coming to mind at the moment, but perhaps I'll also solicit some early opinions from my former colleagues.

• How to handle user defined profiles, how/where to store user ExportProfiles (within project scope/directory, or use QSettings).

In my experience with using Inertial Explorer, it was useful to be able to export profiles for sharing and for re-import when I had a new installation. Perhaps we can build a JSON schema, or we can pickle.

• To allow users to edit/overwrite builtin ExportProfiles, or mark builtin profiles as 'copy-only', allowing users to make a copy and then modify the profile.

I like the latter if only just to save the user from grief when they accidentally overwrite the built-in profiles. The copy and modify functionality is also something I saw in Inertial Explorer and I found it to be quite handy.

These are my initial ideas and we should certainly chat more as we put the pieces in place.