Permanent solution for tracking/testing assets

[timothy-nunn]

Introduction

1. A PR is merged into main.
2. The CI system runs all of the regression input files and produces several MFiles. From these MFiles, it extracts key variables to be 'tracked'.
3. The CI system uploads these MFiles and tracking files to an external repository:

• The MFiles are downloaded by the regression tests both locally and on the CI to provide the 'reference' MFile against which codes changes are tested.
• The tracking files are used by the tracker to create a dashboard showing how key outputs change over time

The current state

The external data repository is currently a GitHub repository (https://github.com/timothy-nunn/process-tracking-data).

This is practically the industry-antistandard and was always intended to be a temporary solution. #3362 exposed the fragility of this solution due to the 1000 file limit on the GitHub API (regression tests silently used incorrect test assets and some tests were skipped where no test assets could be found).

The solution

It is clear that the current system is unsustainable and must be replaced with a proper solution.

I recommend a database solution optimised for storing files and key-value pairs. Some points to consider:

• The tracking data is JSON and could be stored as key-value pairs. The MFiles could then be stored separately in the database or a pointer to them in some CDN.
• In the future (medium to long term) the MFile will no longer exist and will be replaced by (probably) a JSON-compatible structure.
• The data is tree-like (data points for most commits on main, therefore there is some order).