External sources - testing and resilience and automated updates #287
Description
For many of the external datastore sources we rely on 3rd party data sources which may be subject to changes outside of our knowledge. For this reason some of the sources update on a manual basis, we do this so that we can monitor and roll back changes before they affect the data pipeline. However this relies on us remember and baby sitting the update process.
We could use a combination of cron style / github actions and unit tests to verify the 3rd party sources are compatible before they update the live data. There also could be an improved way to roll back this data in the database if there were problems (though this might just be a good use of transactions) .