:bulb: [Feature] Backup *representative* data functionality

[fmigneault]

Description

Using #532, options are provided for backup/restore of databases. This is perfect for recreating VMs with exact configurations. However, as detailed in #534, any DB migration that might be required to switch versions might make this backup data unusable.

Given the varying complexity of various DB migration technologies, one approach could be to simply export the data in a format that is appropriate for the API service that uses the database. Although not as efficient as direct backup/restore of the database, going through the API and underlying data representation can help migrate data across versions, and even potentially across servers not using birdhouse-deploy, while leveraging similar APIs.

Therefore, the backup [create,restore] CLI mode provided by #532 should be further extended to provide an export, which would perform the relevant API/FileSystem/whatever calls to get a "relevant data representation" for that API. An import could be considered as well.

An example use case is STAC API. The exported data should be a nested hierarchy of collections/items. Those could then be reimported in another server / migrated DB using STACpopulator.implementations.DirectoryLoader.

References

• Backup birdhouse data to a restic repository (as a command line option and as a scheduler job) #532
• STAC DB migration for 0.6.10->0.6.13 [WIP post 0.7.x] #534
• https://github.com/crim-ca/stac-populator/tree/master/STACpopulator/implementations/DirectoryLoader

Concerned Organizations

all

[mishaschwartz]

I really like this idea and it should be relatively easy to adapt the backup feature introduced in #532 to accommodate this.

It would be useful here to identify which services could be exported/imported in this way. So far I'm thinking:

• stac as JSON files
• all the postgres data (WPS, geoserver, magpie, ...) using COPY ... WITH (FORMAT CSV)
  • though this will break if a later migration changes the table definitions (column names, data types, etc.)
• all the mongodb data (weaver, cowbird) using mongoexport and mongoimport:
  • this probably has the same issue as the postgres one

Is STAC the only safe one? Am I missing something?

I've already started working on an export solution for stac as part of the stac-populator project so that could be used as part of this. I'll link that here once I've made a PR for that.

[tlvu]

Great idea !

Since we are on the topic of "restore compatibility" how about also recording the various versions that was "effective" at the moment of the backup so on restore we can have more "clues" to ensure compatibility. Maybe a file called <same filename as the backup>.env.

A few versions come to mind:

• birdhouse-deploy version
• image name of each DB and matching services (ex: magpie and prosgres)
  • some of these images are hardcoded directly in the docker-compose-extra.yml by the way, they should be converted to variable so we can set them on restore, via env.local.
• image name of other services that have data (ex: prometheus)
• version of Restic itself !

If we have proper snapshot of the original state, I think we can easily "work-around" any incompatibilities regarding the data format not matching the application version.

This do not replace exporting DB so the data format is easier to restore. This is just an added safety net, if things go south. If all the variables already exist in default.env or env.local, we just dump them to a file, it's not too much extra work for the added safety net.

[tlvu]

• image name of each DB and matching services (ex: magpie and prosgres)

  • some of these images are hardcoded directly in the docker-compose-extra.yml by the way, they should be converted to variable so we can set them on restore, via env.local.

• image name of other services that have data (ex: prometheus)

I just started looking at #532. The env.local is backuped too. In this case, the infos above are not necessary anymore.

But the version of birdhouse-deploy will still be useful. In fact the versions or commit hash of all external repos declared in env.local as well.

[fmigneault]

@mishaschwartz

I guess it would have to be APIs that allow POST/create of the contents/resources to support import.
APIs that do not support POST/create could be limited to export only (eg: THREDDS? not sure if that could be of use?)
I see at STAC, Weaver, Magpie, Cowbird, GeoServer, and the monitoring services as potential candidates for import/export. These are sorted by priority IMO, but feel free to consider other ordering if gradual integration is desired.

The idea of import/export would be to purposely avoid databases, going through the respective APIs of the services themselves, such that internal migration concerns can be avoided. It would have to be a "last resort" if the faster backup create/restore cannot be performed directly, or if the objective is to export where databases do not apply.

The WPS birds' process definitions are enforced by their python code, so they shouldn't be needed.

---

@tlvu
Would the best solution be to simply export the active env variables, so you get every possible value that could be relevant for the exported data?

That would mean env.local is not enough, because the default versions that are not overridden by env.local would be lost at some point, unless a BIRDHOUSE_VERSION or BIRDHOUSE_COMMIT is saved as well.

I think the easiest route is just to export the env. Sometimes, such as in the case of STAC, you could have a specific API version and specific docker image, and still have different data compatibility issues depending on the stac_version or stac_extensions returned by the JSON. Changing API/image version would not impact at all how the data is represented. I wouldn't be surprised that other APIs have their own versioning considerations as well. I think it is better we do not dive too specifically into these implementation details at the birdhouse level. Having all relevant env variables should be sufficient to provide enough context about where the data came from to address import issues.

[mishaschwartz]

@fmigneault

I see what you mean. However we need to be careful about ensuring that the version information is kept because even APIs can have backwards incompatible changes between major versions.

It's definitely possible that STAC could change the JSON format between major versions making a manual migration required there as well.

I'm not saying that we shouldn't do this. I just think we need to be aware that this won't necessarily solve all the problems we are currently seeing when we try to restore databases.

[fmigneault]

I think if we are equipped with backup create/restore/import/export and have all the environment variables related to a given backup call, we would have a sufficiently good toolset and enough version information to handle most complex data management situations. If any migration mismatch still happens beyond this, it most probably is a situation that cannot be easily/generally handled anyway, or the export/import use case is simply not possible because of a limitation imposed by the API provider.

Normally, if an API changes drastically, there will either be a new /v2/ endpoint, or the API provider will indicate how to migrate from v1 to v2 with a very specific operation. I consider both of these cases to be out of scope of this feature, since the fix would realistically be applied outside birdhouse-deploy (it's too specific to a given service).

The main takeaway I want to highlight is that exporting env.local is not sufficient, since most of the services could be left to their default version, or could have custom settings modifying their behavior even for a known version. We would need (at least) something like bin/birdhouse configs -c env to figure out what was actually applied to replicate an instance.