Replicate GOV.UK databases directly instead of via backups

[nacnudus]

Trello

Currently we get the data via https://github.com/alphagov/govuk-s3-mirror, which clones nightly backups of GOV.UK databases from GOV.UK's integration environment to a bucket. We restore the backup file to a running instance of the database, extract what we need, and also copy the original tables into BigQuery. This is a delayed and fragile batch process, running from a non-production environment. Direct replication would be:

• From a production environment
• More reliable, presumably
• Less delayed
• Streamed, not batch
• more expensive, because we would have to pay for a constantly running Cloud SQL instance, instead of ~1h/day compute engine instances

https://cloud.google.com/database-migration/docs/postgres/configure-source-database