Patch: Changed Production Pipeline Config Target

The production pipeline was exporting config to the staging config file resulting in the config not being set.

Patch Networking

The production pipeline was missing the required block that copies that AWS VPC ID from the Networking Module to the Main Module.

Rearrange networking, and add Liberator downloader

Merge pull request #85 from LBHackney-IT/liberator-data-sftp-tp-s3

Lambda to move liberator data from sftp server to s3

Update UPRN rds instance

Update UPRN rds instance

Remove unnecessary import

Remove unnecessary import from the copier lambda. This was causing the lambda to error

Increase export capacity of copier lambda

What

added an additional check to the S3 copier lambda to check if the table objects in the request output is truncated, i.e. there are more objects that still need to be copied and if so copy them as well

Why

Previously, the request output was truncated, meaning not all the tables were being listed and therefore some tables were not being copied over to the Data Platform landing zone

Increase timeout on Ingestion and copier lambda

This is to ensure that the lambdas can finish executiing their tasks

Add Golive RDS instance

An rds instance id has been added to the production environment variables.

Injest Housing UPRN data onto platform

This release will allow us to ingest Housing UPRN data from Production APIs account into the Data Platform Production account landing zone bucket.

It follows a partitioning strategy which is as follows:

{department/service}/{dataset-name}/import_year={import-year}/import_month={import-month}/import_day={import-day}/{file-name}

This process is outlined in the documentation

Export snapshots to s3

Implementing the following process for exporting RDS snapshots from a service account to an S3 bucket in the data platform.