Release prior to merging in datahub

Merge pull request #590 from LBHackney-IT/LBHSBALLEY-patch-4

Create 30-better-conversations-tables-ingestion.tf

Release prior to merging in datahub

Disable glue job triggers on sandbox glue jobs (#591)

Stop them from running in pre-prod and prod as it's not necessary
since these were used for training

Secret manager permissions, Tascomi Tables updates, Glue job permissions and capacity, Sagemaker deployment

• All departments can access the list of secrets, in order to access the secrets they are specifically allowed to access
• Create and update Tascomi tables
• Gives necessary s3 and kms key permissions for glue jobs
• Increase glue job capacity to allocate more memory and storage
• Sagemaker admin notebook deployment

Give ECS sync task more kms permissions

0.5.35

Allow all kms actions except specific dangerous ones on kms keys (#580)

Correct fargate ecs sync task configuration value

0.5.34.1

Correct ecs sync task memory value & format

Increase capacity for ECS sync task

• Increase cpu and memory capacity for raw zone ecs sync task

Also includes:

• Add missing permissions to allow ECS sync task to delete objects
• Correct spark ui logs s3 path

Adds missing KMS permissions to ECS sync task role

Also includes:

• updates to Glue connection permissions on the generic glue role
• updates schedule to ingest Academy data earlier
• notebook to create column type dictionary

MSK cluster

Also includes:

• Kafka schema registry
• Give glue role permission to assume all roles
• Crawl housing raw zone after copying dynamo table data to raw

Create two new ECS tasks to sync prod to pre-prod trusted & refined zones

Also includes the following:

• copying modern tools for housing dynamo DB tables from landing to raw - #566
• Create a department for housing - #564
• bump tflint aws ruleset version

0.5.30

DP-582 - Part 2 (#562)