iCaseworks API Ingestion

Includes:

• iCaseworks API Ingestion with re-usable Lambda function
• Improvements to Google Drive Ingestion process
• Other minor script/ Glue job modifications

Rentsense ingestion

Includes:

• DPP-149 Rentsense income collection (PR)
• DPP-39 Create Qlik Service account (PR)
• DPP-157 Rentsense: Persons, ContactDetails, Assets dynamodb ingestion (PR)
• Add aws-vault install instructions for windows
• Fix department policy for accessing secrets
• Other Tascomi and Alloy script updates

DPP-134 - 1 Glue Job per filter

0.5.56

Refactor g drive lambda and spreadsheet ingestion glue job (#685)

DPP-134 - Allow jobs to auto scale using glue version 3 + testing pipeline changes

0.5.55

Fix networking pipeline

DPP-134 - Revert Split out academy ingestion jobs into separate jobs

0.5.54

Dpp 134 - Revert academy job refactoring - issue still exists (#698)

DPP-134 - Split out academy ingestion jobs into separate jobs

Dpp 134 fix 3 (#695)

* Typo

* Shorten names

Co-authored-by: ryanbratten <[email protected]>

0.5.52

What's Changed

• Create liberator_fpns_refined Glue job
• Create noisework_complaints_refined Glue job
• Ingest Ggeolive boundaries schema
• Update and fix bugs in Lambda to ingest spreadsheets from Google Drive into S3
• Remove Glue job that copies parking/manual S3 files from landing to raw as this is replaced by lambda above

0.5.51

What's Changed

• Fix permissions to department glue jobs for accessing script location
• Disable parking liberator Glue job triggers (whilst non-Terraform-ed jobs' schedules have not been turned off)
• Migrate manually uploaded CSVs to production and ingest from G Drive

0.5.50

What's Changed

• Add liberator parking jobs in terraform by @b-dalton in #639
• Add crawler to crawl all the events data into the raw zone by @elcorbs in #645
• Increase glue job worker types & timeout on academy ingestions by @elcorbs in #646

Full Changelog: 0.5.49...0.5.50

Datahub Performance improvements

0.5.49

Not required, safe cost