Datahub Performance improvements

0.5.48

Dpp 131 pt2 (#643)

DPP-38 - Ingesting CSV files from google drive

0.5.47

DPP-38 - Ingest CSV from Google Drive (#634)

0.5.46

What's Changed

• Update 26-google-sheets-imports.tf by @JBLayton in #628
• Update 26-google-sheets-imports.tf by @sanch-hackney in #629
• Get av zone & vpc ID from the subnet used for the jdbc connection by @elcorbs in #631
• Tascomi trusted fix by @LBHSBALLEY in #636
• add alloy aqs queries by @timburke-hackit in #627

New Contributors

• @JBLayton made their first contribution in #628

Full Changelog: 0.5.45...0.5.46

0.5.45

What's Changed

• Update job_script_template.py by @LBHSBALLEY in #621
• Create a kafka connector for each topic by @elcorbs in #626

Full Changelog: 0.5.44...0.5.45

Post new schema to the registry as a part of the terraform run

What's Changed

• Add import date as a partition to event streaming output data by @elcorbs in #623
• Post new schemas to the kafka schema registry by @elcorbs in #625

Full Changelog: 0.5.43...0.5.44

Manage kafka connectors in production

What's Changed

• Use terraform to deploy kafka connectors by @elcorbs in #619
• Add a later run to the lambda that shuts down notebooks by @elcorbs in #618
• Tascomi trusted job faster and clearing target bucket before writing by @LBHSBALLEY in #614
• Update 26-google-sheets-imports.tf by @sanch-hackney in #622
• missing comma in Tascomi trusted job by @LBHSBALLEY in #624

Full Changelog: 0.5.42...0.5.43

Datahub Google Auth

Kafka connector missing security group rules & s3 permissions (#617)

* Fix up security group for connector

* Fix up s3 permissions for kafka connect

* Rename terraform output

SFTP server using AWS transfer & ringo user

Aws transfer (#595)

* AWS transfer server with ringo user

* Updated permissions following code review

- Removed edit/delete access on S3 for external provider
- Added specific permissions for KMS
- Specify the SFTP server as resource

* Ringo transfer user can only to the ringo prefix in landing zone, update logging role for the server

Co-authored-by: Emma <[email protected]>

Enable autoscaling on Datahub ECS services

Improves performance of Datahub

Also includes:

• updates to Sagemaker notebook permissions

Datahub

0.5.39

Create kafka schema registry log group (#600)