Crafted an ETL pipeline to handle 26 million user ratings and about 45,000 movies. The pipeline has the potential of ingesting data at an efficiency of 10,000 records per minute into AWS Redshift. Implemented a standardized data model and automated data quality checks using Airflow, contributing to a 97% success rate for regular ETL cycles.
-
Notifications
You must be signed in to change notification settings - Fork 0
Crafted an ETL pipeline to handle 26 million user ratings and about 45,000 movies. The pipeline has the potential of ingesting data at an efficiency of 10,000 records per minute into AWS Redshift. Implemented a standardized data model and automated data quality checks using Airflow, contributing to a 97% success rate for regular ETL cycles.
License
ManoharVit/MoviETL-Data-Pipeline
About
Crafted an ETL pipeline to handle 26 million user ratings and about 45,000 movies. The pipeline has the potential of ingesting data at an efficiency of 10,000 records per minute into AWS Redshift. Implemented a standardized data model and automated data quality checks using Airflow, contributing to a 97% success rate for regular ETL cycles.
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published