daily-data-analysis-airflow-pyspark

Here is 1 public repository matching this topic...

Undisputed-jay / Airflow-ETL-Pipeline-with-PySpark-and-Google-Cloud-Dataproc

This project automates daily vehicle data processing on Google Cloud using Apache Airflow. It uploads scripts to Google Cloud Storage, runs specific PySpark jobs on Dataproc based on the day, and shuts down resources when done for efficiency.

cost-effective-data-processing daily-data-analysis-airflow-pyspark automated-etl-airflow-dataproc

Updated Nov 1, 2024
Python

Improve this page

Add a description, image, and links to the daily-data-analysis-airflow-pyspark topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the daily-data-analysis-airflow-pyspark topic, visit your repo's landing page and select "manage topics."

Learn more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

daily-data-analysis-airflow-pyspark

Here is 1 public repository matching this topic...

Undisputed-jay / Airflow-ETL-Pipeline-with-PySpark-and-Google-Cloud-Dataproc

Improve this page

Add this topic to your repo