Support Airflow-specific annotations to schedules, including environment-specific schedules #35
Description
The new annotations spec will be released to Meltano soon. This spec allows projects to define deploy hints and other rules for 3rd-party tools.
schedules:
- name: slack_notifications
interval: 0 12 * * *
job: slack_notifications
annotations:
airflow:
disabled: True
- name: cloudwatch_el
interval: 0 6 * * *
job: cloudwatch_el
annotations:
airflow:
environments: [prod, staging]
- name: meltanohub_el_staging
interval: 0 6 * * *
job: meltanohub_el
annotations:
airflow:
environments: [prod]
- name: spreadsheets_anywhere_el
interval: 0 6 * * *
job: spreadsheets_anywhere_el
annotations:
airflow:
environments: [prod]
- name: daily_staging_dry_run
interval: 0 6 * * *
job: spreadsheets_anywhere_el
annotations:
airflow:
environments: [staging]
The logic here will be ignored by Meltano Core - but it can be used within airflow-ext to modify how schedules will be executed.
The above logic handles two basic rules:
- If
disabled=True, then the job would be disabled for all environments - or simply not deployed to Airflow at all. - If
environmentsis declared, the schedule will only be run if it contains a match to the name of the currently activated Meltano Environment.