Skip to content

feat: Make kubernetes executor default for airflow #869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: Make kubernetes executor default for airflow #869

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1177f32
Make kubernetes executor default for airflow
michaeljguarino Oct 6, 2023
1bfedf7
bump pod creation batch size
michaeljguarino Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airflow/helm/airflow/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: airflow
description: A Helm chart for airflow deployable on plural
type: application
version: 0.3.39
version: 0.3.40
appVersion: "2.6.3"
sources:
- https://github.com/pluralsh/plural-artifacts/airflow/helm/airflow
Expand Down
22 changes: 17 additions & 5 deletions airflow/helm/airflow/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ httpConfig:
password: dummy

secrets:
createPluralRedisSecret: false
createPluralRedisSecret: true
redis_password: example

test-base:
Expand Down Expand Up @@ -257,7 +257,7 @@ airflow:

flower:
enabled: false

rbac:
create: true

Expand All @@ -268,7 +268,7 @@ airflow:
image:
registry: gcr.io/pluralsh
tag: 6.0.9-debian-10-r0
enabled: true
enabled: false
existingSecret: airflow-redis-password
existingSecretPasswordKey: redis-password

Expand Down Expand Up @@ -310,11 +310,20 @@ airflow:
passwordSecretKey: redis-password

airflow:
executor: CeleryExecutor
executor: KubernetesExecutor
image:
repository: ghcr.io/pluralsh/containers/apache/airflow
tag: 2.6.3-python3.10-plural1.4.4

kubernetesPodTemplate:
resources:
requests:
cpu: 100m
memory: 500Mi
limits:
cpu: 500m
memory: 1.5Gi

config:
## security
AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
Expand All @@ -324,6 +333,9 @@ airflow:
## dags
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "30"

## k8s executor
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE: "5"

## remote log storage
AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "plural"
Expand Down Expand Up @@ -383,7 +395,7 @@ airflow:
workers:
## if the airflow workers StatefulSet should be deployed
##
enabled: true
enabled: false

replicas: 2

Expand Down