Skip to content

Commit fd2065f

Browse files
authored
Add a guide for adjusting airflow scheduler resources (#872)
1 parent 4bb87b2 commit fd2065f

File tree

1 file changed

+36
-0
lines changed

1 file changed

+36
-0
lines changed
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
## Adjusting Scheduler Resources
2+
3+
In Airflow, the [scheduler](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html)
4+
monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.
5+
6+
You may find that you need to increase the number of schedulers that you are running as well as increase the resources
7+
available to each scheduler.
8+
9+
### edit values.yaml
10+
11+
In order to update scheduler resources, you'll then want to edit `airflow/helm/airflow/values.yaml` in your installation
12+
repo. If you wanted 2 schedulers each with at least 0.5 cpu & 1.75Gi of resources, you'd make the following change:
13+
14+
```yaml
15+
airflow:
16+
airflow:
17+
airflow:
18+
scheduler:
19+
replicas: 2
20+
resources:
21+
limits:
22+
cpu: 1.5
23+
memory: 2.5Gi
24+
requests:
25+
cpu: 0.5
26+
memory: 1.75Gi
27+
```
28+
29+
> It is highly recommended to set limits on your scheduler(s) to ensure fair resource sharing across your cluster.
30+
31+
Alternatively, you should be able to do this in the configuration section for airflow in your plural console as well.
32+
33+
### redeploy
34+
35+
From there, you should be able to run `plural build --only airflow && plural deploy --commit "update scheduler
36+
resources"` to adjust your Airflow scheduler configuration

0 commit comments

Comments
 (0)