Log groomer sidecars do propagate secrets configurations causing failures

[JDKnobloch]

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.10.3

Kubernetes Version

1.30

Helm Chart configuration

config:
  secrets:
    backend: "custom_providers.vault_secret.VaultSecret"
    backend_kwargs: '{"connections_path": "airflow/connections", "mount_point": null, "url": "vault.vault.svc.cluster.local", "auth_type": "aws_iam"}'
env:
  - name: "AIRFLOW__SECRETS__BACKEND"
     value: "custom_providers.vault_secret.VaultSecret"
  - name: "AIRFLOW__SECRETS__BACKEND_KWARGS"
    value: "{\"connections_path\": \"airflow/connections\", \"mount_point\": null, \"url\": \"vault.vault.svc.cluster.local\", \"auth_type\": \"aws_iam\"}"

Docker Image customizations

No response

What happened

All log groomer sidecars (both on triggerer and scheduler) fail w/ the error:

ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in
from airflow.main import main
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/init.py", line 53, in
from airflow import configuration, settings
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/configuration.py", line 2371, in
secrets_backend_list = initialize_secrets_backends()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/configuration.py", line 2279, in initialize_secrets_backends
custom_secret_backend = get_custom_secret_backend()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/configuration.py", line 2267, in get_custom_secret_backend
return secrets_backend_cls(**backend_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/hashicorp/secrets/vault.py", line 141, in init
self.vault_client = _VaultClient(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/hashicorp/_internal_client/vault_client.py", line 134, in init
raise VaultError("The 'token' authentication type requires 'token' or 'token_path'")
hvac.exceptions.VaultError: The 'token' authentication type requires 'token' or 'token_path', on None None

When I ssh to the container and echo for the env variables in question the following outputs occur:

airflow@ci-airflow-triggerer-0:/opt/airflow$ echo $AIRFLOW__SECRETS__BACKEND
custom_providers.vault_secret.VaultSecret
airflow@ci-airflow-triggerer-0:/opt/airflow$ echo $AIRFLOW__SECRETS__BACKEND_KWARGS

airflow@ci-airflow-triggerer-0:/opt/airflow$

I can also see that the env section is properly filled out for other sidecars / init containers in the live manifest but not for groomers - running echos from the main containers have the expected results.

What you think should happen instead

The groomers should populate configuration properly and be able to function when using a Vault secret backend.

How to reproduce

Attempt to use groomers while supplying a Vault secret backend

Anything else

Initially we were supplying these arguments only via env variables - at which point the Jobs created by the chart (create-user and run-airflow-migration) were also failing with the same logs / the env variables are not populated properly for these either.

Changing to hard coding the Values.config section fixed this issue. We attempted only env variables, only Values.config, and both, and none populate properly.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[gbazad93]

It seems the log groomer sidecars aren't picking up the necessary environment variables for the Vault secret backend, particularly AIRFLOW__SECRETS__BACKEND_KWARGS. Since sidecars run in separate containers, they might not inherit the env vars you've set elsewhere.

I believe you can fix this by adding the required environment variables directly to the log groomer sidecar configuration in your Helm chart. Try adding the following to your values.yaml:

logGroomerSidecar:
  enabled: true
  extraEnv:
    - name: AIRFLOW__SECRETS__BACKEND
      value: custom_providers.vault_secret.VaultSecret
    - name: AIRFLOW__SECRETS__BACKEND_KWARGS
      value: '{"connections_path": "airflow/connections", "mount_point": null, "url": "vault.vault.svc.cluster.local", "auth_type": "aws_iam"}'