Problem with Rotation of Fernet Key

[dacamposol]

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.9.3

Kubernetes Version

1.31.4

Helm Chart configuration

createUserJob:
  applyCustomEnv: false
  useHelmHooks: false
dags:
  gitSync:
    branch: main
    credentialsSecret: airflow-git-credentials
    enabled: true
    repo: https://github.com/dacamposol/kg-infra.git
    subPath: dags
extraEnv: |
  - name: AIRFLOW__API__AUTH_BACKENDS
    value: 'airflow.api.auth.backend.basic_auth'
migrateDatabaseJob:
  applyCustomEnv: false
  jobAnnotations:
    'argocd.argoproj.io/hook': Sync
  useHelmHooks: false
useStandardNaming: true

Docker Image customizations

No response

What happened

Important

This is an instance managed via ArgoCD.

1. I created a connection from type HTTP directly from the Web Server UI.
2. I modified a field in the Argo Application in regards of adding the .extraEnv to allow REST API calls.
3. I accessed again to the Web Server UI.
4. Cannot open the Connections section (all others work fine).

Error in the webserver Pod:

File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/mapper.py", line 3702, in _event_on_load
    instrumenting_mapper._reconstructor(state.obj())
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 213, in on_db_load
    if self.password:
       ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 606, in __get__
    retval = self.descriptor.__get__(instance, owner)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 340, in get_password
    return fernet.decrypt(bytes(self._password, "utf-8")).decode()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/cryptography/fernet.py", line 211, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

When I go to my Argo CD deployment, I can indeed see how for the Airflow application, the airflow-fernet-key got recreated at the same time than the sync happened.

I am assuming the Secret got recreated and somehow the rotation process failed, so the Connections are now encrypted with the old key, and thus cannot be accessed anymore.

What you think should happen instead

No response

How to reproduce

1. Have an Airflow Cluster managed by Argo CD.
2. Create a connection in that cluster.
3. Modify settings in the Application, and wait for Sync.
4. Attempt to access again to the Connections in the aforementioned cluster.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[nevcohen]

As you said, the secret that contain the fernet key is initialized at every sync, try to add this in your argocd app.

[stephsi]

Your recommendation (@nevcohen ) regarding selective sync does not work.
The fernet key is created by a pre-install helm hook. ArgoCD executes this hook on every sync (see below)

Some context for the problem in general:

When upgrading airflow using helm, one uses helm upgrade and thus the pre-install hook is not executed and the fernet key is not replaced.
This is different to how ArgoCD works (doc link)

Argo CD cannot know if it is running a first-time "install" or an "upgrade" - every operation is a "sync'. This means that, by default, apps that have pre-install and pre-upgrade will have those hooks run at the same time.

This means ArgoCD triggers the pre-install hook everytime as you described.

[nevcohen]

If so, the solution would be to take the fernet key created the first time and put it directly in the values ​​file.

That way ArgoCD will use the same one every time and not create a new one.