airflow.cfg can't pull secrets from LocalFilesystemBackend

[thedustinmiller]

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When starting an Airflow instance or checking a property via airflow config list/get-value properties that have the _secret suffix return their literal value and the main property isn't set at all. The default values instead are applied.

What you think should happen instead?

According to the docs, the following snippet should retrieve the key sql_alchemy_conn from the secret backend (the local filesystem one, in my case) and assign it to the sql_alchemy_conn under the [database] section.

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html

How to reproduce

Reproducing requires only configuring the following properties in airflow.cfg and the .env file in the same directory. The secret backend still works from within a DAG, however, it just seems to not be invoked while parsing the airflow.cfg. I tried bringing the [secrets] section to the top of the airflow.cfg, since by default it comes after [database], but that had no effect.

[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "./.env"}

[database]
sql_alchemy_conn_secret = sql_alchemy_conn

and an .env file (or yaml or json) like this

sql_alchemy_conn=postgresql://airflow_user@localhost/airflow_db

Operating System

Arch linux 6.10.4-arch2-1

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.7.2
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1

Deployment

Virtualenv installation

Deployment details

venv setup from python 3.12.4
pip install "apache-airflow[celery]==2.9.3" --constraint "./constraints-3.8.txt"

Anything else?

This is from a completely fresh installation. The issue also appears on an ARM Mac. I don't have the other possible secret backends set up, so I don't know how those behave. I can also understand that this might be a documentation issue and this isn't supported anymore, since the secrets backend would be retrieving secrets for the same file it is configured in itself.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[kurtrwall]

This is just not implemented for the LocalFilesystemBackend unfortunately. I'm not sure if you could consider it a bug, per se, but it's certainly implied it should work in the documentation (IIRC). Create a pip installable secrets backend yourself, here's the code I use and it works well, though you should know it will leak secrets to anyone with access to the airflow config CLI.

class YourLocalFilesystemBackend(LocalFilesystemBackend):
    """Airflow filesystem secrets backend that can pull config values from secrets"""

    def get_config(self, key: str) -> str | None:
        config = super().get_config(key)
        if config is not None:
            return config
        var = self.get_variable(key)
        if var is not None:
            return var
        conn = self.get_connection(key)
        if conn is not None:
            return conn.get_uri()

[shahar1]

I feel quite uncomfortable with implementing it, knowing that it may become a security issue as stated above.
However, I support adding a paragraph to the docs about it with a very explicit warning.

[potiuk]

No. We should raise "Not Implemented" error in this case. If we implement it, this will give false sense of security - raising NotImplemented is expicitly stating that .... it's not implemented - and it can also explain why