Azure blob remote logging using the official helm chart

[jaruji]

Hello everyone,
wonder if anyone would be able to point me to the format the WASB connection env variable should have if I want to use a blob connection string to authenticate? In the example at https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html, the example connection string is set like this:

export AIRFLOW_CONN_WASB_DEFAULT='wasb://blob%20username:blob%[email protected]?tenant_id=tenant+id'

I can set this enviroment variable using the helm chart - but what exact format would this string have if I want to use the azure blob connection string as the method of authentication instead of username, password and tennantId?

[potiuk]

No idea about Azure, but the page you linked above contains all the parameters that you can set and Airflow has a helper that allows you to generate URI based on connection parameters + extras:
https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#uri-format

If in doubt - look at the code of the Hook and figure out the parameters it needs.

[jaruji]

Thank you for the link, I will take a look and see whether I can figure it out.

[jaruji]

Ok so I did check out the airflow helper for generating URI's, and got the following one:

AIRFLOW_CONN_WASB_CONNECTION='wasb:///?connection_string=BLOB_CONNECTION_STRING'

Now I need to attach this connection to the airflow helm chart - according to the docs (https://airflow.apache.org/docs/helm-chart/stable/adding-connections-and-variables.html), you need to pass it through the secret parameter in the values.yaml of the chart.

kubectl create secret generic airflow-wasb-secret --from-literal "AIRFLOW_CONN_WASB_CONNECTION='BLOB_CONNECTION_STRING'" -n airflow

  secret:
    - envName: "AIRFLOW_CONN_WASB_CONNECTION"
      secretName: "airflow-wasb-secret"
      secretKey: "AIRFLOW_CONN_WASB_CONNECTION"

But when I pass it, the connection does not appear in the UI nor in the postgres connection table. Any ideas why?

[Unb0rn]

@jaruji @dada-engineer I have the same issue. Did you find a solution? Like how should this "BLOB_CONNECTION_STRING" be formatted? I've tried with the connstring from Access Keys section:
DefaultEndpointsProtocol=https;AccountName=<storageaccount>;AccountKey=<base64key>;EndpointSuffix=core.windows.net
But it doesn't work for me.
If someone has the full envvar example that would be awesome!

[dada-engineer]

sorry @Unb0rn forgot to reply here, We did manage to make it work with the following config (this is a terraform template rendered so this is terraform template language:

AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb/
AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER: ${blob_container_name}
AIRFLOW__CORE__LOGGING_LEVEL: DEBUG
AIRFLOW_CONN_REMOTE_LOGGING: '{"conn_type": "wasb", "login": "${storage_account}", "password": "${account_key}"}'

The odd part is the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER MUST start with wasb meaning in our blob container which is called logs we end up with another wasb/ folder...

Hope this helps.

This has been the only way I got it working for now.

[Unb0rn]

You're a lifesaver) Thank you!

[tcarr-kr]

I've gotten to the point where I can store logs in an Azure Blob (with the same wasb: as a prefix, which sounds like a bug to me), but I haven't been able to read them back via the web UI. I'm doing this on a Mac at the moment so there is always the possibility of an issue there, but when I try to read a task's log a child Python process SEGVs-- I get the Mac crash reporter window which tells me the child in a fork died before exec. So I have yet to close the circle on this. I don't have access to a Linux dev machine unfortunately, and I do worry this is some Mac/Python bad mojo going on. Anyone been able to read their logs in the web ui? Or are running this on a Mac?

[dada-engineer]

I only host this in k8s running the official airflow docker image (okay the official one is the base as we pack our dags in there) without any issue. Can you try to run this via docker / docker-compose on the mac?