How to use blobfuse2 with GitHub OIDC via azure/login?

[berney]

I am struggling with blobfuse2 mount to work in a GitHub Workflow that has authenticated to Azure with the azure/login action.
The azure/login action is working with GitHub OIDC, and I'm able to run az commands, including az account get-access-token, upload blobs etc., but I cannot get blobfuse2 to work with mode: azcli.

I am using a GitHub's runner, I am NOT using self-hosted runners.

Is this meant to work?

I thought since I have a working az, the login has worked, that blobfuse2 should be able to leverage that login and also work, but it seems it doesn't work.

Does anyone have a working workflow?

GitHub Workflow

jobs:
  azure:
    name: Azure
    runs-on: ubuntu-latest
    steps:
      - name: Azure login
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: az direct
        run: |
            az account show
            az group list
            az storage account list
            az storage container list --account-name ${{secrets.AZURE_STORAGE_ACCOUNT}}
            az storage container-rm list --storage-account ${{secrets.AZURE_STORAGE_ACCOUNT}}
            az storage blob list --account-name ${{secrets.AZURE_STORAGE_ACCOUNT}} --container-name ${{secrets.AZURE_STORAGE_COUNTAINER}} --auth-mode login --output table

      - name: Blob FUSE
        continue-on-error: true
        run: |
          sudo mkdir -p /mnt/blobfuse2
          sudo mkdir -p /mnt/blobfuse2tmp
          sudo blobfuse2 mount /mnt/blobfuse2 --config-file=config.yaml --tmp-path=/mnt/blobfuse2tmp

Blobfuse2 config.yaml

This is my config.yaml for blobfuse2

logging:
  level: log_debug
  file-path: "blobfuse2-logs.txt"
  type: base
azstorage:
  account-name: ${{ secrets.AZURE_STORAGE_ACCOUNT }}
  container: ${{ secrets.AZURE_STORAGE_CONTAINER }}
  mode: azcli

Blobfuse2 Logs

on Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (163)]: BlockBlob::SetupPipeline : Setting up
Mon Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (144)]: BlockBlob::createServiceClient : Getting service client
Mon Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_DEBUG [azauth.go (82)]: azAuth::getAzAuth : Account: berneygithub, AccountType: BLOCK, Protocol: https, Endpoint: https://REDACTED.blob.core.windows.net/
Mon Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (226)]: BlockBlob::SetPrefixPath : path 
Mon Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (180)]: BlockBlob::TestPipeline : Validating
Mon Jul  8 10:08:11 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : =====> Try=1
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : error AzureCLICredential: ERROR: Please run 'az login' to setup account.

Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : non-retriable error *errorinfo.nonRetriableError
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x889cc0)
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [AzureCLICredential: ERROR: Please run 'az login' to setup account.
]
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Mon Jul  8 10:08:12 UTC 2024 : blobfuse2[3036] : [/mnt/blobfuse2] LOG_ERR [mount.go (413)]: mount : failed to initialize new pipeline [failed to authenticate credentials for azstorage]

Notes

After the azure/login step the workflow is able to run commands like az storage account list and az storage container list successfully. I've tested and it can upload blobs etc.

Similar Discussions

I found #1396, but I couldn't work out how to do it now that azcli mode is available.

Workaround

I was able to get it to work by modifying the configuration to this:

allow-other: true
logging:
  level: log_debug
  file-path: "blobfuse2-logs.txt"
  type: base
azstorage:
  account-name: ${{ secrets.AZURE_STORAGE_ACCOUNT }}
  container: ${{ secrets.AZURE_STORAGE_CONTAINER }}
  #mode: azcli
  mode: spn
  clientid: ${{ secrets.AZURE_CLIENT_ID }}
  tenantid: ${{ secrets.AZURE_TENANT_ID }}
  oauth-token-path: /token.txt

Where I have retrieved the GitHub OIDC token and saved it to a file. Note, I had to use the full file path.
Originally, I had just used token.txt (not full path), and it mounted successfully, but I couldn't list files in the mount.
Looking at the logs it couldn't find token.txt - it seems due to sudo and the CWD changing was causing problems after the initial mount.

I don't like that I have to retrieve and store the OIDC token.
This feels overly complicated and "wrong", especially when there was an existing, working, az login.
Does anyone have a better working solution?

[souravgupta-msft]

Hi @berney. Thanks for reaching out. For using azcli auth mode, you will have to first login using the az login command and then run the mount command. The reason for mount failure can be seen in the log,
error AzureCLICredential: ERROR: Please run 'az login' to setup account.

azcli mode internally calls the below command for fetching the token.
az account get-access-token

[berney]

I am already logged in with az login.
The azure/login action has done az login --service-principal --username <servicePrincipalId> --tenant <tenantId> --federated-token <oidcToken>.

This is why the 2nd step can do az storage account list and az storage container list and see the account and container I'm trying to mount with blobfuse2. And it is also possible to successfully call az account get-access-token. Despite everything else working, it is not possible to call blobfuse2 mount.

[souravgupta-msft]

Can you try these steps in a VM and see if you are able to replicate the error?

[berney]

berney@vm:~$ az account show | jq
{
  "environmentName": "AzureCloud",
  "homeTenantId": "e5dfREDACTED",
  "id": "bd0cREDACTED",
  "isDefault": true,
  "managedByTenants": [],
  "name": "Free Trial",
  "state": "Enabled",
  "tenantId": "e5dREDACTED",
  "user": {
    "name": "7058REDACTED",
    "type": "servicePrincipal"
  }
}
berney@vm:~$ az storage account list --query '[].name'
[
  "MY_STORAGE_ACCOUNT"
]
berney@vm:~$ az storage container list --account-name MY_STORAGE_ACCOUNT--auth-mode login --query '[].name'
[
  "packages"
]
berney@vm:~$ az account get-access-token > access-token.json
berney@vm:~$ jq 'del(.accessToken)' access-token.json
{
  "expiresOn": "2024-07-11 09:36:25.000000",
  "expires_on": 1720690585,
  "subscription": "bd0REDACTED",
  "tenant": "e5dfREDACTED",
  "tokenType": "Bearer"
}
berney@vm:~$ cat config.yaml
logging:
  level: log_debug
  file-path: "blobfuse2-logs.txt"
  type: base
azstorage:
  account-name: REDACTED
  container: packages
  mode: azcli
berney@vm:~$ sudo mkdir /tmp/blobfuse2
berney@vm:~$ sudo mkdir /tmp/blobfuse2tmp
berney@vm:~$ sudo blobfuse2 mount /mnt/blobfuse2 --config-file=config.yaml --tmp-path=/mnt/blobfuse2tmp
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]
berney@vm:~$ tail -n 15 blobfuse2-logs.txt
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (163)]: BlockBlob::SetupPipeline : Setting up
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (144)]: BlockBlob::createServiceClient : Getting service client
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_DEBUG [azauth.go (82)]: azAuth::getAzAuth : Account: REDACTED, AccountType: BLOCK, Protocol: https, Endpoint: https://REDACTED.blob.core.windows.net/
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (226)]: BlockBlob::SetPrefixPath : path
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_TRACE [block_blob.go (180)]: BlockBlob::TestPipeline : Validating
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : =====> Try=1
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : error AzureCLICredential: ERROR: Please run 'az login' to setup account.

Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_DEBUG [utils.go (151)]: SDK(Retry) : non-retriable error *errorinfo.nonRetriableError
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_ERR [block_blob.go (199)]: BlockBlob::TestPipeline : Failed to validate account with given auth %!s(func() string=0x889cc0)
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_ERR [azstorage.go (161)]: AzStorage::configureAndTest : Failed to validate credentials [AzureCLICredential: ERROR: Please run 'az login' to setup account.
]
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_ERR [azstorage.go (101)]: AzStorage::Configure : Failed to validate storage account [failed to authenticate credentials for azstorage]
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_ERR [pipeline.go (69)]: Pipeline: error creating pipeline component azstorage [failed to authenticate credentials for azstorage]
Wed Jul 10 09:53:37 UTC 2024 : blobfuse2[3835] : [/mnt/blobfuse2] LOG_ERR [mount.go (413)]: mount : failed to initialize new pipeline [failed to authenticate credentials for azstorage]

berney@vm:~$ az storage blob upload --account-name MY_STORAGE_ACCOUNT --container-name packages --name blobfuse2-logs.2.txt --file blobfuse2-logs.txt --auth-mode login
Finished[#############################################################]  100.0000%
{
  "client_request_id": "1e845924-3ea3-11ef-a31e-03ad36494d37",
  "content_md5": "iWZzzuNZHIfIe/TFkShjgw==",
  "date": "2024-07-10T09:59:39+00:00",
  "encryption_key_sha256": null,
  "encryption_scope": null,
  "etag": "\"0x8DCA0C70371CD5D\"",
  "lastModified": "2024-07-10T09:59:39+00:00",
  "request_id": "e11c2a09-101e-0039-3caf-d2ef70000000",
  "request_server_encrypted": true,
  "version": "2022-11-02",
  "version_id": null
}

As you can see I am logged in, I can list storage accounts, list storage containers, I can get an access token, and I can upload blobs, but when I try to use blobfuse2 mount it says I need to login, when I already am logged in.

Do you have a working GitHub Workflow where you use blobfuse2 mount where azure login has used OIDC?

[souravgupta-msft]

@berney, thanks for the detail explanation. The az login error is because you are trying to mount blobfuse2 as sudo user whereas you have logged in using az cli as regular user.
sudo blobfuse2 mount /mnt/blobfuse2 --config-file=config.yaml --tmp-path=/mnt/blobfuse2tmp

You can replicate this if you run any of the above az cli commands as sudo user. For example,
sudo az account show | jq

Please try mounting as regular user and see if are getting the same error.

[berney]

Thanks @souravgupta-msft I'm able to mount using mode: azcli without using sudo.