Unable to run gcloud commands with non-root container user

[hinewin]

I'm experiencing an issue while attempting to execute gcloud commands with a non root container user. The issue appears to stem from restricted access to /var/run/secrets/sts.googleapis.com/serviceaccount/..data/token. This particular path needs to be accessible for gcloud commands to function properly. The problem persists even when I include the annotation cloud.google.com/gcloud-run-as-user in the service account.

[ordovicia]

Cloud you please share the details of your gcp-workload-identity-federation-webhook deployment (version, args, etc.) and ServiceAccounts and pods with which you are experiencing the issue, so that we can try to reproduce it?

[hinewin]

Webhook's version: 0.4.1
I've verified the workload identity pool and provider are working properly and was able to access resources with a container that runs as root.
Using the same working pool, provider and GSA, I've created a new KSA to support running a pod with a non-root container user and verified the correct ID for the non-root user.

Below is the KSA & Pod manifest file

# GSA has access to bucket `foo-gcs` and can read the secret file
# gcloud storage cat gs://foo-gcs/secret
# KSA Manifest
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-foo
  namespace: default
  labels:
    app: wif-bar
  annotations:
    cloud.google.com/workload-identity-provider: "correct-identity-provider"
    cloud.google.com/service-account-email: "sa-foo@foo-bar-wif.iam.gserviceaccount.com"
    cloud.google.com/audience: "correct-audience"
    cloud.google.com/injection-mode: "gcloud"
    cloud.google.com/gcloud-run-as-user: "1001"


# Pod Manifest
---
apiVersion: v1
kind: Pod
metadata:
  name: sa-foo
  namespace: default
  labels:
    app: wif-bar
spec:
  serviceAccountName: sa-foo
  containers:
  - name: gcloud-kubectl
    image: foobar-image-with-non-root-user
    command: ["/bin/sh"]
    args: ["-c", "gcloud storage cat gs://foo-gcs/secret && while true; do echo 'keep alive'; sleep 10; done"]

logs from the following container return:

ERROR: gcloud crashed (PermissionError):   [Errno 13]  Permission denied: '/var/run/secrets/sts.googleapis.com/serviceacount/token'

[hinewin]

Hi, I would like to check in on this issue. I am currently still unable to run non-root containers using this webhook.

[ordovicia]

Sorry for late response.

logs from the following container return:

ERROR: gcloud crashed (PermissionError):   [Errno 13]  Permission denied: '/var/run/secrets/sts.googleapis.com/serviceacount/token'

What is the container's UID?
I guess setting cloud.google.com/gcloud-run-as-user annotation value to the UID might work.

[hinewin]

Thanks for your respond @ordovicia , I've already included the correct UID for the container in the above manifest cloud.google.com/gcloud-run-as-user: "1001" but the issue persists ):
Are you able to reproduce?

[hinewin]

Hello, just a friendly nudge on this topic. Are you able to find any further details regarding this problem?

[hinewin]

Just checking in to see if theres any update on this?

[sosimon]

For posterity, @hinewin and I worked around this by setting the token default mode: token-default-mode=0444 (default was 0440).

The gloud-run-as-user option never seemed to do anything for us.