Support for Google Cloud Universe Domain in GCP KMS Auto-Unseal (gcpckms)

[benji78]

When attempting to configure Vault HA on GKE with GCP KMS for auto-unseal using gcpckms, the current implementation implicitly defaults to googleapis.com as the "Universe Domain". This poses a problem for users who operate within a different Google Cloud Universe Domain (e.g., specific sovereign cloud environments or custom domain configurations).

The error observed in the Vault logs clearly indicates an authentication failure when Vault tries to interact with the KMS service to verify the existence of the keys:

2025-06-23T11:33:24.919Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""  
2025-06-23T11:33:24.919Z [WARN]  storage.raft.fsm: raft FSM db file has wider permissions than needed: needed=-rw------- existing=-rw-rw----  
error parsing Seal configuration: error checking key existence: rpc error: code = Unauthenticated desc = Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.  
error details: name = ErrorInfo reason = ACCESS_TOKEN_TYPE_UNSUPPORTED domain =  metadata = map[method:google.cloud.kms.v1.KeyManagementService.GetCryptoKey service:cloudkms.googleapis.com]

This error, specifically Request had invalid authentication credentials with the cloudkms.googleapis.com service in the metadata, strongly suggests that the KMS client within Vault is attempting to reach the default googleapis.com endpoint, even when the underlying environment might require a different universe domain for authentication and service discovery.

Upon investigation, it appears the underlying Google Cloud Go client library [1] supports setting the universe domain via the GOOGLE_CLOUD_UNIVERSE_DOMAIN environment variable. However, the Vault gcpckms wrapper, which uses s.client.GetCryptoKey [2] to interact with KMS, does not currently expose a mechanism to explicitly configure this universe domain.

Expected Behavior:

Vault's GCP KMS auto-unseal functionality should allow users to specify the Google Cloud Universe Domain. Ideally, this could be achieved by honoring the GOOGLE_CLOUD_UNIVERSE_DOMAIN environment variable, similar to how the underlying Google Cloud Go client library functions. This would provide flexibility for deployments in diverse Google Cloud environments.

Steps to Reproduce (Conceptual):

1. Attempt to deploy Vault HA on GKE in a Google Cloud Universe Domain other than googleapis.com.
2. Configure the Vault seal stanza to use gcpckms for auto-unseal.
3. Observe the authentication errors in the Vault logs, similar to those provided above, indicating an inability to reach the KMS service.

Proposed Solution:

Implement support within the Vault GCP KMS seal (gcpckms) for specifying the Google Cloud Universe Domain. The most straightforward approach would be to read and utilize the GOOGLE_CLOUD_UNIVERSE_DOMAIN environment variable when initializing the GCP KMS client. This would align with the behavior of the underlying Google Cloud Go client library and provide a simple, effective configuration mechanism for users.

References:

• [1] google-cloud-go/auth/internal/internal.go#L43:
  defines UniverseDomainEnvVar = "GOOGLE_CLOUD_UNIVERSE_DOMAIN"
• [2] hashicorp/go-kms-wrapping/blob/main/wrappers/gcpckms/gcpckms.go:
  s.client.GetCryptoKey call within gcpckms.go

This enhancement would significantly improve the flexibility and compatibility of Vault's GCP KMS auto-unseal feature for users operating in non-default Google Cloud Universe Domains.

As a side note, I use the helm chart and I have tested my configuration on GCP's default universe domain where everything works as expected. However, on a different one the container crashes right after starting and just boot loops.