[New Data Source]: `vault_kv_secret_metadata`

[octo]

Description

I'm requesting a new data source called vault_kv_secret_metadata that provides only the metadata information of a KV v2 secret without including any sensitive data. This data source would function similarly to the existing vault_kv_secret_v2 but would intentionally omit the data and data_json attributes.

The primary need for this feature is to obtain non-ephemeral version information from Vault secrets that can be used as write-only version arguments in downstream resources. Currently, we face a significant challenge: when using the ephemeral vault_kv_secret_v2 data source, we get the version number as an ephemeral value, but Terraform prevents using ephemeral values in write-only version arguments that need to be persisted to state.

Our current workaround requires us to use the regular vault_kv_secret_v2 data source to get non-ephemeral version numbers, but this defeats the entire purpose of ephemeral resources since it puts the actual secret content into the Terraform state. This creates a security risk we're trying to avoid.

A dedicated metadata-only data source would allow us to:

1. Keep sensitive secret data out of state using ephemeral resources
2. Still have access to non-ephemeral version numbers for proper change tracking
3. Use these version numbers as write-only version arguments in downstream resources

Potential Terraform Configuration

# Based on the example provided in
# https://registry.terraform.io/providers/hashicorp/vault/latest/docs/guides/using_ephemeral_resources

provider "vault" {
}

# Securely obtain an already provisioned secret from Vault as ephemeral
ephemeral "vault_kv_secret_v2" "db_secret" {
  mount = "my-kvv2"
  name = "pgx-root"
}

# NEW: Securely obtain metadata (including version) without the secret content
data "vault_kv_secret_metadata" "db_secret" {
  mount = "my-kvv2"
  name = "pgx-root"
}

# Enable database secrets engine
resource "vault_mount" "db" {
  path = "postgres"
  type = "database"
}

# CHANGED: Configure a secure Postgres connection using ephemeral resource and write-only attributes
resource "vault_database_secret_backend_connection" "postgres" {
  backend       = vault_mount.db.path
  name          = "postrgres-db"
  allowed_roles = ["pgx-role"]

  postgresql {
    connection_url = "postgresql://{{username}}:{{password}}@localhost:5432/postgres"
    username = "postgres"
    password_wo = tostring(ephemeral.vault_kv_secret_v2.db_secret.data.password)
    
    # Use non-ephemeral version from metadata data source for tracking changes
    password_wo_version = data.vault_kv_secret_metadata.db_secret.version # <-- CHANGE
  }
}

# Create a role to generate Postgres DB credentials
resource "vault_database_secret_backend_role" "role" {
  backend             = vault_mount.db.path
  name                = "pgx-role"
  db_name             = vault_database_secret_backend_connection.postgres.name
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
    "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"
  ]
}

# Securely obtain database credentials using ephemeral resource
ephemeral "vault_db_secret" "db_user_credentials" {
  mount = vault_mount.db.path
  name = vault_database_secret_backend_role.role.name
  mount_id = vault_mount.db.id
}

References

• Update write-only arguments with versions in the Terraform documentation
• Ephemeral Resources in the Vault provider

Would you like to implement a fix?

Yes

[mmcguinn]

I recently started trying to implement ephemeral values (specifically for transfering secrets between vault and aws resources) to clear up the long time issue of these secrets persisting into state and ran headfirst into this problem. Honestly I am really surprised, this seems like it is mandatory to the most obvious user-central use case: take a secret from vault and provide it to a write-only resource argument. Sure as-is the problem of secret in state is fixed, but it introduces an entirely new problem of needing to manually manage incrementing write-only versions across every secret!

I hope this can be adopted soon, I see it appears that the linked PR has acquired some conflicts since it was made.

[octo]

the linked PR has acquired some conflicts since it was made.

The conflict was due to the ChangeLog entry, which I have now removed and rebased the change onto the current main.