Vault-injector race condition

[dberuben]

Vault agent injector race condition with environment variables

Describe the bug

There appears to be a race condition with vault-agent-init where the application starts before the environment variables from the injected secrets are properly loaded. While the secrets are successfully written to /vault/secrets/config, they are not present in the application's environment when the process starts, causing authentication failures with dependent services.

Specifically, when checking /proc/1/environ for the main application process, the injected secrets (e.g., REDIS_PASSWORD) are not present in the environment, even though they exist in the secrets file. This causes the application to fail with Redis authentication errors.

The race condition appears to occur in the following sequence:

1. vault-agent-init writes the initial secrets file
2. Two processes then start in parallel:
   • The application container executes: source /vault/secrets/config && ./app start
   • vault-agent continues updating /vault/secrets/config with fresh secrets
3. Due to this parallel execution, sometimes when the source command runs, it encounters an empty or partially written secrets file, resulting in environment variables not being properly loaded

Environment

• Kubernetes version: v1.29.10-eks-7f9249a
• Distribution: Amazon EKS
• vault-k8s version: helm chart vault-0.29.0
• Vault-server: 1.17.2

To Reproduce

Steps to reproduce the behavior:

1. Deploy application with vault-agent injection annotations
2. Application pod starts and vault-agent injects secrets into /vault/secrets/config
3. Application process starts and attempts to connect to Redis
4. Connection fails with "NOAUTH Authentication required" because environment variables are not loaded
5. Verify by checking /proc/1/environ - injected secrets are missing from environment

Application deployment annotations:

podAnnotations:
  vault.hashicorp.com/agent-cache-enable: "true"
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "service-sa"
  vault.hashicorp.com/agent-inject-secret-config: path/${lsp.env}/service
  vault.hashicorp.com/agent-inject-template-config: |
    {{ with secret "path/${lsp.env}/service" -}}
     {{ range $k, $v := .Data.data }}
        export {{ $k | replaceAll "." "_" | toUpper }}='{{ $v }}'
      {{ end }}
    {{- end }}
command: ["sh", "-c", "source /vault/secrets/config && node index.js"]

Expected behavior

The secrets should be properly loaded into the application's environment before the application starts. The source /vault/secrets/config command should execute after the secrets file is fully written and before the application process begins.

Additional context

Verification steps performed:

1. Checking /vault/secrets/config shows the secrets are properly written:

export REDIS_PASSWORD='Password'

2. Checking process environment shows secrets are missing:

$ cat /proc/1/environ | grep REDIS_PASSWORD
(no output)

3. Error logs from application:

{
  "level": "error",
  "time": 1733389377936,
  "err": {
    "type": "ReplyError",
    "message": "NOAUTH Authentication required.",
    "stack": "ReplyError: NOAUTH Authentication required.\n    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:302:14)",
    "command": {
      "name": "info",
      "args": []
    }
  },
  "msg": "Redis error."
}

This suggests the application starts before the environment variables are properly loaded, despite using the source command in the container's command string.

Solution
We were able to mitigate the issue by adding this annotation

vault.hashicorp.com/agent-pre-populate-only: "true"