Skip to content

Why use Secret to pass configuration (values.yaml content)? #2622

@XDanny322

Description

@XDanny322

Firstly, i think this vcluster project is very cool, have great value, and at least i am learning a lot about both k8s and vcluster while exploring this project.

High level: Why use Secrets to pass configs (values.yaml) to pods, and not say, configmap?

In more details: I see that the within the helm chart (as of 0.24.0) configurations, the content of values.yaml, are passed from helm, to pods, via a Secret.

helm template team-x-vcluster loft-sh/vcluster --version v0.24.0

.
.
.
---
# Source: vcluster/templates/config-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: "vc-config-team-x-vcluster"
  namespace: default
  labels:
    app: vcluster
    chart: "vcluster-0.24.0"
    release: "team-x-vcluster"
    heritage: "Helm"
type: Opaque
data:
  config.yaml: "Y29udHJ...." # << Base64 of values.yaml
---
.
.
.
---
# Source: vcluster/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: team-x-vcluster
  namespace: default
  labels:
    app: vcluster
    chart: "vcluster-0.24.0"
    release: team-x-vcluster
    heritage: "Helm"
spec:
  selector:
    matchLabels:
      app: vcluster
      release: "team-x-vcluster"
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
  serviceName: team-x-vcluster-headless
  podManagementPolicy: Parallel
  .
  .
  .
  template:
    metadata:
      annotations:
        vClusterConfigHash: "b1768483e2256a4f33a31821c0a9122b283e532dd7decbd7c361caf4540066ec"
      labels:
        app: vcluster
        release: team-x-vcluster
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: vc-team-x-vcluster
      volumes:        
        - name: helm-cache
          emptyDir: {}
        - emptyDir: {}
          name: binaries
        - name: tmp
          emptyDir: {}
        - name: certs
          emptyDir: {}
        - name: vcluster-config
          secret:
            secretName: vc-config-team-x-vcluster # <<<< Mounted here
        - name: coredns
          configMap:
            name: vc-coredns-team-x-vcluster
.
.
.

When installing the vcluster helm chart directly on say, my local k8s cluster, all is well. However, we use ArgoCD as our deployment platform, and we disallow Secret syncing. https://github.com/argoproj/argo-cd/blob/a1f90b5cb6d3b5a2d642a0cfe0b2da2f6bd89342/docs/operator-manual/argocd-cm.yaml#L207

k get cm -n argocd argocd-cm -o jsonpath='{.data.resource\.exclusions}'

- apiGroups:
  - cilium.io
  kinds:
  - CiliumIdentity
  clusters:
- apiGroups:
  - ""
  kinds:
  - Secret
  clusters:

Why? because apparently, its best practice, safer, not available from the ArgoCD API Server / redis cache, etc. xref https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/. (FWIW, if i remove this exclusions VCluster installs fine via ArgoCD)

When we try to install vcluster with secrets blocked by ArgoCD, then the pods fails to start, coz it can't find the secret.

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    78s                default-scheduler  Successfully assigned ....
  Warning  FailedMount  14s (x8 over 77s)  kubelet            MountVolume.SetUp failed for volume "vcluster-config" : secret "vc-config-myvcluster-da1d240407" not found

The question is, why is vcluster using secrets to pass configs, and not say, configmap or something else?

FWIW, i tried to tweak the helm chart in such a way where we create a ConfigMap with the same content as that of the secret, and then tweak the statefileset to read / mount the Config map, and not secret. While all the changes worked, but the vcluster software failed to start, as it seems the software (container) is "hardcoded" (?) to look for secrets

>> k logs -n team-x vcluster-0
syncer 2025-03-25 19:13:53    ERROR    cmd/root.go:50    error    {"component": "vcluster", "error": "using secret annotations: get secret: secrets \"vc-config-vcluster\" not found"}  `

Questions: why is vcluster using secrets to pass configs, and not say, configmap or something? AFAICT, there is no senativity data in the secret content; its just values.yaml. Is there some inter detail / reason im missing? (e.g. the vcluster software needs to check secrets? why?) Can we change it so that we use a config map and not a secret (or a toggle on which to use for backward compatibility reasons)

Thanks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions