-
Notifications
You must be signed in to change notification settings - Fork 506
Description
Firstly, i think this vcluster project is very cool, have great value, and at least i am learning a lot about both k8s and vcluster while exploring this project.
High level: Why use Secrets to pass configs (values.yaml) to pods, and not say, configmap?
In more details: I see that the within the helm chart (as of 0.24.0) configurations, the content of values.yaml, are passed from helm, to pods, via a Secret.
helm template team-x-vcluster loft-sh/vcluster --version v0.24.0
.
.
.
---
# Source: vcluster/templates/config-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: "vc-config-team-x-vcluster"
namespace: default
labels:
app: vcluster
chart: "vcluster-0.24.0"
release: "team-x-vcluster"
heritage: "Helm"
type: Opaque
data:
config.yaml: "Y29udHJ...." # << Base64 of values.yaml
---
.
.
.
---
# Source: vcluster/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: team-x-vcluster
namespace: default
labels:
app: vcluster
chart: "vcluster-0.24.0"
release: team-x-vcluster
heritage: "Helm"
spec:
selector:
matchLabels:
app: vcluster
release: "team-x-vcluster"
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
serviceName: team-x-vcluster-headless
podManagementPolicy: Parallel
.
.
.
template:
metadata:
annotations:
vClusterConfigHash: "b1768483e2256a4f33a31821c0a9122b283e532dd7decbd7c361caf4540066ec"
labels:
app: vcluster
release: team-x-vcluster
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: vc-team-x-vcluster
volumes:
- name: helm-cache
emptyDir: {}
- emptyDir: {}
name: binaries
- name: tmp
emptyDir: {}
- name: certs
emptyDir: {}
- name: vcluster-config
secret:
secretName: vc-config-team-x-vcluster # <<<< Mounted here
- name: coredns
configMap:
name: vc-coredns-team-x-vcluster
.
.
.
When installing the vcluster helm chart directly on say, my local k8s cluster, all is well. However, we use ArgoCD as our deployment platform, and we disallow Secret syncing. https://github.com/argoproj/argo-cd/blob/a1f90b5cb6d3b5a2d642a0cfe0b2da2f6bd89342/docs/operator-manual/argocd-cm.yaml#L207
k get cm -n argocd argocd-cm -o jsonpath='{.data.resource\.exclusions}'
- apiGroups:
- cilium.io
kinds:
- CiliumIdentity
clusters:
- apiGroups:
- ""
kinds:
- Secret
clusters:
Why? because apparently, its best practice, safer, not available from the ArgoCD API Server / redis cache, etc. xref https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/. (FWIW, if i remove this exclusions VCluster installs fine via ArgoCD)
When we try to install vcluster with secrets blocked by ArgoCD, then the pods fails to start, coz it can't find the secret.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 78s default-scheduler Successfully assigned ....
Warning FailedMount 14s (x8 over 77s) kubelet MountVolume.SetUp failed for volume "vcluster-config" : secret "vc-config-myvcluster-da1d240407" not found
The question is, why is vcluster using secrets to pass configs, and not say, configmap or something else?
FWIW, i tried to tweak the helm chart in such a way where we create a ConfigMap with the same content as that of the secret, and then tweak the statefileset to read / mount the Config map, and not secret. While all the changes worked, but the vcluster software failed to start, as it seems the software (container) is "hardcoded" (?) to look for secrets
>> k logs -n team-x vcluster-0
syncer 2025-03-25 19:13:53 ERROR cmd/root.go:50 error {"component": "vcluster", "error": "using secret annotations: get secret: secrets \"vc-config-vcluster\" not found"} `
Questions: why is vcluster using secrets to pass configs, and not say, configmap or something? AFAICT, there is no senativity data in the secret content; its just values.yaml. Is there some inter detail / reason im missing? (e.g. the vcluster software needs to check secrets? why?) Can we change it so that we use a config map and not a secret (or a toggle on which to use for backward compatibility reasons)
Thanks