Issues with RabbitMQ Cluster Operator and Secrets Store CSI Driver Integration

[dannotes]

I am facing challenges integrating RabbitMQ with Azure Key Vault using the Secrets Store CSI Driver in my Kubernetes environment.

Environment

• Kubernetes Version: 1.30.5
• RabbitMQ Version: 3.12
• Secrets Store CSI Driver Version: v2.11.1

Reproduction steps

1. Created a SecretProviderClass to fetch TLS certificates and credentials from Azure Key Vault.
2. Attempted to deploy RabbitMQ Cluster Operator.

SecretServiceProvider.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: rabbitmq-keyvault-secrets
  namespace: rabbitmq-system
spec:
  provider: azure
  secretObjects:
  - data:
    - key: tls.crt
      objectName: rabbitmq-server-cert
    - key: tls.key
      objectName: rabbitmq-server-key
    secretName: rabbitmq-server-certs
    type: kubernetes.io/tls
  - data:
    - key: admin-password
      objectName: rabbitmq-admin-password
    - key: erlang-cookie
      objectName: rabbitmq-erlang-cookie
    secretName: rabbitmq-credentials
    type: Opaque
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "f2120383-a285-489e-aa69-45a1283b02c9"
    keyvaultName: "aks-vault-jkadbhabadjad"
    cloudName: AzurePublicCloud
    objects: |
      array:
        - |
          objectName: rabbitmq-server-cert
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-server-key
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-admin-password
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-erlang-cookie
          objectType: secret
          objectVersion: ""
    tenantId: "f19a22fe-f19f-4a89-8879-face6c476625"

rabbitmq-cluster-definitions.yaml

# rabbitmq-definitions.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-definitions
  namespace: rabbitmq-system
data:
  definitions.json: |-
    {
      "users": [
        {
          "name": "monitoring",
          "password_hash": "pssEXpJo+paqFO4HMFdSRy0gNGe9vzJZB+e/n7OyF6G34ntN",
          "tags": "monitoring"
        }
      ],
      "vhosts": [
        {
          "name": "/"
        }
      ],
      "permissions": [
        {
          "user": "monitoring",
          "vhost": "/",
          "configure": "^$",
          "write": "^$",
          "read": ".*"
        }
      ],
      "policies": [
        {
          "name": "ha-policy",
          "pattern": ".*",
          "vhost": "/",
          "priority": 0,
          "definition": {
            "ha-mode": "all",
            "ha-sync-mode": "automatic",
            "queue-master-locator": "min-masters"
          }
        }
      ]
    }

rabbitmq-cluster.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: production-rabbitmq
  namespace: rabbitmq-system
  labels:
    app.kubernetes.io/component: rabbitmq
spec:
  replicas: 3
  image: rabbitmq:3.12-management
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"
  persistence:
    storageClassName: rabbitmq-premium-ssd
    storage: "50Gi"
  tls:
    secretName: rabbitmq-server-certs  # TLS secret with cert and key
    caSecretName: rabbitmq-ca-cert-secret  # Separate secret for CA certificate
  rabbitmq:
    additionalPlugins:
      - rabbitmq_prometheus
      - rabbitmq_shovel
      - rabbitmq_shovel_management
      - rabbitmq_federation
      - rabbitmq_federation_management
    additionalConfig: |
      load_definitions = /etc/rabbitmq/custom/definitions.json
      cluster_partition_handling = autoheal
      vm_memory_high_watermark.relative = 0.8
      disk_free_limit.absolute = 5GB
      queue_master_locator = min-masters
      prometheus.tcp.port = 15692
      collect_statistics_interval = 10000
      ssl_options.verify = verify_peer
      ssl_options.fail_if_no_peer_cert = true
      listeners.ssl.default = 5671
      ssl_options.cacertfile = /etc/rabbitmq-certs/ca.crt  # Path to CA cert
      ssl_options.certfile = /etc/rabbitmq-certs/tls.crt    # Path to server cert
      ssl_options.keyfile = /etc/rabbitmq-certs/tls.key      # Path to server key
  override:
    statefulSet:
      spec:
        template:
          spec:
            volumes:
              - name: definitions
                configMap:
                  name: rabbitmq-definitions
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: rabbitmq-keyvault-secrets
            containers:
              - name: rabbitmq
                env:
                  - name: RABBITMQ_ERLANG_COOKIE
                    valueFrom:
                      secretKeyRef:
                        name: rabbitmq-credentials  # Kubernetes secret created by SecretProviderClass
                        key: erlang-cookie
                  - name: RABBITMQ_DEFAULT_PASS
                    valueFrom:
                      secretKeyRef:
                        name: rabbitmq-credentials  # Kubernetes secret created by SecretProviderClass
                        key: admin-password
                  - name: RABBITMQ_DEFAULT_USER
                    value: "admin"
                volumeMounts:
                  - name: definitions
                    mountPath: /etc/rabbitmq/custom/
                  - name: secrets-store-inline
                    mountPath: "/etc/rabbitmq-certs"  # Mount path for certificates
                    readOnly: true
  service:
    type: ClusterIP
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "15692"

Expected behavior

The RabbitMQ cluster should start successfully with secrets populated from Azure Key Vault.

Additional context

The RabbitMQ deployment fails due to missing secrets, even though they are configured in Azure Key Vault and referenced in the SecretProviderClass.

Additional Information

• I have ensured that the managed identity has access to read secrets from Azure Key Vault by creating a test pod.
• I manually created the secret with the annotation secrets-store.csi.k8s.io/used: "true", but the secrets remain empty in Kubernetes even after deploying using cluster operator that references the SecretProviderClass.

Please advise on how to resolve this integration issue or if there are any known limitations with this setup.