diff --git a/source/operations/cert-manager.rst b/source/operations/cert-manager.rst new file mode 100644 index 00000000..0ca0721d --- /dev/null +++ b/source/operations/cert-manager.rst @@ -0,0 +1,424 @@ +.. _minio-certmanager: + +============ +Cert Manager +============ + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + +TLS certificate management with cert-manager +-------------------------------------------- + +`cert-manager `__ works as a controller for managing certificates within Kubernetes clusters. +cert-manager obtains valid certificates from an ``Issuer`` or ``ClusterIssuer`` and can renew certificates prior to expiration. + +The MinIO Operator supports using cert-manager for certificates as an alternative to the MinIO Operator managing certificates for itself and its tenants. + +On a Kubernetes cluster, cert-manager requires a global level ``ClusterIssuer`` to generate intermediary ``Issuers`` and certificates. +At the namespace level, cert-manager issues certificates derived from an ``Issuer``. +A ``ClusterIssuer`` can issue certificates for multiple namespaces. +An ``Issuer`` only issues certificates for its own namespace. + +To learn more about cert-manager Issuers, refer to their `Issuer documentation `__. + +Below is a logical view of a Kubernetes cluster: + +![Cert-manager-namespaces.png](images%2FCert-manager-namespaces.png) + +This cluster contains 4 namespaces: + +- minio-operator +- tenant-1 +- tenant-2 +- other-namespace + +This guide shows you how to set up different Certificate Authorities (CA) in each namespace, all of them referencing a global `Cluster Issuer`. + +![Cert-manager Issuers.png](images%2FCert-manager%20Issuers.png) + +Cert Manager is installed in the `cert-manager` namespace. + +The global `Cluster Issuer` is created in the default namespace. + +A local `Issuer` is created in each tenant namespace. + +An `Issuer` is also created in the `minio-operator` namespace. More about services that require TLS certificates +in the `minio-operator` namespace are covered below in [MinIO Operator services with cert-manager](#minio-operator-services-with-cert-manager).. + +> [!NOTE] +> This guide uses a self-signed `Cluster Issuer`. You can also use [other Issuers supported by Cert Manager](https://cert-manager.io/docs/configuration/issuers/). +> The main difference is that you must provide the `Issuer` CA certificate to minio, instead of the CA's mentioned in this guide. + +## Getting Started + +### Prerequisites + +- Kubernetes version `+v1.21`. While cert-manager + supports [earlier K8s versions](https://cert-manager.io/docs/installation/supported-releases/), MinIO Operator requires 1.21 or later. +- [kustomize](https://kustomize.io/) installed +- `kubectl` access to your `k8s` cluster + +### Setup Cert-manager + +Install [cert-manager](https://cert-manager.io/docs/releases/release-notes/release-notes-1.12/) 1.12.X LTS is preferred, +or install latest, for example using kubectl. + +```bash +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.9/cert-manager.yaml +``` + +For more details on the Cert Manager refer to https://cert-manager.io/docs/installation/. + +### Create Cluster Self-signed root Issuer + +The `Cluster Issuer` is the cluster level Issuer all other certificates are derived from. Request Cert Manager to +generate this by creating a `ClusterIssuer` resource: + +```yaml +# selfsigned-root-clusterissuer.yaml +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: selfsigned-root +spec: + selfSigned: {} +``` + +```shell +kubectl apply -f selfsigned-root-clusterissuer.yaml +``` + +# MinIO Operator services with cert-manager + +MinIO Operator manages the TLS certificate issuing for the services hosted in the `minio-operator` namespace. That is the Secure Token Service `sts`. + +This section describes how to generate the `sts` TLS certificate with Cert Manager. +These certificates must be issued before installing Operator. +Be sure to follow step [Create Cluster Self-signed root Issuer](#create-cluster-self-signed-root-issuer) mentioned above. + +## Secure Token Service (STS) + +MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. In essence +this service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials +for each application. For more information on the Service see the MinIO docs at https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.html. +There is also an [STS](https://github.com/minio/operator/blob/master/docs/STS.md) guide in the docs and example client code in +https://github.com/minio/operator/tree/master/examples/kustomization/sts-example. + +For the purpose of this guide, STS Service can be considered a webserver presented with a TLS certificate for https traffic. +This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using +Cert Manager instead. + +### Create minio-operator namespace CA Issuer + +To create the Certificate Authority (CA) certificate used to issue certificates for services in the `minio-operator` +namespace, first create the minio-operator namespace + +```shell +kubectl create ns minio-operator +``` + +Request a Certificate with `spec.isCA: true` specified. This is our CA for the `minio-operator` namespace. + +```yaml +# operator-ca-tls-secret.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: minio-operator-ca-certificate + namespace: minio-operator +spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f operator-ca-tls-secret.yaml +``` +A new secret with the name `operator-ca-tls` is created in the `minio-operator` namespace, this is the CA issuing TLS certificates for the services in the `minio-operator` namespace. + +> [!IMPORTANT] +> Make sure to trust this certificate in your applications that need to interact with the `sts` service. + + +Now create the `Issuer`: + +```yaml +# operator-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: minio-operator-ca-issuer + namespace: minio-operator +spec: + ca: + secretName: operator-ca-tls +``` + +```shell +kubectl apply -f operator-ca-issuer.yaml +``` + +### Create TLS certificate for STS service + +Request Cert Manager to issue a new certificate containing following DNS domains: + +```shell +sts +sts.minio-operator.svc. +sts.minio-operator.svc. +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> `cluster domain` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) + +Create a `Certificate` for the domains mentioned above: + +```yaml +# sts-tls-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: sts-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer +``` + +```shell +kubectl apply -f sts-tls-certificate.yaml +``` +This creates a secret called `sts-tls` in the `minio-operator` namespace. + +> [!IMPORTANT] +> The secret name is not optional. Make sure the secret name is `sts-tls` by setting `spec.secretName: sts-tls` as in the example above. + +### Install Operator with Auto TLS disabled for STS + +When installing the Operator deployment, make sure to set `OPERATOR_STS_AUTO_TLS_ENABLED: off` env variable in the `minio-operator` container. This prevents +MinIO Operator from issuing the certificate for STS and instead wait for you to provide the TLS certificate issued by Cert Manager. + +> [!WARNING] +> Missing to provide the secret `sts-tls` containing the TLS certificate or providing an invalid key-pair in the secret will +> prevent the STS service from start. + +A way to make sure that the env variables are properly set is using kustomization to patch the `minio-operator` deployment: + +```yaml +# minio-operator/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- github.com/minio/operator/resources + +patches: +- patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: minio-operator + namespace: minio-operator + spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" +``` + +```shell +kubectl apply -k minio-operator +``` + +# MinIO tenant TLS certificates with cert-manager + +## Create tenant-1 namespace CA Issuer + +To create the Certificate Authority (CA) certificate in the namespace `tenant-1`, first create the namespace + +```shell +kubectl create ns tenant-1 +``` + +Next, request a Certificate with `spec.isCA: true` specified: + +```yaml +# tenant-1-ca-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-1-ca-certificate + namespace: tenant-1 +spec: + isCA: true + commonName: tenant-1-ca + secretName: tenant-1-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f tenant-1-ca-certificate.yaml +``` + + +Then create the `Issuer`: + +```yaml +# tenant-1-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: tenant-1-ca-issuer + namespace: tenant-1 +spec: + ca: + secretName: tenant-1-ca-tls +``` + +```shell +kubectl apply -f tenant-1-ca-issuer.yaml +``` + +## Deploy the tenant + +### Create a certificate for Tenant + +Request Cert Manager issue a new TLS server certificate for MinIO that includes the following DNS domains: + +```shell +minio. +minio..svc +minio..svc. +*.-hl..svc. +*..svc. +*..minio..svc.' +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> * `` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o jsonpath="{.data}"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) +> * `tenant-name` is the name provided to your tenant in the `metadata.name` of the Tenant YAML. For this example it is `myminio`. +> * `namespace` is the namespace where the tenant is created, the `metadata.namespace` notes that in the Tenant YAML. For this example it is `tenant-1`. + +Create a `Certificate` for the domains mentioned above: + +```yaml +# tenant-1-minio-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-certmanager-cert + namespace: tenant-1 +spec: + dnsNames: + - "minio.tenant-1" + - "minio.tenant-1.svc" + - 'minio.tenant-1.svc.cluster.local' + - '*.minio.tenant-1.svc.cluster.local' + - '*.myminio-hl.tenant-1.svc.cluster.local' + - '*.myminio.minio.tenant-1.svc.cluster.local' + secretName: myminio-tls + issuerRef: + name: tenant-1-ca-issuer +``` + +> [!TIP] +> For this example, the Tenant name is `myminio`. We recommend naming the secret in the field `spec.secretName` as `-tls`, following the naming +> convention MinIO Operator uses when creates certificates with Autocert enabled (`spec.requestAutoCert: true`). + +```shell +kubectl apply -f tenant-1-minio-certificate.yaml +``` + +### Configure the tenant to use the certificate created by cert-manager + +In the tenant spec, do the following: + +* Disable Autocert `spec.requestAutoCert: false`. This instructs Operator to not attempt to issue certificates and instead rely on Cert Manager to provide them in a secret. +* Reference the Secret containing the TLS certificate from the previous step in `spec.externalCertSecret`. + +```yaml +apiVersion: minio.min.io/v2 +kind: Tenant +metadata: + name: myminio + namespace: tenant-1 +spec: +... + ## Disable default tls certificates. + requestAutoCert: false + ## Use certificates generated by cert-manager. + externalCertSecret: + - name: myminio-tls + type: cert-manager.io/v1 +... +``` + +## Trust tenant-1 CA in MinIO Operator + +MinIO Operator can trust as many CA certificates as provided. To do this, create a secret with the prefix `operator-ca-tls-` +followed by a unique identifier in the `minio-operator` namespace. + +MinIO Operator mounts and trusts all certificates issued by the provided CA's. This is required because +MinIO Operator performs health checks using the */minio/health/cluster* endpoint. + +If Operator is not correctly configured to trust the MinIO Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: + +```error +Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": +x509: certificate signed by unknown authority +``` +For more details about health checks, see https://min.io/docs/minio/Kubernetes/upstream/operations/monitoring/healthcheck-probe.html#cluster-write-quorum. + +### Create `operator-ca-tls-tenant-1` secret + +Copy the Cert Manager generated CA public key (ca.crt) into the `minio-operator` namespace. This allows Operator to trust +the cert-manager issued CA and the derived certificates. + +Create a `ca.crt` file containing the CA: +```sh +kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt +``` + +Create the secret: +```sh +kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator +``` +> [!TIP] +> In this example we choose a secret name of `operator-ca-tls-tenant-1`. Note the tenant namespace +> `tenant-1` is used as suffix for easy identification of which namespace the CA is coming from. \ No newline at end of file