Skip to content
139 changes: 139 additions & 0 deletions content/docs/installation/continuous-deployment-and-gitops.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,142 @@ Check the cert-manager logs for warnings and errors:
```bash
kubectl logs -n cert-manager -l app.kubernetes.io/instance=cert-manager --prefix --all-containers
```


## Using ArgoCD
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

### Pre-requisites
Ensure the following are in place before proceeding:
- A Kubernetes cluster
- ArgoCD deployed on the Kubernetes cluster: [installation guide](https://argo-cd.readthedocs.io/en/stable/getting_started/)
- Optional: A GitOps repository connected with ArgoCD: [setup guide](https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/)

### Setting up cert-manager
1. Create an [ArgoCD Application](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#applications) manifest file with the provided configuration to set up cert-manager.

```yaml
# application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: cert-manager
server: https://kubernetes.default.svc
project: default
source:
chart: cert-manager
repoURL: https://charts.jetstack.io
targetRevision: [[VAR::cert_manager_latest_version]]
helm:
values: |
installCRDs: true
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Comment on lines +129 to +157
Copy link
Member

@inteon inteon Mar 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: This still has to be verified

Normally by using ServerSideApply and ServerSide diffing, there should be no issues with mutating webhooks:

Suggested change
```yaml
# application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: cert-manager
server: https://kubernetes.default.svc
project: default
source:
chart: cert-manager
repoURL: https://charts.jetstack.io
targetRevision: [[VAR::cert_manager_latest_version]]
helm:
values: |
installCRDs: true
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```yaml
# application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
annotations:
argocd.argoproj.io/compare-options: ServerSideDiff=true
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: cert-manager
server: https://kubernetes.default.svc
project: default
source:
chart: cert-manager
repoURL: https://charts.jetstack.io
targetRevision: [[VAR::cert_manager_latest_version]]
helm:
values: |
installCRDs: true
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true

This change is based on https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/ and https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#server-side-apply.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I tested with this manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager-meta
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: cert-manager
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: cert-manager
    repoURL: https://charts.jetstack.io
    targetRevision: v1.19.0
    helm:
      values: |
        installCRDs: true
  ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-meta-webhook
    jqPathExpressions:
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      respectIgnoreDifferences: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

but I am still seeing a metadata.generation diff (I first tried without the ignoreDifferences and respectIgnoreDifferences but that still results in matchExpressions diff.

```
2. Commit the manifest file and sync the changes in ArgoCD. If a GitOps repository is not set up, use `kubectl apply -f application.yaml` to apply the manifest [installation guide for kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl).
3. ArgoCD will synchronize the `DESIRED MANIFEST` and deploy cert-manager on Kubernetes based on the provided configuration.


### Troubleshooting

#### Scenario 1:
`OutOfSync` cert-manager in the AKS (Azure Kubernetes Service) cluster

##### Issue:
cert-manager in the AKS cluster remains `OutOfSync` due to discrepancies between the `DESIRED MANIFEST` and `LIVE MANIFEST` files.

##### Potential Reasons
Multiple factors could cause the `OutOfSync` issue; refer to [ArgoCD documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#diffing-customization) for potential causes.

##### Example configuration differences
The below configurations are observed to be present in the `LIVE MANIFEST` but not in the `DESIRED MANIFEST` file.

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
...
webhooks:
- admissionReviewVersions:
namespaceSelector:
matchExpressions:
...
...
- key: control-plane
operator: NotIn
values:
- 'true'
- key: kubernetes.azure.com/managedby
operator: NotIn
values:
- aks
```

##### Root Cause Analysis
The discrepancy stems from how AKS manages admission controllers to protect internal services in the `kube-system` namespace. More details can be found in [Frequently Asked Questions about Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces)

##### Suggested Fix
It is also possible to ignore differences from fields owned by specific managers defined in `metadata.managedFields` in live resources. More details can be found in [(ArgoCD) Diffing Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration)

To resolve this issue, modify the cert-manager manifest file under spec to ignore specific differences:
```
ignoreDifferences:
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
name: cert-manager-webhook
jqPathExpressions:
- .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
- .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")
```

In that case, the updated cert-manager manifest would be as follows:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
namespace: argocd
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: cert-manager
server: https://kubernetes.default.svc
project: default
source:
chart: cert-manager
repoURL: https://charts.jetstack.io
targetRevision: [[VAR::cert_manager_latest_version]]
helm:
values: |
installCRDs: true
ignoreDifferences:
- group: admissionregistration.k8s.io
kind: ValidatingWebhookConfiguration
name: cert-manager-webhook
jqPathExpressions:
- .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
- .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```

Once ArgoCD syncs the updated manifest, the differences due to the above two keys will be ignored, and cert-manager will be in a complete synchronization state.