Slower Reconciles with v1.5.1

[ChuckQuinnIV]

When upgrading from kustomize-controller 1.5.0 to 1.5.1, we're seeing a significant slowdown in reconcile times.

Using one Kustomization as an example, which manages ~1500 NetworkPolicies:

Average reconcile time on 1.5.0: 25s
Average reconcile time on 1.5.1: 2m30s

Example Kustomization setup:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: example-kustomize
spec:
  force: false
  interval: 60m
  path: ./path/to/netpols
  prune: true
  sourceRef:
    kind: GitRepository
    name: example-repo

[stefanprodan]

Is the 2m duration for 1.5.1 consistent, if you move the controller pod to a different node is the timing diffrent?

If you set the image back to 1.5.0 does the reconcile time comes back to normal?

Do you have wait: true in the Flux Kustomization?

[aorith]

I just tested it using a kind k8s cluster. It doesn't make a difference whether wait is true or false.

To test it I created some dummy namespaces (250) each with a configmap using this script:

#!/usr/bin/env bash

set -euo pipefail

NUM_NAMESPACES=${1:-250}

for i in $(seq 1 "$NUM_NAMESPACES"); do
    NS="test-$i"
    DIR="$NS"
    mkdir -p "$DIR"

    cat >"$DIR/namespace.yaml" <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: $NS
EOF

    cat >"$DIR/cm.yaml" <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: dummy-config
  namespace: $NS
data:
  dummy-key: "dummy-value-${NS}"
EOF

done

With kustomize-controller 1.5.1:

2025-04-24T14:32:11.453Z info Kustomization/apps.flux-system - Reconciliation finished in 2m10.053372016s, next run in 15m0s

With With kustomize-controller 1.5.0 I don't even see the log about the time it took to reconcile, metrics tell me that it takes 1.5s

To switch between versions I used this patch:

  - patch: |
      - op: replace
        path: /spec/template/spec/containers/0/image
        value: ghcr.io/fluxcd/kustomize-controller:v1.5.0
    target:
      kind: Deployment
      name: kustomize-controller

[stefanprodan]

It doesn't make a difference whether wait is true or false.

It should make a huge difference, in 1.5.0 the health check watchers use the new Kubernetes Aggregated Discovery while in 1.5.1 we had to revert it.

Are the above tests with wait: true?

[aorith]

I just repeated the tests again with wait: false, using flux reconcile ks apps --with-source for both kustomize-controller versions (no actual changes to the manifests):

1.5.0:

2025-04-24T15:02:03.252Z info Kustomization/apps.flux-system - server-side apply for cluster definitions completed
2025-04-24T15:02:03.448Z info Kustomization/apps.flux-system - server-side apply completed
2025-04-24T15:02:03.484Z info Kustomization/apps.flux-system - Reconciliation finished in 968.028625ms, next run in 15m0s

1.5.1:

2025-04-24T15:03:38.173Z info Kustomization/apps.flux-system - server-side apply for cluster definitions completed
2025-04-24T15:04:14.223Z info Kustomization/apps.flux-system - server-side apply completed
2025-04-24T15:04:14.285Z info Kustomization/apps.flux-system - Reconciliation finished in 59.949239944s, next run in 15m0s

[stefanprodan]

Can you please run the test with ghcr.io/fluxcd/kustomize-controller:rc-a5e315e9

This RC is build from main where we've rolled back the controller-runtime change from 1.5.1, it uses the same RestMapper as in 1.5.0

[aorith]

k -n flux-system get deployments.apps kustomize-controller -o yaml | grep image:
        image: ghcr.io/fluxcd/kustomize-controller:rc-a5e315e9

2025-04-24T15:43:09.451Z info Kustomization/apps.flux-system - server-side apply for cluster definitions completed
2025-04-24T15:43:45.503Z info Kustomization/apps.flux-system - server-side apply completed
2025-04-24T15:43:45.581Z info Kustomization/apps.flux-system - Reconciliation finished in 1m0.312682902s, next run in 15m0s

Doesn't seem to help, maybe its another change made between 1.5.0 and 1.5.1

[stefanprodan]

The other changes were:

• sigs.k8s.io/controller-runtime v0.20.1 -> v0.20.2
• k8s.io/apimachinery v0.32.1 -> v0.32.2

[aorith]

Just ran some more tests building the controller on different commits, the first one going back from 1.5.1 that reconciles fast is 9ac2bba

Building against 8b6f4bc makes the reconciliation slow again...

[stefanprodan]

The StatusPoller is used for health check only, but if this regression happens with wait: false then my guess is that the issue is in Kubernetes packages v1.32.1 -> v1.32.2 or controller-runtime v0.20.1 -> v0.20.2

[aorith]

Yep, this happens when using wait: False.
The additional time seems to be added during the manager.ApplyAll and manager.WaitForSet calls, but I wasn't able to debug it much further ...

[stefanprodan]

We've published an image containing the fix for this issue. Until the next Flux release you can use:

• ghcr.io/fluxcd/kustomize-controller:rc-d1570458

[lodotek]

YAY! Happy I found this issue, as it was causing us lots of headaches. I think it would be best if this stayed open until the fix is officially released, as I am lucky that I just happened to look in the closed issues.