Higher CPU usage without load

[pjbgf]

The Image reflector controller consumes a lot more CPU than all other controllers combined, even when under a lot less load.

In my setup it is responsible for over 95% of all Flux CPU time:

The memory usage looks more representative:

The flux setup has the objects below with an interval range of 1-5min:

• 17 GitRepositories
• 3 HelmRepositories
• 7 Kustomizations
• 1 Image Repository (~300 tags)
• 1 Image Policy
• 5 Image Automations

Flux version (although the same has been observed in earlier versions):

flux: v0.38.0
helm-controller: v0.28.0
image-automation-controller: v0.28.0
image-reflector-controller: v0.23.1
kustomize-controller: v0.32.0
notification-controller: v0.30.0
source-controller: v0.33.0

An initial investigation pointed to badger being the culprit, as even without any ImageRepository it still has high CPU usage.
We probably would benefit from some fine tuning of badger's default settings.

[glad2os]

I have the same issue.

Thats all my resources

% flux get all
NAME                            REVISION                SUSPENDED       READY   MESSAGE                                           
gitrepository/flux-system       main@sha1:c09065c0      False           True    stored artifact for revision 'main@sha1:c09065c0'

NAME                            REVISION        SUSPENDED       READY   MESSAGE                                      
helmchart/tg-bot-tg-bot-release 0.1.4           False           True    packaged 'tg-bot' chart with version '0.1.4'

NAME                    LAST SCAN       SUSPENDED       READY   MESSAGE                                                                 
imagerepository/tg-bot                  True            False   failed to configure authentication options: secrets "regcred" not found

NAME                            LATEST IMAGE    READY   MESSAGE                                             
imagepolicy/tg-bot-policy                       False   referenced ImageRepository has not been scanned yet

NAME                                    LAST RUN                        SUSPENDED       READY   MESSAGE               
imageupdateautomation/tg-bot-automation 2024-10-26T00:48:05+03:00       True            True    repository up-to-date

NAME                            REVISION                SUSPENDED       READY   MESSAGE                              
kustomization/flux-system       main@sha1:c09065c0      False           True    Applied revision: main@sha1:c09065c0

NAME                            SUSPENDED       READY   MESSAGE                                                                                               
receiver/gitlab-flux-system     False           True    Receiver initialized for path: /hook/44720991f2ce92bdfb144da4b110b193c39bfa7a1a63b1bdaf847de789c01a2d

NAME                    READY   MESSAGE           
provider/telegram       True    Provider is Ready

NAME                            SUSPENDED       READY   MESSAGE        
alert/all-events-telegram       False           True    Alert is Ready

The flux restarts due out of the CPU limits

% kubectl get pods -n flux-system
NAME                                           READY   STATUS    RESTARTS        AGE
helm-controller-69d588546b-2qmgl               1/1     Running   10 (168m ago)   3d6h
image-automation-controller-8666c8ddfc-dgr9g   1/1     Running   11 (168m ago)   3d6h
image-reflector-controller-6689cfc4bd-bsvlm    1/1     Running   12 (168m ago)   3d6h
kustomize-controller-74fb56995-ttv26           1/1     Running   11 (168m ago)   3d6h
notification-controller-5d794dd575-pv4cx       1/1     Running   11 (168m ago)   3d6h
source-controller-6d597849c8-tbnnt             1/1     Running   10 (168m ago)   3d6h

Top command:

% kubectl top pods -n flux-system
NAME                                           CPU(cores)   MEMORY(bytes)   
helm-controller-69d588546b-2qmgl               5m           17Mi            
image-automation-controller-8666c8ddfc-dgr9g   4m           12Mi            
image-reflector-controller-6689cfc4bd-bsvlm    29m          32Mi            
kustomize-controller-74fb56995-ttv26           4m           20Mi            
notification-controller-5d794dd575-pv4cx       4m           20Mi            
source-controller-6d597849c8-tbnnt             8m           19Mi            

% flux version
flux: v2.4.0
distribution: flux-v2.4.0
helm-controller: v1.1.0
image-automation-controller: v0.39.0
image-reflector-controller: v0.33.0
kustomize-controller: v1.4.0
notification-controller: v1.4.0
source-controller: v1.4.1