Image reflector controller uses a lot of ephemeral storage.

[ilya-korol]

Hi, I'm using the Image Reflector Controller, which uses BadgerDB for storage. Over time, the controller creates a large number of 0000*.vlog files in the /data/ volume. Eventually, the storage gets full, leading to Kubernetes evicting the pod.

Tried setting --storage-value-log-file-size=2gb to reduce the number of vlog files and there was no effect.
I can see that many vlog files contain the same key-value pairs with minimal differences, because they all start with the same image name and its tags. It’s worth mentioning that we have a lot of images in ecr.
There is no available configuration options to restrict either the total number of files or the maximum storage size used by Badger.

Is there a way to configure automatic cleanup or GC to prevent unbounded storage growth?
Are there any recommended best practices for managing storage in this use case?

Image Reflector Controller version: v0.31.2
Would appreciate any insights on how to resolve this issue. Thanks!

[stealthybox]

Hi @ilya-korol 👋 nice catch on this.
We needed to manually manage Badger's garbage collection processes.
I've done some synthetic tests and merged #757 which should clean up your vlogs and resolve your issue.

As a followup, we're considering updating the default Badger configuration to more accurately tune for typical use of OCI repos and tags.

Are you able to share some details about the scale you are using the controller in?

• Number of image repos
• Length and size of image tag lists (skopeo list-tags docker://debian | wc -lc)
  (This is a really important to get stats on because badger's default vlog threshold is 1MB, but public repos never really have this many tags)
• CPU and Memory allocated to Flux's image controllers + Actual Memory Usage
• Number of vlog files you observe building up
• Size of disk
• Time period over which your disk-space runs out

[stealthybox]

CHANGELOG.md shows the effective changes between v0.31.2 and v0.34.0, and these are the commits since then: v0.34.0..rc-c4913021

If you're able to upgrade in an affected cluster, could try using this RC image for the Garbage Collector?

• ghcr.io/fluxcd/image-reflector-controller:rc-c4913021
• docker.io/fluxcd/image-reflector-controller:rc-c4913021

You may also want to set --log-level=debug to see the badger GC logs

I'd also update the CRD's in the affected cluster for the unrelated additional status fields:

• kustomize build https://github.com/fluxcd/image-reflector-controller//config/crd/

Depending on if you are using ephemeral storage, I don't know how long it takes for your disk to fill up, but hopefully the GC will remove vlogs for you in a relatively short time-frame.

[ilya-korol]

Hi, thank you for the changes. I will try to use the new image.

Number of image repos: ~500
Length and size of image tag lists: it's quite difficult to count, there are small repos (0-100), medium (100-2000) and large ones (2000+) (the largest is 10000k), average ~ 330 images per repos but it is a rough estimate.
CPU and Memory allocated: CPU requests - 100, limits 2000, Memory - requests 64Mb, limits 2Gb for reflector controller. Actual Memory Usage: 350Mb, with an increase to 950Mb every 40mins.
Number of vlog files: 53 files (253Mb each)
Size of disk: 15Gb
Time period over which your disk-space runs out: 7-10 days

[stefanprodan]

@ilya-korol you can upgrade to Flux 2.6 we shipped the BadgerDB GC in this release.

[ilya-korol]

We use terraform to bootstrap Flux, so we will update provider to 1.6.0
Thanks.

[stefanprodan]

Wait for 1.6.1 will be released today