Feature Request: Introduce a deletionPolicy for deleting resources when the Kustomization is suspended

[dan0sh]

This issue has been identified as a feature request, it's not a bug.

Original issue description:

Describe the bug

When a Flux Kustomization resource is removed from the cluster, the Kubernetes resources managed by that Kustomization remain in the cluster ("orphaned") instead of being deleted.

It appears that the default value for .spec.deletionPolicy=MirrorPrune is not working as expected to clean up resources upon deletion of the parent object. Explicitly setting .spec.deletionPolicy=Delete is required to successfully garbage collect the resources.

Note that setting .spec.deletionPolicy: Delete will not trigger garbage collection if the Kustomization is suspended (.spec.suspend: true) at the time of deletion. The Kustomization must be active for the finalizer to execute the prune logic

Steps to reproduce

1. Deploy a Flux Kustomization without .spec.deletionPolicy set.
2. Observe that resources are synced to the cluster.
3. Delete the Kustomization manifest/object.
4. Observe that the child resources remain running in the cluster.

Expected behavior

When the Kustomization is deleted, all resources defined in the path should be garbage collected from the cluster.

Screenshots and recordings

No response

OS / Distro

k8s v1.33.9 - Linux 9.6 (Blue Onyx)

Flux version

2.7.3

Flux check

✗ flux 2.7.3 <2.8.1 (new CLI version is available, please upgrade) ✔ Kubernetes 1.33.9 >=1.32.0-0 ► checking version in cluster ✔ distribution: flux-v2.7.3 ✔ bootstrapped: true ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v1.4.3 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.7.2 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.7.4 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.7.3 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta3 ✔ buckets.source.toolkit.fluxcd.io/v1 ✔ externalartifacts.source.toolkit.fluxcd.io/v1 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1 ✔ helmreleases.helm.toolkit.fluxcd.io/v2 ✔ helmrepositories.source.toolkit.fluxcd.io/v1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1 ✔ providers.notification.toolkit.fluxcd.io/v1beta3 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Did you set .spec.prune to true?

[dan0sh]

Apologies for not mentioning this earlier. We have prune=true configured. While individual resource deletions work fine (e.g., deleting a Deployment), removing an entire Kustomization doesn't remove its associated resources.

[matheuscscp]

I tried to reproduce this issue in both Flux 2.7.3 and Flux 2.8.1 and was not successful.

Also, the scenario you are describing is a very core/basic expectation for Flux. I think that if this was actually broken, we would be receiving a lot more reports, especially since you are on Flux 2.7.3 which is out there for a while.

Please confirm if I understood the scenario correctly: You have a simple Kustomization with prune: true, deletionPolicy not set, and suspend also not set (hence set to false). Garbage collection does not work when the Kustomization is deleted. Is this your claim?

[dan0sh]

Thanks for looking into this. To clarify the scenario, this is not a standalone Kustomization but part of a Tenant/Nested Kustomization structure.

1. We have a parent Kustomization (named tenants) which manages the definitions of other Kustomizations. This parent has prune: true set.
2. The issue occurs with a child Kustomization managed by that parent.
3. This child Kustomization initially had only prune: true configured (relying on the default deletionPolicy).

When the child Kustomization manifest was deleted (removed from the source managed by the parent tenants), the parent correctly deleted the child Kustomization object from the cluster. However, the resources managed by that child were orphaned and not garbage collected.

I have confirmed that changing the child Kustomization's configuration to explicitly include deletionPolicy: Delete (alongside prune: true) fixes the issue. This fix works only if the child Kustomization is not suspended (e.g. via CLI) at the time of deletion. If it is suspended, the finalizer does not run, and resources remain orphaned regardless of the policy.

[matheuscscp]

Now I mirrored this exact same setup, and again I was not able to reproduce.

I only did that to follow procedure, but I already knew nothing would change: parent-child relationship does not affect garbage collection at all. The parent does not know that the child being deleted is a Flux Kustomization.

Also, let's forget suspension here, if garbage collection is being skipped for suspended objects, that's great, it's the expected behavior.

[dan0sh]

Thank you for confirming the expected behavior regarding suspension.

It appears the root cause is indeed suspend=true. We are removing Kustomizations from the source that are currently suspended in the cluster.

Is there a configuration option to force garbage collection for suspended Kustomizations? We need the finalizer to execute and clean up resources when the Kustomization object is deleted, even if it is currently in a suspended state.

[matheuscscp]

Maybe a new deletionPolicy

[stefanprodan]

The whole point of suspending a Flux object is to avoid GC!

[matheuscscp]

The whole point of suspending a Flux object is to avoid GC!

That was my first thought as well, but suspension also prevents reconciliation...

[dan0sh]

We are using the suspend feature specifically because we want to prevent reconciliation.

[matheuscscp]

By the way, are you saying that deletionPolicy: Delete is running GC when the KS is suspended?? This is a bug!

[matheuscscp]

By the way, are you saying that deletionPolicy: Delete is running GC when the KS is suspended?? This is a bug!

I checked this out and it's correct, Delete is only checked if the object is not suspended, as it should:

// finalizerShouldDeleteResources determines if resources should be deleted
// based on the object's inventory and deletion policy.
// A suspended Kustomization or one without an inventory will not delete resources.
func finalizerShouldDeleteResources(obj *kustomizev1.Kustomization) bool {
	if obj.Spec.Suspend {
		return false
	}

	if obj.Status.Inventory == nil || len(obj.Status.Inventory.Entries) == 0 {
		return false
	}

	switch obj.GetDeletionPolicy() {
	case kustomizev1.DeletionPolicyMirrorPrune:
		return obj.Spec.Prune
	case kustomizev1.DeletionPolicyDelete:
		return true
	case kustomizev1.DeletionPolicyWaitForTermination:
		return true
	default:
		return false
	}
}