e2e error: already exists in namespace '' and cannot be managed by operator-controller

[joelanford]

See https://github.com/operator-framework/operator-controller/actions/runs/11019902090/job/30603456402#step:4:417

Other runs with this error:

• https://github.com/operator-framework/operator-controller/actions/runs/11012389326/job/30578486504
• https://github.com/operator-framework/operator-controller/actions/runs/11012652037/job/30579308852
• https://github.com/operator-framework/operator-controller/actions/runs/11013963290/job/30583494035
• https://github.com/operator-framework/operator-controller/actions/runs/11014007708/job/30583637853
• https://github.com/operator-framework/operator-controller/actions/runs/11019902090/job/30603456402

=== RUN   TestClusterExtensionInstallReResolvesWhenManagedContentChanged
    cluster_extension_install_test.go:746: When a cluster extension is installed from a catalog
    cluster_extension_install_test.go:747: It resolves again when managed content is changed
    cluster_extension_install_test.go:769: It installs the specified package with correct bundle path
    cluster_extension_install_test.go:770: By creating the ClusterExtension resource
    cluster_extension_install_test.go:773: By reporting a successful installation
    cluster_extension_install_test.go:774: 
        	Error Trace:	/home/runner/work/operator-controller/operator-controller/test/e2e/cluster_extension_install_test.go:780
        	            				/opt/hostedtoolcache/go/1.22.5/x64/src/runtime/asm_amd64.s:1695
        	Error:      	Not equal: 
        	            	expected: "True"
        	            	actual  : "False"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(v1.ConditionStatus) (len=4) "True"
        	            	+(v1.ConditionStatus) (len=5) "False"
        	            	 
    cluster_extension_install_test.go:774: 
        	Error Trace:	/home/runner/work/operator-controller/operator-controller/test/e2e/cluster_extension_install_test.go:781
        	            				/opt/hostedtoolcache/go/1.22.5/x64/src/runtime/asm_amd64.s:1695
        	Error:      	Not equal: 
        	            	expected: "Succeeded"
        	            	actual  : "Failed"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-Succeeded
        	            	+Failed
    cluster_extension_install_test.go:774: 
        	Error Trace:	/home/runner/work/operator-controller/operator-controller/test/e2e/cluster_extension_install_test.go:782
        	            				/opt/hostedtoolcache/go/1.22.5/x64/src/runtime/asm_amd64.s:1695
        	Error:      	"CustomResourceDefinition 'probes.monitoring.coreos.com' already exists in namespace '' and cannot be managed by operator-controller" does not contain "Installed bundle"
    cluster_extension_install_test.go:774: 
        	Error Trace:	/home/runner/work/operator-controller/operator-controller/test/e2e/cluster_extension_install_test.go:774
        	Error:      	Condition never satisfied
        	Test:       	TestClusterExtensionInstallReResolvesWhenManagedContentChanged
--- FAIL: TestClusterExtensionInstallReResolvesWhenManagedContentChanged (63.29s)

[tmshort]

This is inconsistent, when it shows up, a re-run of the e2e usually succeeds. This is a fairly frequent flake.

[joelanford]

Sounds like the reason for this flake is that we reuse bundle objects in different tests and do not properly wait for cleanup to occur.

Short term fix:

• Wait for bundle objects to be fully deleted in the test cleanup

Long term fix:

• Automatically generate random test fixtures and run tests in parallel. This setup resolves the conflict problem and helps exercise the controller under more typical load where multiple ClusterExtensions and ClusterCatalogs may exist at once.

[rashmi43]

hi @joelanford for the short term fix, should I add a grace period?

[rashmi43]

/assign rashmi43

[m1kola]

I think this is related #1354

[tmshort]

#1354 seems to be WIP. I'm taking a look at this as well.