Spammy log collectors in clusterctl-upgrade tests

[lentzi90]

We do not clean up BMO before we delete the management cluster in clusterctl-upgrade tests. This leads to error spam in the console log when the log collectors are trying to get logs from the non-existing cluster. It looks like this in clusterctl-upgrade:

[2025-03-26T15:17:15.450Z]   STEP: Dumping all the Cluster API resources in the "clusterctl-upgrade-xg2q0h" namespace @ 03/26/25 15:17:12.427
[2025-03-26T15:17:15.450Z]   INFO: clusterctl describe cluster clusterctl-upgrade-management-84xbf3 --show-conditions=all --show-machinesets=true --grouping=false --echo=true --v1beta2
[2025-03-26T15:17:15.450Z]   STEP: Dumping Pods and Nodes of Cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.676
[2025-03-26T15:17:15.450Z]   STEP: Deleting cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.755
[2025-03-26T15:17:15.450Z]   STEP: Deleting cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.766
[2025-03-26T15:17:15.450Z]   INFO: Waiting for the Cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 to be deleted
[2025-03-26T15:17:15.450Z]   STEP: Waiting for cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 to be deleted @ 03/26/25 15:17:12.792
[2025-03-26T15:19:09.155Z]   INFO: Error starting logs stream for pod baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm, container manager: Get "https://192.168.111.101:10250/containerLogs/baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm/manager?follow=true": dial tcp 192.168.111.101:10250: i/o timeout
[2025-03-26T15:19:09.155Z]   INFO: Error starting logs stream for pod baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm, container manager: Get "https://192.168.111.101:10250/containerLogs/baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm/manager?follow=true": dial tcp 192.168.111.101:10250: i/o timeout
[2025-03-26T15:19:57.621Z] W0326 15:19:54.886924   72143 reflector.go:492] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
[2025-03-26T15:19:57.621Z] W0326 15:19:57.078664   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:19:57.621Z] E0326 15:19:57.079796   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:03.703Z] W0326 15:20:03.222440   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:03.703Z] E0326 15:20:03.222510   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:12.804Z] W0326 15:20:12.438309   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:12.804Z] E0326 15:20:12.438363   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:21.990Z] W0326 15:20:21.654277   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:21.990Z] E0326 15:20:21.654328   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:50.073Z] W0326 15:20:46.230333   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:50.073Z] E0326 15:20:46.230389   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:53.156Z]   STEP: Deleting namespace used for hosting the "clusterctl-upgrade" test spec @ 03/26/25 15:20:52.987
[2025-03-26T15:20:53.156Z]   INFO: Deleting namespace clusterctl-upgrade-xg2q0h

Here is the same for a e2e-integration test:

[2025-03-28T04:53:01.317Z]   STEP: Dumping all the Cluster API resources in the "metal3" namespace @ 03/28/25 04:53:00.48
[2025-03-28T04:53:01.317Z]   INFO: clusterctl describe cluster test1 --show-conditions=all --show-machinesets=true --grouping=false --echo=true --v1beta2
[2025-03-28T04:53:01.317Z]   STEP: Deleting cluster metal3/test1 @ 03/28/25 04:53:00.965
[2025-03-28T04:53:01.317Z]   STEP: Deleting cluster metal3/test1 @ 03/28/25 04:53:00.981
[2025-03-28T04:53:01.317Z]   INFO: Waiting for the Cluster metal3/test1 to be deleted
[2025-03-28T04:53:01.317Z]   STEP: Waiting for cluster metal3/test1 to be deleted @ 03/28/25 04:53:01.025
[2025-03-28T04:54:24.996Z]   INFO: Error starting logs stream for pod baremetal-operator-system/baremetal-operator-controller-manager-76d5b88db8-p2xvm, container manager: Get "https://192.168.111.101:10250/containerLogs/baremetal-operator-system/baremetal-operator-controller-manager-76d5b88db8-p2xvm/manager?follow=true": dial tcp 192.168.111.101:10250: i/o timeout
[2025-03-28T04:56:03.722Z] W0328 04:55:51.759046   68506 reflector.go:492] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
[2025-03-28T04:56:03.722Z] W0328 04:55:55.189308   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:56:03.722Z] E0328 04:55:55.189639   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:56:03.722Z] W0328 04:55:58.264860   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:56:03.722Z] E0328 04:55:58.265021   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:56:04.476Z] W0328 04:56:04.404948   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:56:04.476Z] E0328 04:56:04.405113   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:56:15.577Z] W0328 04:56:13.621077   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:56:15.577Z] E0328 04:56:13.621255   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:56:29.018Z] W0328 04:56:28.984481   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:56:29.018Z] E0328 04:56:28.984548   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:57:17.559Z] W0328 04:57:15.060893   68506 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-28T04:57:17.559Z] E0328 04:57:15.061147   68506 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=4312\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-28T04:57:58.103Z]   STEP: Checking leftover Metal3Datas, Metal3DataTemplates and Metal3DataClaims @ 03/28/25 04:57:51.582

We should simply delete BMO in the preCleanupManagementCluster function.

/area ci
/triage accepted

[metal3-io-bot]

@lentzi90: The label(s) area/ci cannot be applied, because the repository doesn't have them.

Details

In response to this:

We do not clean up BMO before we delete the management cluster in clusterctl-upgrade tests. This leads to error spam in the console log when the log collectors are trying to get logs from the non-existing cluster. It looks like this:

[2025-03-26T15:17:15.450Z]   STEP: Dumping all the Cluster API resources in the "clusterctl-upgrade-xg2q0h" namespace @ 03/26/25 15:17:12.427
[2025-03-26T15:17:15.450Z]   INFO: clusterctl describe cluster clusterctl-upgrade-management-84xbf3 --show-conditions=all --show-machinesets=true --grouping=false --echo=true --v1beta2
[2025-03-26T15:17:15.450Z]   STEP: Dumping Pods and Nodes of Cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.676
[2025-03-26T15:17:15.450Z]   STEP: Deleting cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.755
[2025-03-26T15:17:15.450Z]   STEP: Deleting cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 @ 03/26/25 15:17:12.766
[2025-03-26T15:17:15.450Z]   INFO: Waiting for the Cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 to be deleted
[2025-03-26T15:17:15.450Z]   STEP: Waiting for cluster clusterctl-upgrade-xg2q0h/clusterctl-upgrade-management-84xbf3 to be deleted @ 03/26/25 15:17:12.792
[2025-03-26T15:19:09.155Z]   INFO: Error starting logs stream for pod baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm, container manager: Get "https://192.168.111.101:10250/containerLogs/baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm/manager?follow=true": dial tcp 192.168.111.101:10250: i/o timeout
[2025-03-26T15:19:09.155Z]   INFO: Error starting logs stream for pod baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm, container manager: Get "https://192.168.111.101:10250/containerLogs/baremetal-operator-system/baremetal-operator-controller-manager-6f774ccc48-tcdnm/manager?follow=true": dial tcp 192.168.111.101:10250: i/o timeout
[2025-03-26T15:19:57.621Z] W0326 15:19:54.886924   72143 reflector.go:492] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
[2025-03-26T15:19:57.621Z] W0326 15:19:57.078664   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:19:57.621Z] E0326 15:19:57.079796   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:03.703Z] W0326 15:20:03.222440   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:03.703Z] E0326 15:20:03.222510   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:12.804Z] W0326 15:20:12.438309   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:12.804Z] E0326 15:20:12.438363   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:21.990Z] W0326 15:20:21.654277   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:21.990Z] E0326 15:20:21.654328   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:50.073Z] W0326 15:20:46.230333   72143 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: Get "https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129": dial tcp 192.168.111.249:6443: connect: no route to host
[2025-03-26T15:20:50.073Z] E0326 15:20:46.230389   72143 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://192.168.111.249:6443/api/v1/pods?resourceVersion=10129\": dial tcp 192.168.111.249:6443: connect: no route to host" logger="UnhandledError"
[2025-03-26T15:20:53.156Z]   STEP: Deleting namespace used for hosting the "clusterctl-upgrade" test spec @ 03/26/25 15:20:52.987
[2025-03-26T15:20:53.156Z]   INFO: Deleting namespace clusterctl-upgrade-xg2q0h

We should simply delete BMO in the preCleanupManagementCluster function.

/area ci
/triage accepted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[lentzi90]

/assign

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[Rozzii]

/remove-lifecycle stale

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[Rozzii]

/remove-lifecycle stale