kola port clashing for qemu user mode networking #4117
Description
When we run kola in the pipeline we run many many tests in a single run, but only 4 at a time (--parallel=4).
It appears, though, that some qemu processes are hanging around longer. For example:
core@coreos-ppc64le-builder:~$ sudo ss -lutnp | grep qemu
udp UNCONN 0 0 0.0.0.0:32771 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=143))
udp UNCONN 0 0 0.0.0.0:33353 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=163))
udp UNCONN 0 0 0.0.0.0:33513 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=185))
udp UNCONN 0 0 0.0.0.0:35914 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=122))
udp UNCONN 0 0 0.0.0.0:36387 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=152))
udp UNCONN 0 0 0.0.0.0:36766 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=133))
udp UNCONN 0 0 0.0.0.0:36842 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=169))
udp UNCONN 0 0 0.0.0.0:36847 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=108))
udp UNCONN 0 0 0.0.0.0:37014 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=118))
udp UNCONN 0 0 0.0.0.0:37312 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=114))
udp UNCONN 0 0 0.0.0.0:37328 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=156))
udp UNCONN 0 0 0.0.0.0:37600 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=165))
udp UNCONN 0 0 0.0.0.0:37746 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=145))
udp UNCONN 0 0 0.0.0.0:37902 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=107))
udp UNCONN 0 0 0.0.0.0:38864 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=168))
udp UNCONN 0 0 0.0.0.0:38962 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=110))
udp UNCONN 0 0 0.0.0.0:40629 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=112))
udp UNCONN 0 0 0.0.0.0:41679 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=159))
udp UNCONN 0 0 0.0.0.0:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=142))
udp UNCONN 0 0 0.0.0.0:42216 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=170))
udp UNCONN 0 0 0.0.0.0:43546 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=149))
udp UNCONN 0 0 0.0.0.0:44382 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=121))
udp UNCONN 0 0 0.0.0.0:44477 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=146))
udp UNCONN 0 0 0.0.0.0:44969 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=106))
udp UNCONN 0 0 0.0.0.0:45113 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=135))
udp UNCONN 0 0 0.0.0.0:45711 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=30))
udp UNCONN 0 0 0.0.0.0:45852 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=130))
udp UNCONN 0 0 0.0.0.0:46064 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=126))
udp UNCONN 0 0 0.0.0.0:47320 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=154))
udp UNCONN 0 0 0.0.0.0:47657 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=131))
udp UNCONN 0 0 0.0.0.0:48291 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=187))
udp UNCONN 0 0 0.0.0.0:48428 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=148))
udp UNCONN 0 0 0.0.0.0:50286 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=155))
udp UNCONN 0 0 0.0.0.0:50475 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=140))
udp UNCONN 0 0 0.0.0.0:50478 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=120))
udp UNCONN 0 0 0.0.0.0:51084 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=164))
udp UNCONN 0 0 0.0.0.0:51150 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=158))
udp UNCONN 0 0 0.0.0.0:51467 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=160))
udp UNCONN 0 0 0.0.0.0:51639 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=115))
udp UNCONN 0 0 0.0.0.0:51643 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=162))
udp UNCONN 0 0 0.0.0.0:51867 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=138))
udp UNCONN 0 0 0.0.0.0:51911 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=134))
udp UNCONN 0 0 0.0.0.0:52978 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=144))
udp UNCONN 0 0 0.0.0.0:53949 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=171))
udp UNCONN 0 0 0.0.0.0:54155 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=129))
udp UNCONN 0 0 0.0.0.0:54285 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=166))
udp UNCONN 0 0 0.0.0.0:54296 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=167))
udp UNCONN 0 0 0.0.0.0:54434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=137))
udp UNCONN 0 0 0.0.0.0:55749 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=189))
udp UNCONN 0 0 0.0.0.0:55831 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=113))
udp UNCONN 0 0 0.0.0.0:55904 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=139))
udp UNCONN 0 0 0.0.0.0:56132 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=111))
udp UNCONN 0 0 0.0.0.0:56266 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=182))
udp UNCONN 0 0 0.0.0.0:56434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=151))
udp UNCONN 0 0 0.0.0.0:56636 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=141))
udp UNCONN 0 0 0.0.0.0:56930 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=153))
udp UNCONN 0 0 0.0.0.0:57442 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=117))
udp UNCONN 0 0 0.0.0.0:57848 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=136))
udp UNCONN 0 0 0.0.0.0:57890 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=150))
udp UNCONN 0 0 0.0.0.0:58370 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=116))
udp UNCONN 0 0 0.0.0.0:58738 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=157))
udp UNCONN 0 0 0.0.0.0:59310 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=161))
udp UNCONN 0 0 0.0.0.0:59466 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=128))
udp UNCONN 0 0 0.0.0.0:60703 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=119))
udp UNCONN 0 0 *:37513 *:* users:(("qemu-system-ppc",pid=1975486,fd=127))
udp UNCONN 0 0 *:39929 *:* users:(("qemu-system-ppc",pid=1975486,fd=125))
udp UNCONN 0 0 *:46748 *:* users:(("qemu-system-ppc",pid=1975486,fd=124))
udp UNCONN 0 0 *:49099 *:* users:(("qemu-system-ppc",pid=1975486,fd=132))
udp UNCONN 0 0 *:50749 *:* users:(("qemu-system-ppc",pid=1975486,fd=123))
udp UNCONN 0 0 *:54827 *:* users:(("qemu-system-ppc",pid=1975486,fd=147))
tcp LISTEN 0 1 127.0.0.1:33359 0.0.0.0:* users:(("qemu-system-ppc",pid=1975455,fd=19))
tcp LISTEN 0 1 127.0.0.1:33229 0.0.0.0:* users:(("qemu-system-ppc",pid=1975987,fd=19))
tcp LISTEN 0 1 127.0.0.1:38673 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=19))
tcp LISTEN 0 1 127.0.0.1:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975753,fd=19))
The majority of those are owned by the same process so there might be some opportunity for us to figure out how to clean those up, or maybe they are legitimate somehow?
The bigger problem however is that we had a port clash with the kdump.crash.nfs test. The test requires the use of the "Qemu Host" IP:Port 10.0.2.2:2049 so it can end up clashing if something else happens to be using that port already.
Something like this happened recently in the RHCOS pipeline:
[2025-05-30T14:53:17.900Z] + set -xeuo pipefail
[2025-05-30T14:53:17.900Z] ++ umask
[2025-05-30T14:53:17.900Z] + '[' 0022 = 0000 ']'
[2025-05-30T14:53:17.900Z] + cd /home/jenkins/agent/workspace/build-arch
[2025-05-30T14:53:17.900Z] + cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=latest --output-dir=/home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola --on-warn-failure-exit-77 --arch=ppc64le '--tag=!reprovision' '--tag=!openshift' --parallel=4
[2025-05-30T14:53:19.782Z] warning: /sys/fs/selinux appears to be mounted but should not be; enabling workaround
[2025-05-30T14:53:19.782Z] kola -p qemu --build latest run --rerun --allow-rerun-success=tags=needs-internet --on-warn-failure-exit-77 --arch=ppc64le --tag=!reprovision --tag=!openshift --parallel=4 --output-dir /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T14:53:20.708Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T14:53:20.708Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T14:53:21.275Z] === RUN ostree.hotfix
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.disks.lvmdevices
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.clhm.ignition-warnings
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.containers.quadlet
[2025-05-30T14:53:21.275Z] === RUN coreos.misc.disk.varlibcontainers
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.mount.disks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.nameserver
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.boot.grub2-install
[2025-05-30T14:53:21.275Z] === RUN fips.enable
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ntp.chrony.dhcp-propagation
[2025-05-30T14:53:21.275Z] === RUN multipath.day1
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.systemd-enable-units
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.kargs
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.simple
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.reboot
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.delete-config
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.mtu-on-bond-ignition
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.sethostname
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file
[2025-05-30T14:53:21.275Z] === RUN fips.enable.partitions
[2025-05-30T14:53:21.275Z] === RUN rhcos.sssd
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.no-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.gshadow
[2025-05-30T14:53:21.275Z] === RUN rhcos.selinux.manage
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.ignition.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.podman.rootless-systemd
[2025-05-30T14:53:21.275Z] === RUN non-exclusive-test-bucket-0
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.luks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.force-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ssh.custom-host-key-permissions
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.prefer-ignition-networking
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.groups
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.network-online
[2025-05-30T14:53:21.276Z] === RUN coreos.selinux.boolean
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.rd-net-timeout-carrier
[2025-05-30T14:53:21.276Z] === RUN ext.config.disk.nvme-compat-symlink
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.boot.bootupd-validate
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.systemd.enable-service
[2025-05-30T14:53:21.276Z] === RUN rpmostree.upgrade-rollback
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.stable-boot
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.ssh
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.hostname.fallback-hostname
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.failure
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.v2.users
[2025-05-30T14:53:21.276Z] === RUN coreos.auth.verify
[2025-05-30T14:53:21.276Z] === RUN multipath.day2
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.policy
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.remote
[2025-05-30T14:53:21.276Z] === RUN basic
[2025-05-30T14:53:21.276Z] === RUN rpmostree.status
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.team-dhcp-via-ignition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.mtu-on-bond-kargs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.replace-rt-kernel
[2025-05-30T14:53:21.276Z] === RUN ostree.sync
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.selinux.enforcing
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation
[2025-05-30T14:53:21.276Z] === RUN ostree.remote
[2025-05-30T14:53:21.276Z] === RUN ext.config.systemd.journal-compat
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.once
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.symlink
[2025-05-30T14:53:21.276Z] === RUN ostree.unlock
[2025-05-30T14:53:21.276Z] === RUN multipath.partition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.kargs-rd-net
[2025-05-30T14:53:21.276Z] === RUN rhcos.services-disabled
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.nfs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.startlimit
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.resource.remote
[2025-05-30T14:53:21.276Z] === RUN rpmostree.install-uninstall
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.kdump.crash
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.no-default-initramfs-net-propagation.bootif
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.condition-needs-update
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.state
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.var-mount.scsi-id
[2025-05-30T14:53:21.276Z] === RUN rhcos.network.multiple-nics
[2025-05-30T14:53:21.276Z] === RUN podman.base
[2025-05-30T14:53:21.276Z] === RUN rhcos.selinux.boolean.persist
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.bridge-static-via-kargs
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.mount.partitions
[2025-05-30T14:53:59.909Z] === RUN ostree.hotfix/unlock
[2025-05-30T14:54:01.789Z] --- PASS: rpmostree.status (40.41s)
[2025-05-30T14:54:03.679Z] --- PASS: ext.config.rpm-ostree.startlimit (42.28s)
[2025-05-30T14:54:03.934Z] --- PASS: ext.config.shared.systemd.network-online (42.65s)
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/install
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/uninstall
[2025-05-30T14:54:19.010Z] === RUN ostree.hotfix/persist
[2025-05-30T14:54:40.871Z] === RUN basic/MachineID
[2025-05-30T14:54:41.125Z] === RUN basic/RHCOSGrowpart
[2025-05-30T14:54:41.380Z] === RUN basic/PortSSH
[2025-05-30T14:54:41.380Z] === RUN basic/DbusPerms
[2025-05-30T14:54:41.947Z] === RUN basic/ServicesActive
[2025-05-30T14:54:41.947Z] === RUN basic/ReadOnly
[2025-05-30T14:54:42.201Z] === RUN basic/Useradd
[2025-05-30T14:54:44.084Z] --- PASS: ext.config.shared.networking.hostname.fallback-hostname (42.11s)
[2025-05-30T14:54:45.450Z] --- PASS: basic (41.98s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/MachineID (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/RHCOSGrowpart (0.17s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/PortSSH (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/DbusPerms (0.47s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ServicesActive (0.15s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ReadOnly (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/Useradd (0.26s)
[2025-05-30T14:54:53.518Z] === RUN ostree.hotfix/rollback
[2025-05-30T14:55:15.379Z] --- PASS: ext.config.shared.ignition.remote (68.46s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix (131.57s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/unlock (19.34s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/install (1.01s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/uninstall (0.58s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/persist (33.57s)
[2025-05-30T14:55:33.411Z] --- PASS: ostree.hotfix/rollback (36.74s)
[2025-05-30T14:55:51.430Z] --- PASS: coreos.auth.verify (38.99s)
[2025-05-30T14:56:13.300Z] --- PASS: coreos.ignition.v2.users (40.33s)
[2025-05-30T14:56:16.560Z] --- PASS: coreos.ignition.failure (25.20s)
[2025-05-30T14:56:19.066Z] --- PASS: multipath.day2 (93.63s)
[2025-05-30T14:56:57.700Z] --- PASS: coreos.ignition.systemd.enable-service (39.70s)
[2025-05-30T14:57:07.623Z] --- PASS: ext.config.shared.networking.nmstate.policy (142.97s)
[2025-05-30T14:57:14.139Z] --- PASS: ext.config.shared.networking.mtu-on-bond-ignition (54.41s)
[2025-05-30T14:57:35.993Z] --- PASS: ext.config.shared.ignition.stable-boot (42.16s)
[2025-05-30T14:57:48.145Z] --- PASS: coreos.ignition.groups (39.44s)
[2025-05-30T14:57:50.649Z] === RUN rpmostree.upgrade-rollback/upgrade
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.networking.prefer-ignition-networking (48.79s)
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.ssh.custom-host-key-permissions (42.20s)
[2025-05-30T14:58:51.152Z] === RUN rpmostree.upgrade-rollback/rollback
[2025-05-30T14:58:52.512Z] --- PASS: kdump.crash.ssh (155.81s)
[2025-05-30T14:58:52.512Z] cluster.go:151: Running as unit: run-r1a8a98404bfa4f99ba5db0a15aceba72.service
[2025-05-30T14:59:14.374Z] --- PASS: ext.config.shared.networking.force-persist-ip (49.33s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback (147.12s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/upgrade (60.35s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/rollback (46.89s)
[2025-05-30T15:00:12.931Z] --- PASS: ext.config.shared.var-mount.luks (101.53s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps
[2025-05-30T15:00:39.405Z] --- PASS: fips.enable (105.63s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.license
[2025-05-30T15:00:39.964Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.growfs
[2025-05-30T15:00:40.887Z] === RUN non-exclusive-test-bucket-0/ext.config.files.localtime
[2025-05-30T15:00:41.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink
[2025-05-30T15:00:42.369Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot
[2025-05-30T15:00:44.250Z] === RUN non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga
[2025-05-30T15:00:45.172Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables
[2025-05-30T15:01:03.255Z] === RUN non-exclusive-test-bucket-0/ext.config.files.conntrack-tools
[2025-05-30T15:01:03.815Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.sudoers
[2025-05-30T15:01:04.737Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme
[2025-05-30T15:01:06.618Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start
[2025-05-30T15:01:08.507Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service
[2025-05-30T15:01:09.867Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite
[2025-05-30T15:01:10.790Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.content-origins
[2025-05-30T15:01:12.671Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version
[2025-05-30T15:01:13.231Z] === RUN non-exclusive-test-bucket-0/ext.config.files.env-godebug
[2025-05-30T15:01:16.485Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links
[2025-05-30T15:01:17.045Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled
[2025-05-30T15:01:18.928Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking
[2025-05-30T15:01:25.454Z] --- PASS: ext.config.shared.reboot (74.49s)
[2025-05-30T15:01:27.974Z] === RUN non-exclusive-test-bucket-0/ext.config.files.composefs-maybe
[2025-05-30T15:01:28.228Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.dns
[2025-05-30T15:01:31.495Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota
[2025-05-30T15:01:32.056Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service
[2025-05-30T15:01:32.614Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log
[2025-05-30T15:01:34.497Z] --- PASS: ext.config.shared.ignition.delete-config (141.08s)
[2025-05-30T15:01:34.497Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default
[2025-05-30T15:01:35.419Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setuid
[2025-05-30T15:01:37.301Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client
[2025-05-30T15:01:37.861Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir
[2025-05-30T15:01:38.785Z] === RUN non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service
[2025-05-30T15:01:39.345Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro
[2025-05-30T15:01:40.270Z] === RUN non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service
[2025-05-30T15:01:41.201Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service
[2025-05-30T15:01:41.761Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.logging.printk
[2025-05-30T15:01:42.684Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change
[2025-05-30T15:01:44.567Z] === RUN non-exclusive-test-bucket-0/ext.config.ssh.config
[2025-05-30T15:01:46.447Z] === RUN non-exclusive-test-bucket-0/ext.config.version.rhel-major-version
[2025-05-30T15:01:56.372Z] --- PASS: coreos.unique.boot.ignition.failure (21.50s)
[2025-05-30T15:02:00.528Z] --- PASS: ext.config.shared.podman.rootless-systemd (82.57s)
[2025-05-30T15:02:02.410Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config
[2025-05-30T15:02:02.969Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info
[2025-05-30T15:02:03.893Z] === RUN non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp
[2025-05-30T15:02:04.451Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context
[2025-05-30T15:02:05.374Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts
[2025-05-30T15:02:07.257Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.console-config
[2025-05-30T15:02:08.624Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.tls
[2025-05-30T15:02:15.158Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs
[2025-05-30T15:02:15.412Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read
[2025-05-30T15:02:17.292Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions
[2025-05-30T15:02:19.173Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions
[2025-05-30T15:02:20.531Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy
[2025-05-30T15:02:25.763Z] === RUN non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder
[2025-05-30T15:02:26.322Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setgid
[2025-05-30T15:02:28.203Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context
[2025-05-30T15:02:36.276Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match
[2025-05-30T15:02:38.157Z] --- PASS: ext.config.shared.var-mount.simple (73.45s)
[2025-05-30T15:02:50.312Z] --- PASS: ext.config.shared.ignition.kargs (54.48s)
[2025-05-30T15:03:05.133Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin
[2025-05-30T15:03:06.493Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-bash
[2025-05-30T15:03:07.415Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables
[2025-05-30T15:03:08.337Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming
[2025-05-30T15:03:10.217Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust
[2025-05-30T15:03:12.096Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd
[2025-05-30T15:03:13.979Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers
[2025-05-30T15:03:14.537Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression
[2025-05-30T15:03:15.463Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch
[2025-05-30T15:03:17.972Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.system-generators
[2025-05-30T15:03:19.330Z] --- PASS: ext.config.shared.ignition.systemd-enable-units (41.43s)
[2025-05-30T15:03:19.887Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kdump.service
[2025-05-30T15:03:20.811Z] --- PASS: rhcos.selinux.manage (80.02s)
[2025-05-30T15:03:20.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0 (224.07s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps (1.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.license (0.74s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.growfs (0.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.localtime (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot (1.85s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks (12.47s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables (4.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.conntrack-tools (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.sudoers (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme (1.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.content-origins (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.env-godebug (2.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (8.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.composefs-maybe (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (2.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service (0.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log (1.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setuid (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.logging.printk (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.ssh.config (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.rhel-major-version (15.93s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config (0.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp (0.75s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context (0.74s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.console-config (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.tls (5.80s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions (1.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy (4.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder (0.77s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setgid (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context (7.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match (28.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin (1.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-bash (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables (0.84s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression (0.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch (2.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.system-generators (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kdump.service (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default (0.90s)
[2025-05-30T15:03:33.032Z] --- PASS: ext.config.shared.gshadow (42.37s)
[2025-05-30T15:04:11.662Z] --- PASS: ext.config.shared.networking.no-persist-ip (48.25s)
[2025-05-30T15:04:13.543Z] --- PASS: rhcos.sssd (40.81s)
[2025-05-30T15:04:52.177Z] --- PASS: ext.config.shared.ntp.chrony.dhcp-propagation (86.90s)
[2025-05-30T15:04:54.683Z] --- PASS: multipath.day1 (95.28s)
[2025-05-30T15:05:41.270Z] --- PASS: coreos.ignition.sethostname (39.65s)
[2025-05-30T15:05:42.628Z] --- PASS: ext.config.shared.var-mount.scsi-id (51.02s)
[2025-05-30T15:05:49.154Z] --- PASS: ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file (95.74s)
[2025-05-30T15:05:59.077Z] --- PASS: fips.enable.partitions (108.57s)
[2025-05-30T15:06:31.074Z] --- PASS: ext.config.shared.networking.bridge-static-via-kargs (48.44s)
[2025-05-30T15:06:33.583Z] === RUN podman.base/info
[2025-05-30T15:06:34.505Z] === RUN podman.base/resources
[2025-05-30T15:06:49.331Z] --- PASS: podman.base (50.21s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/info (1.07s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/resources (9.93s)
[2025-05-30T15:06:49.331Z] --- PASS: coreos.ignition.mount.partitions (73.95s)
[2025-05-30T15:06:50.254Z] --- FAIL: kdump.crash.nfs (2.02s)
[2025-05-30T15:06:50.254Z] kdump.go:291: listen tcp :2049: bind: address already in use
[2025-05-30T15:07:05.075Z] --- PASS: rhcos.selinux.boolean.persist (75.01s)
[2025-05-30T15:07:26.941Z] --- PASS: ext.config.systemd.journal-compat (52.94s)
[2025-05-30T15:07:41.760Z] === RUN ostree.unlock/unlock
[2025-05-30T15:07:42.016Z] === RUN ostree.unlock/install
[2025-05-30T15:07:43.376Z] === RUN ostree.unlock/uninstall
[2025-05-30T15:07:43.934Z] === RUN ostree.unlock/discard
[2025-05-30T15:08:01.956Z] === RUN rhcos.services-disabled/ServicesDisabled
[2025-05-30T15:08:03.840Z] --- PASS: multipath.partition (73.31s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled (40.27s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled/ServicesDisabled (0.40s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock (76.70s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock/unlock (1.03s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/install (1.34s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/uninstall (0.67s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/discard (33.76s)
[2025-05-30T15:08:24.302Z] --- PASS: rhcos.network.multiple-nics (96.31s)
[2025-05-30T15:08:46.163Z] --- PASS: coreos.ignition.symlink (39.76s)
[2025-05-30T15:08:52.687Z] --- PASS: ext.config.shared.networking.kargs-rd-net (48.23s)
[2025-05-30T15:09:14.561Z] --- PASS: ext.config.shared.networking.no-default-initramfs-net-propagation.bootif (50.21s)
[2025-05-30T15:09:32.584Z] --- PASS: coreos.ignition.once (71.58s)
[2025-05-30T15:09:59.059Z] --- PASS: ext.config.shared.boot.grub2-install (43.53s)
[2025-05-30T15:09:59.059Z] --- PASS: coreos.misc.disk.varlibcontainers (75.83s)
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/var-lib-containers.mount.wants/[email protected] → /etc/systemd/system/[email protected].
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/local-fs.target.wants/var-lib-containers.mount → /etc/systemd/system/var-lib-containers.mount.
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nameserver (48.43s)
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nmstate.state (75.73s)
[2025-05-30T15:11:03.279Z] --- PASS: ext.config.shared.systemd.condition-needs-update (130.35s)
[2025-05-30T15:11:13.206Z] --- PASS: coreos.ignition.mount.disks (72.43s)
[2025-05-30T15:11:19.736Z] === RUN rpmostree.install-uninstall/install
[2025-05-30T15:11:29.662Z] --- PASS: ext.config.shared.clhm.ignition-warnings (40.83s)
[2025-05-30T15:11:56.145Z] --- PASS: ext.config.shared.containers.quadlet (44.55s)
[2025-05-30T15:12:08.297Z] --- PASS: ext.config.shared.ignition.resource.remote (38.49s)
[2025-05-30T15:12:09.657Z] --- FAIL: ostree.sync (2.02s)
[2025-05-30T15:12:09.657Z] sync.go:117: listen tcp :2049: bind: address already in use
[2025-05-30T15:12:36.136Z] === RUN rpmostree.install-uninstall/uninstall
[2025-05-30T15:13:14.783Z] --- PASS: ext.config.shared.disks.lvmdevices (75.45s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation (80.97s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.kdump.crash (149.09s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall (176.30s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/install (74.08s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/uninstall (65.40s)
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/add
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/list
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/show-url
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/refs
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/summary
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/delete
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote (41.97s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/add (0.33s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/list (0.13s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/show-url (0.25s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/refs (0.23s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/summary (0.52s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/delete (0.51s)
[2025-05-30T15:14:22.406Z] --- PASS: ext.config.shared.networking.mtu-on-bond-kargs (49.03s)
[2025-05-30T15:14:25.663Z] --- PASS: ext.config.shared.selinux.enforcing (73.70s)
[2025-05-30T15:14:25.663Z] mke2fs 1.47.2 (1-Jan-2025)
[2025-05-30T15:14:30.896Z] --- PASS: ext.config.shared.networking.team-dhcp-via-ignition (48.24s)
[2025-05-30T15:14:45.724Z] --- PASS: coreos.unique.boot.failure (18.75s)
[2025-05-30T15:15:03.744Z] --- PASS: ext.config.disk.nvme-compat-symlink (41.07s)
[2025-05-30T15:15:30.215Z] --- PASS: ext.config.shared.boot.bootupd-validate (59.30s)
[2025-05-30T15:16:16.824Z] --- PASS: ext.config.shared.networking.rd-net-timeout-carrier (86.07s)
[2025-05-30T15:16:16.824Z] --- PASS: coreos.selinux.boolean (71.74s)
[2025-05-30T15:18:53.236Z] KVM: Failed to create TCE64 table for liobn 0x80000001
[2025-05-30T15:18:59.774Z] --- PASS: ext.config.rpm-ostree.replace-rt-kernel (287.08s)
[2025-05-30T15:18:59.774Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ======== Re-running failed tests (flake detection) ========
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T15:18:59.774Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T15:18:59.774Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T15:19:00.028Z] === RUN ostree.sync
[2025-05-30T15:19:00.028Z] === RUN kdump.crash.nfs
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
[2025-05-30T15:21:38.968Z] --- PASS: kdump.crash.nfs (157.10s)
[2025-05-30T15:21:38.968Z] cluster.go:151: Running as unit: run-rbc26bb21f6424c438e84e7781f60c9e3.service
[2025-05-30T15:21:38.968Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola/rerun
[2025-05-30T15:21:38.968Z] Error: harness: test suite failed
[2025-05-30T15:21:38.968Z] 2025-05-30T15:21:37Z cli: harness: test suite failed
[2025-05-30T15:21:38.968Z] failed to execute cmd-kola: exit status 1
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
Basically for whatever reason the host clashed: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'. I don't know if another kdump.crash.nfs test was running for another pipeline (or if they would even clash).
You can also reproduce something similar with cosa kola run kdump.crash.nfs --multiply 2 --parallel 2.