Test failures on openqa.opensuse.org

[dcermak]

• all download of disk images are currently failing on o3: https://progress.opensuse.org/issues/94997
• kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk fails to boot: https://openqa.opensuse.org/tests/2044724, https://openqa.opensuse.org/tests/2044725 (EFI & non-EFI)
• kiwi-test-image-luks.x86_64-1.15.1-Build11.19.raw: fails to boot but not only on o3, this is a known issue. https://openqa.opensuse.org/tests/2044726, https://openqa.opensuse.org/tests/2044727
• kiwi-test-image-lvm.x86_64-1.42.1-Build12.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044728, https://openqa.opensuse.org/tests/2044729
• kiwi-test-image-overlayroot.x86_64-1.42.1-Build12.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044732, https://openqa.opensuse.org/tests/2044733
• kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build11.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044734, https://openqa.opensuse.org/tests/2044735
• kiwi-test-image-disk-simple.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044784, https://openqa.opensuse.org/tests/2044785
• kiwi-test-image-lvm.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044788, https://openqa.opensuse.org/tests/2044789
• kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044790, https://openqa.opensuse.org/tests/2044791

All items on that list (except the busted LUKS image) are vmdk images, which leads me to believe that something is wrong with vmdk on o3, whereas it works on my box. @okurz @foursixnine @asmorodskyi any ideas?

[okurz]

• all download of disk images are currently failing on o3: https://progress.opensuse.org/issues/94997

  • kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk fails to boot: https://openqa.opensuse.org/tests/2044724, https://openqa.opensuse.org/tests/2044725 (EFI & non-EFI)

Following https://openqa.opensuse.org/tests/2044724#settings I can see the setting for HDD_1 being kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk but I can not find that asset over https://openqa.opensuse.org/admin/assets so it was either never there or (likely) was pruned already. https://openqa.opensuse.org/admin/job_templates/85 so far is configured for only the default 5GB and 4.8GB are used so likely any other image does not fit meaning other images will need to be deleted. Also the history https://openqa.opensuse.org/tests/2044724#next_previous shows that jobs did work. So I have now assigned 30GB to the group so I suggest you try again.

[dcermak]

I have retriggered all the tests yesterday but sadly all the tests involving the vmdk images are still failing. Here is the full list of tests:

• https://openqa.opensuse.org/tests/2055674
• https://openqa.opensuse.org/tests/2055673
• https://openqa.opensuse.org/tests/2055672
• https://openqa.opensuse.org/tests/2055671
• https://openqa.opensuse.org/tests/2055668
• https://openqa.opensuse.org/tests/2055667
• https://openqa.opensuse.org/tests/2055618
• https://openqa.opensuse.org/tests/2055617
• https://openqa.opensuse.org/tests/2055617
• https://openqa.opensuse.org/tests/2055616
• https://openqa.opensuse.org/tests/2055615
• https://openqa.opensuse.org/tests/2055611
• https://openqa.opensuse.org/tests/2055612
• https://openqa.opensuse.org/tests/2055608
• https://openqa.opensuse.org/tests/2055607

It really looks like openQA is not downloading these images properly. When I look at https://openqa.opensuse.org/admin/assets then that page only lists the vmdk images for the Leap and none for Tumbleweed (the vmdk images are also missing under the "Logs & Assets" tab there).

@okurz Do you know if the openQA logs could be in some way helpful here?

[asmorodskyi]

if you open os-autoinst.log you will see such lines in very beginning :

[2021-11-25T01:11:38.102292+01:00] [info] +++ setup notes +++
[2021-11-25T01:11:38.102627+01:00] [info] Running on openqaworker4:7 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64)
[2021-11-25T01:11:38.108770+01:00] [debug] Found HDD_1, caching kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk
[2021-11-25T01:11:38.112266+01:00] [info] Downloading kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk, request #480464 sent to Cache Service
[2021-11-25T01:12:03.561175+01:00] [info] Download of kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk processed:
[info] [#480464]
Cache size of "/var/lib/openqa/cache" is 48 GiB, with limit 50 GiB
[info] [#480464]
Downloading "kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" from "http://openqa1-opensuse/tests/2055611/asset/hdd/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk"
[info] [#480464]
Size of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" is 954 MiB, with ETag ""3b9d0000-5d18c5fe40064""
[info] [#480464]
Download of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" successful, new cache size is 49 GiB

so asset is downloaded BUT it is vmdk not qcow and here I pass mike to @okurz because I have no idea if openQA support VMWare images ( spoiler : most probably not )

[dcermak]

I have quickly taken a peek at /var/log/factory/hdd from ariel and the assets are not there:

dancermak@ariel:~> ll /var/lib/openqa/factory/hdd/|grep vmdk
-rw-r--r-- 1 geekotest nogroup   816513024 Nov 24 17:43 kiwi-test-image-disk-simple.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   844431360 Nov 24 17:43 kiwi-test-image-lvm.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   249823232 Nov 24 17:43 kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk

But some assets are there and the corresponding tests still failed… (e.g. https://openqa.opensuse.org/tests/2055673 uses kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk)

[okurz]

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

[asmorodskyi]

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

@okurz something to take into consideration . @dcermak stating that same thing is working on his local instance . I did some comparison of log files from his local instance and one thing which I noticed that he has os-autoinst version=23 which saying about potential regression

[dcermak]

Reported as https://progress.opensuse.org/issues/103203

[okurz]

@asmorodskyi good hint. @dcermak thanks for the ticket. Asked for more information (if possible) in the ticket.

[dcermak]

The vmdk disk failures have been fixed via os-autoinst/os-autoinst#1907.