Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: update expeted error codes of qubes-vm-update #604

Merged
merged 2 commits into from
Jul 5, 2024

Conversation

marmarek
Copy link
Member

@marmarek marmarek commented Jul 3, 2024

When refreshing metadata fails, it's supposed to exit with code 23.

Related to QubesOS/qubes-core-admin-linux#160

When refreshing metadata fails, it's supposed to exit with code 23.

Related to QubesOS/qubes-core-admin-linux#160
Copy link

codecov bot commented Jul 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.35%. Comparing base (3362ffc) to head (212ff69).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #604   +/-   ##
=======================================
  Coverage   69.35%   69.35%           
=======================================
  Files          58       58           
  Lines       11919    11919           
=======================================
  Hits         8266     8266           
  Misses       3653     3653           
Flag Coverage Δ
unittests 69.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@qubesos-bot
Copy link

qubesos-bot commented Jul 3, 2024

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024070414-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024062115-4.3&flavor=update

  • system_tests_extra

    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
  • system_tests_network_ipv6

  • system_tests_network_updates

    • TC_11_QvmTemplateMgmtVM_debian-12-xfce: test_000_template_list (failure)
      AssertionError: libvirt event impl drain timeout
  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-copy-notifi...
  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

Failed tests

13 failures
  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_fedora-40-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-40-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_splitgpg

  • system_tests_extra

    • [unstable] TC_00_QVCTest_fedora-40-xfce: test_020_webcam (failure)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0

    • [unstable] TC_00_QVCTest_whonix-gateway-17: test_020_webcam (failure + cleanup)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

    • [unstable] TC_00_QVCTest_whonix-workstation-17: test_020_webcam (failure)
      AssertionError: 'qubes-video-companion webcam' exited early (0): b'...

  • system_tests_network_ipv6

  • system_tests_network_updates

    • TC_11_QvmTemplateMgmtVM_debian-12-xfce: test_000_template_list (failure)
      AssertionError: libvirt event impl drain timeout
  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-copy-notifi...
  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/103633#dependencies

4 fixed
  • system_tests_basic_vm_qrexec_gui

  • system_tests_network_updates

  • system_tests_basic_vm_qrexec_gui_zfs

    • TC_20_AudioVM_Pulse_debian-12-xfce-pool: test_223_audio_play_hvm (failure)
      AssertionError: only silence detected, no useful audio data

Unstable tests

  • system_tests_update

    update2/Failed (1/5 times with errors)
    • job 103358 # Test died: command 'script -c 'qubes-vm-update --max-concurrency=...
  • system_tests_suspend

    suspend/ (3/5 times with errors)
    suspend/Failed (3/5 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 103660 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (3/5 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 103660 # wait_serial expected: qr/2E8vz-\d+-/...
  • system_tests_basic_vm_qrexec_gui

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (3/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    • job 103642 AssertionError: too short audio, expected 10s, got 8.64043083900226...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_221_audio_rec_muted_pulseaudio (1/4 times with errors)
    • job 103642 Cannot process volume group qubes_dom0...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/4 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (2/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 102413 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_pvgrub_salt_storage

    TC_41_HVMGrub_debian-12-xfce/test_000_standalone_vm (2/4 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 102428 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    StorageFile/test_001_non_volatile (1/4 times with errors)
    • job 101124 subprocess.CalledProcessError: Command '/usr/lib/qubes/destroy-snap...
    TC_41_HVMGrub_debian-12-xfce/test_010_template_based_vm (2/4 times with errors)
    • job 101773 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    • job 102428 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_extra

    TC_00_QVCTest_whonix-workstation-17/test_010_screenshare (1/4 times with errors)
    • job 101116 self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
    TC_00_QVCTest_debian-12-xfce/test_020_webcam (2/4 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    • job 102420 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_fedora-40-xfce/test_020_webcam (1/4 times with errors)
    • job 101765 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    TC_00_QVCTest_whonix-gateway-17/test_020_webcam (2/4 times with errors)
    • job 102420 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    • job 103649 self.assertNotEqual(vm.run('test -e /dev/vid... AssertionError: 0 == 0
    TC_00_QVCTest_whonix-workstation-17/test_020_webcam (3/4 times with errors)
    • job 101116 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    • job 102420 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
    • job 103649 AssertionError: 'qubes-video-companion webcam' exited early (0): b'...
  • system_tests_usbproxy

    TC_20_USBProxy_core3_whonix-gateway-17/test_070_attach_not_installed_front (1/4 times with errors)
    • job 101101 qubesusbproxy.core3ext.QubesUSBException: Device attach failed: 202...
  • system_tests_network_updates

    TC_00_Dom0Upgrade_whonix-gateway-17/test_006_update_flag_clear (1/4 times with errors)
    • job 101123 Error: Failed to download metadata for repo 'test': Cannot download...
    TC_10_QvmTemplate_debian-12-xfce/test_010_template_install (1/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_fedora-40-xfce/test_010_template_install (1/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_10_QvmTemplate_whonix-gateway-17/test_010_template_install (2/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    • job 103656 AssertionError: libvirt event impl drain timeout
    TC_11_QvmTemplateMgmtVM_debian-12-xfce/test_010_template_install (1/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_fedora-40-xfce/test_010_template_install (1/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
    TC_11_QvmTemplateMgmtVM_whonix-gateway-17/test_010_template_install (1/4 times with errors)
    • job 101123 AssertionError: qvm-template failed: Downloading 'qubes-template-de...
  • system_tests_dispvm

    TC_20_DispVM_fedora-40-xfce/test_100_open_in_dispvm (2/4 times with errors)
    • job 101764 AssertionError: './open-file test.txt' failed with ./open-file test...
    • job 102419 AssertionError: './open-file test.txt' failed with ./open-file test...
  • system_tests_basic_vm_qrexec_gui_zfs

    TC_00_Basic/test_120_start_standalone_with_cdrom_dom0 (1/3 times with errors)
    • job 101113 AssertionError: 1 != 0 : b'Timeout waiting for dom0:loop6 device to...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/3 times with errors)
    • job 101113 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/3 times with errors)
    • job 101113 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_debian-12-xfce-pool/test_223_audio_play_hvm (1/3 times with errors)
    • job 103646 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/3 times with errors)
    • job 101113 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/3 times with errors)
    • job 101113 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui_btrfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/4 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/4 times with errors)
    • job 101110 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/4 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_fedora-40-xfce-pool/test_225_audio_rec_unmuted_hvm (1/4 times with errors)
    • job 101782 AssertionError: too short audio, expected 10s, got 6.28875283446712...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (2/4 times with errors)
    • job 101110 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101782 AssertionError: pacat for test-inst-vm1 (xid 90) running(False) in ...
  • system_tests_basic_vm_qrexec_gui_ext4

    TC_00_Basic/test_141_libvirt_objects_reconnect (1/4 times with errors)
    • job 101111 AssertionError: libvirt event impl drain timeout
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (1/4 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (1/4 times with errors)
    • job 101111 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_debian-12-xfce-pool/test_223_audio_play_hvm (1/4 times with errors)
    • job 102415 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (1/4 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (1/4 times with errors)
    • job 101111 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_basic_vm_qrexec_gui_xfs

    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_220_audio_play_pulseaudio (2/4 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 8.33947845804988...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_222_audio_rec_unmuted_pulseaudio (2/4 times with errors)
    • job 101112 AssertionError: only silence detected, no useful audio data
    • job 101781 AssertionError: too short audio, expected 10s, got 8.17784580498866...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_223_audio_play_hvm (2/4 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101781 AssertionError: too short audio, expected 10s, got 7.71551020408163...
    TC_20_AudioVM_PipeWire_debian-12-xfce-pool/test_226_audio_playback_pipewire (1/4 times with errors)
    • job 102416 AssertionError: too short audio, expected 10s, got 8.31034013605442...
    TC_20_AudioVM_PipeWire_debian-12-xfce-pool/test_228_audio_rec_unmuted_pipewire (1/4 times with errors)
    • job 102416 AssertionError: too short audio, expected 10s, got 8.89784580498866...
    TC_20_AudioVM_PipeWire_fedora-40-xfce-pool/test_228_audio_rec_unmuted_pipewire (1/4 times with errors)
    • job 102416 AssertionError: too short audio, expected 10s, got 9.45630385487528...
    TC_20_AudioVM_PipeWire_debian-12-xfce-pool/test_250_audio_playback_audiovm_pipewire (1/4 times with errors)
    • job 102416 AssertionError: too short audio, expected 10s, got 7.48553287981859...
    TC_20_AudioVM_PipeWire_debian-12-xfce-pool/test_251_audio_playback_audiovm_pipewire_late_start (1/4 times with errors)
    • job 102416 AssertionError: too short audio, expected 10s, got 9.19612244897959...
    TC_20_AudioVM_Pulse_whonix-workstation-17-pool/test_252_audio_playback_audiovm_switch_hvm (2/4 times with errors)
    • job 101112 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 102416 AssertionError: too short audio, expected 10s, got 6.77941043083900...
  • system_tests_update@hw7

    update2/Failed (1/5 times with errors)
    • job 103358 # Test died: command 'script -c 'qubes-vm-update --max-concurrency=...
  • system_tests_update@hw1

    update2/Failed (1/5 times with errors)
    • job 103358 # Test died: command 'script -c 'qubes-vm-update --max-concurrency=...
  • system_tests_basic_vm_qrexec_gui@hw1

    TC_20_AudioVM_Pulse_whonix-workstation-17/test_220_audio_play_pulseaudio (3/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 101758 AssertionError: too short audio, expected 10s, got 8.73532879818594...
    • job 103642 AssertionError: too short audio, expected 10s, got 8.64043083900226...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_221_audio_rec_muted_pulseaudio (1/4 times with errors)
    • job 103642 Cannot process volume group qubes_dom0...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_222_audio_rec_unmuted_pulseaudio (2/4 times with errors)
    • job 101109 AssertionError: only silence detected, no useful audio data
    • job 101758 AssertionError: too short audio, expected 10s, got 9.36446712018140...
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_223_audio_play_hvm (2/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
    • job 102413 AssertionError: only silence detected, no useful audio data
    TC_20_AudioVM_Pulse_whonix-workstation-17/test_252_audio_playback_audiovm_switch_hvm (1/4 times with errors)
    • job 101109 AssertionError: Command 'timeout 20s paplay --format=float32le --ra...
  • system_tests_suspend@hw1

    suspend/ (3/5 times with errors)
    suspend/Failed (3/5 times with errors)
    • job 102431 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 102443 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    • job 103660 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/wait_serial (3/5 times with errors)
    • job 102431 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 102443 # wait_serial expected: qr/2E8vz-\d+-/...
    • job 103660 # wait_serial expected: qr/2E8vz-\d+-/...

Previously the expected_ret_codes applied to both qubes-vm-update and
later check if a package was updated (`rpm -q|grep ...` etc). With
unified qubes-vm-update exit codes, this approach is too broad as
qubes-vm-update is now expected to exit with very specific error, not
just "any error" - and this doesn't match grep's exit code anymore.
Fix this by adding another parameter for expected grep result (whether
updated package is expected or not).
@marmarek marmarek merged commit 212ff69 into QubesOS:main Jul 5, 2024
3 of 4 checks passed
@marmarek marmarek deleted the tests20240703 branch October 8, 2024 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants