bhyve VM cannot be immediately destroyed

[kwitaszczyk]

While developing changes in the kernel, I sporadically encounter the case that my kernel freezes due to some bug and sysctl hw.vmm.destroy=foo hangs instead of immediately destroying the VM. My host is:

FreeBSD stevnsbaer 15.0-CURRENT FreeBSD 15.0-CURRENT #0 dev-n274544-1fe5d6df7242: Tue May 20 15:43:55 BST 2025     root@stevnsbaer:/usr/obj/usr/src/arm64.aarch64c/sys/GENERIC-MORELLO-PURECAP arm64

Kernel backtraces of threads of the bhyve process handling the VM when the sysctl is hanging are:

  PID    TID COMM                TDNAME              KSTACK                       
44748 101744 bhyve               mevent              mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 kqueue_kevent+0x9f0 kern_kevent_fp+0xb8 kern_kevent+0x98 kern_kevent_generic+0x7c sys_kevent+0x80 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175076 bhyve               blk-2:0-0           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175077 bhyve               blk-2:0-1           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175078 bhyve               blk-2:0-2           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175079 bhyve               blk-2:0-3           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175080 bhyve               blk-2:0-4           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175081 bhyve               blk-2:0-5           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175082 bhyve               blk-2:0-6           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175083 bhyve               blk-2:0-7           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175084 bhyve               slirp pollfd        mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_timedwait_sig+0x18 _cv_timedwait_sig_sbt+0x1b4 seltdwait+0xa0 kern_poll_kfds+0x43c kern_poll+0xc0 sys_poll+0x68 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175085 bhyve               vtnet-3:0 tx        mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175086 bhyve               blk-4:0-0           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175087 bhyve               blk-4:0-1           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175088 bhyve               blk-4:0-2           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175089 bhyve               blk-4:0-3           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175090 bhyve               blk-4:0-4           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175091 bhyve               blk-4:0-5           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175092 bhyve               blk-4:0-6           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175093 bhyve               blk-4:0-7           mi_switch+0x17c sleepq_switch+0x114 sleepq_catch_signals+0x2c0 sleepq_wait_sig+0x10 _sleep+0x2c4 umtxq_sleep+0x300 do_wait+0x2a0 __umtx_op_wait_uint_private+0x6c sys__umtx_op+0xb4 do_el0_sync+0x5bc handle_el0_sync+0x34 
44748 175094 bhyve               vcpu 0              mi_switch+0x17c sleepq_switch+0x114 sleepq_timedwait+0x4c msleep_spin_sbt+0x154 vm_run+0x6e0 vmmdev_machdep_ioctl+0x130 vmmdev_ioctl+0x900 devfs_ioctl+0xe4 vn_ioctl+0xd8 devfs_ioctl_f+0x2c kern_ioctl+0x35c user_ioctl+0x178 do_el0_sync+0x5bc handle_el0_sync+0x34 

I do have an image that I can use to reproduce this and I will update this message once it's uploaded.

You can reproduce this using an example minimal image. I run the VM using:

sudo bhyve -G 1235 -m 1G -c 1 -s 0,hostbridge -s 2,virtio-blk,/zdata/cheri/output/repos/cheribsd/kernel-c18n/morello-purecap-minimal.img -s 3,virtio-net,slirp,hostfwd=tcp::1237-:22 -s 4,virtio-blk,/zdata/projects/kernel-c18n/zpool.img -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin -o console=stdio kernel-c18n-minimal

[markjdb]

This bug exists on amd64 too, but it is mitigated by the hw.vmm.halt_detection tunable: when that tunable is set to 1 (the default), if all cores execute hlt with interrupts disabled, the VM automatically exits.

No similar mechanism is implemented for arm64. But this shouldn't hang in any case.

vmmdev_lookup_and_destroy() should somehow signal threads blocked in vm_handle_wfi()/vm_handle_hlt() and ensure they don't block again before calling destroy_dev().

[markjdb]

This is fixed by https://reviews.freebsd.org/D51761