netkvm: Control message timeout causes corrupt VQ

[eelbaz1]

Control messages sent to device when link disconnected.
When link reconnect, device reads corrupted messages, responds with errors.

[0]0EF8.0F0C::04/06/2025-11:07:07.255 [netkvm.sys] PostLinkStateIndicating Disconnected
[1]0004.0144::04/06/2025-11:07:07.256 [netkvm.sys] ParaNdis_OnOidSetNetworkAddressesaddress 0, type 3, len 28 (0000000000000000FE80000000000000FC9D26FFFE79127204000020)
[0]0004.00C8::04/06/2025-11:07:07.256 [netkvm.sys] ParaNdis_OnOidSetNetworkAddressesReceived IP address 169.254.209.116
[0]0004.00D0::04/06/2025-11:07:10.552 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: get_buf failed
[1]0004.00D0::04/06/2025-11:07:13.475 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: get_buf failed
[0]0004.00D0::04/06/2025-11:07:14.150 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00D0::04/06/2025-11:07:14.312 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00D0::04/06/2025-11:07:14.312 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00D0::04/06/2025-11:07:14.312 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: error 1 returned for class 1
[0]0004.00D0::04/06/2025-11:07:14.313 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00D0::04/06/2025-11:07:14.313 [netkvm.sys] PostLinkStateIndicating Connected
[0]06D0.07AC::04/06/2025-11:07:14.326 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00C8::04/06/2025-11:07:15.514 [netkvm.sys] ParaNdis_OnOidSetNetworkAddressesaddress 0, type 3, len 28 (0000000000000000FE80000000000000FC9D26FFFE79127204000020)
[1]0004.00C0::04/06/2025-11:07:15.532 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: error 1 returned for class 0
[1]0004.00C0::04/06/2025-11:07:15.532 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: error 1 returned for class 0
[1]0004.00C0::04/06/2025-11:07:15.532 [netkvm.sys] CParaNdisCX::SendControlMessageCParaNdisCX::SendControlMessage - ERROR: error 1 returned for class 1
[0]0000.0000::04/06/2025-11:07:15.532 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0004.00D0::04/06/2025-11:07:15.532 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[0]0000.0000::04/06/2025-11:07:15.533 [netkvm.sys] CParaNdisCX::FireDPC[CParaNdisCX::FireDPC] message 4
[1]0004.00D0::04/06/2025-11:07:17.514 [netkvm.sys] ParaNdis_OnOidSetNetworkAddressesReceived IP address 169.254.209.116

Reproduce: unplug + plug cable
netkvm-ctrl-msg-send-when-unpluged.log

VQ corruption can also be caused by device slow response to control messages (more then 0.5 seconds)
netkvm-ctrl-msg-send-timout.log

Host:

• Disto: Ubuntu 22.04 LTS
• Kernel version: 5.15.0-131-generic
• QEMU version: 6.2.0

QEMU command line

/usr/bin/qemu-system-x86_64 \
-name guest=Win10_Ent_2019_LTSC_x64,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-Wi/master-key.aes"}' \
-machine pc-i440fx-jammy,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu 'core2duo,ibpb=on,md-clear=on,spec-ctrl=on,ssbd=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=NVP Hv' \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 3642cec1-17a1-4a13-b35f-f115fd121609 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x5 \
-blockdev '{"driver":"file","filename":"/images/004.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device ide-hd,bus=sata0.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=1 \
-netdev tap,fd=34,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=13:63:59:15:40:17,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 127.0.0.1:0,audiodev=audio1 \
-device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 \
-device vfio-pci,host=0000:0f:04.4,id=hostdev0,bus=pci.0,addr=0x7 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

• libvirt version: 8.0.0
• libvirt XML file:

libvirt XML file

<domain type='kvm'>
  <name>Win10_Ent_2019_LTSC_x64</name>
  <uuid>3642cec1-17a1-4a13-b35f-f115fd121609</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-jammy'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='NVP Hv'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>core2duo</model>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='spec-ctrl'/>
    <feature policy='require' name='ssbd'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/images/004.img'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='13:63:59:15:40:17'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x04' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

VM:

• Windows version: Windows 10 Enterprise LTSC 1809 build 17763.6293
• Which driver has a problem: netkvm
• Driver version or commit hash that was used to build the driver: 0.1.266 (100.100.104.26600)

[YanVugenfirer]

@eelbaz1 Can you please provide additional information on the scenario and the issues that you experience

1. Are you using a HW device or toggling link state with QEMU monitor?
2. What is the NIC behavior other than a control queue error messages?

[eelbaz1]

@eelbaz1 Can you please provide additional information on the scenario and the issues that you experience

1. Are you using a HW device or toggling link state with QEMU monitor?
2. What is the NIC behavior other than a control queue error messages?

1. using HW device VF (physical link)
2. the driver continue to work after the errors when device is back up, the state of the Ctrl VQ remain corrupted (further messages may result with errors as well).

[YanVugenfirer]

I think the problem is with the HW. It stops delivering CQ messages during link down (tx, Rx queues should stop, but CQ should work).

Are you a vendor engineer or a user?

Btw: interesting experiment is to disable the driver, take the cable out and then enable the driver

[eelbaz1]

I think the problem is with the HW. It stops delivering CQ messages during link down (tx, Rx queues should stop, but CQ should work).

Are you a vendor engineer or a user?

Btw: interesting experiment is to disable the driver, take the cable out and then enable the driver

i'm an engineer, we got to test the same flow on the linux driver as well and there we don't see this issue.
linux driver does not send control messages when device is unplug (and block configs until device is plugged)

i think the main issue is that some delays in device response causes communication with driver to break

*for the experiment, all control messages just time out

[YanVugenfirer]

On init the driver must send messages , even if link is unplugged.
I need to double check the spec.

[YanVugenfirer]

Anyway, there is no reason for hw not to proceed control queue when link is down. Especially during device initialization

[YanVugenfirer]

From the brief overview I don’t see in the spec that the control queue processing should be blocked by HW.

I suggest to run the experiment on Linux when booting device with link down and then checking if the device got configured for advanced features like filtering or if the locally administered MAC address was configured correctly

[YanVugenfirer]

Delayed response can indeed cause a problem

kvm-guest-drivers-windows/NetKVM/Common/ParaNdis_CX.cpp

Line 93 in b7b94a0

if (0 <= m_VirtQueue.AddBuf(sg, nOut, 1, (PVOID)1, NULL, 0))

What are the other case, other than link down, you experience delay?

[YanVugenfirer]

Btw: why would a device respond for more than 0.5 seconds? Is it FPGA?