BDF port gets into bad state - MTL fails to init

[cwhite102]

As requested, this issue has been split off from #1176 (lcore management gets into a bad state) as it appears to be a different issue (just with similar symptoms).

In this issue, the VFIO BDF port itself (or a data-structure related to it) gets into a bad state, and attempting to initialize MTL in a new process that uses the troubled port fails to initialize.
When in this state (unlike #1176), changing the lcores list doesn't help, as it is the port itself which is problematic.
A reboot restores normal functionality. (or switching to a different BDF port)

We're not sure what leads to this situation, or if there's a way to resolve it other than a reboot.
We start up multiple processes at once, each with their own uniquely assigned BDF / VFIO port.

Any guidance, experiments, or commands to help get to the root cause (and solution) would be appreciated.

Here's a log of attempting to initialize MTL on a port that is in this bad state:
(Note - these logs are from launching a new, single MTL process with no other MTL processes running)
(MTL main branch - I believe this includes commits up to July 3rd)

07-05 04:25:53.890 INFO Intel2110Config_GLOBAL [Blueprint-GraphStateNotifier-2]: /: Initializing MTL... 
07-05 04:25:53.892 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: Logging thread started 
07-05 04:25:53.893 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init(0), port_param: 0000:83:01.2 
07-05 04:25:53.893 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init, wait eal_init_thread done 
07-05 04:25:54.347 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL version: 25.2.0.DEV Thu Jul  3 11:02:26 2025 e285345a gcc-11.4.0, dpdk version: DPDK 25.03.0 
07-05 04:25:54.347 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL_HAS_USDT is defined for this build 
07-05 04:25:54.347 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_instance_init, connect to manager fail, assume single instance mode 
07-05 04:25:54.347 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init(0), socket_id 0 port 0000:83:01.2 
07-05 04:25:54.348 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_stat_init, stat period 10s 
07-05 04:25:54.348 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), use user ptp source 
07-05 04:25:54.348 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), user request queues tx 11 rx 40 
07-05 04:25:54.348 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), deprecated sessions tx 0 rx 0 
07-05 04:25:54.348 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: stat_thread, start 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: dev_config_port(0), rte_eth_dev_configure fail -1 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: mt_dev_if_init(0), dev_config_port fail -1 
07-05 04:25:54.698 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_close_port(0), port not started 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: mtl_init, st dev if init fail -5 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9761742a0 priv 0x11802b6680 not found 
07-05 04:25:54.698 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_cni_uinit, no cni for all ports 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: sch_lcore_shm_uinit, can not stat shared memory, Invalid argument 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802bf738 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802c6e58 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802ce578 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802d5c98 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802dd3b8 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802e4ad8 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802ec1f8 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802f3918 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11802fb038 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180302758 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180309e78 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180311598 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180318cb8 not found 
07-05 04:25:54.698 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x11803203d8 not found 
07-05 04:25:54.699 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180327af8 not found 
07-05 04:25:54.699 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x118032f218 not found 
07-05 04:25:54.699 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x1180336938 not found 
07-05 04:25:54.699 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760a71c0 priv 0x118033e058 not found 
07-05 04:25:54.699 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_stat_unregister, cb 0x7fe9760e5600 priv 0x11802b6680 not found 
07-05 04:25:54.699 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_stop_port(0), port not started 
07-05 04:25:54.699 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_free, succ 
07-05 04:25:54.699 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_main_free, succ 
07-05 04:25:54.699 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_close_port(0), port not started 
07-05 04:25:54.699 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: stat_thread, stop 
07-05 04:25:54.700 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_uinit, succ 
07-05 04:25:54.700 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_uninit, succ 
07-05 04:25:54.700 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: mtl_init: mtl_init fail 
07-05 04:25:54.700 SEVERE Intel2110Config_GLOBAL [Blueprint-GraphStateNotifier-2]: /: mtl_init fail -1

With the same settings, switching to the next BDF port works, and has output as follows:

07-05 04:27:43.955 INFO Intel2110Config_GLOBAL [Blueprint-GraphStateNotifier-2]: /: Initializing MTL... 
07-05 04:27:43.957 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: Logging thread started 
07-05 04:27:43.957 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init(0), port_param: 0000:83:01.3 
07-05 04:27:43.957 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init, wait eal_init_thread done 
07-05 04:27:44.456 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL version: 25.2.0.DEV Thu Jul  3 11:02:26 2025 e285345a gcc-11.4.0, dpdk version: DPDK 25.03.0 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL_HAS_USDT is defined for this build 
07-05 04:27:44.457 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_instance_init, connect to manager fail, assume single instance mode 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init(0), socket_id 0 port 0000:83:01.3 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_stat_init, stat period 10s 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), use user ptp source 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), user request queues tx 11 rx 40 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), deprecated sessions tx 0 rx 0 
07-05 04:27:44.457 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: stat_thread, start 
07-05 04:27:44.594 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_config_port(0), tx_q(41 with 512 desc) rx_q (41 with 2048 desc) 
07-05 04:27:44.595 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11804fdf00 size 4.310394m n 2047 d 2048 for T_P0_SYS_0 
07-05 04:27:44.595 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_if_init_tx_queues(0), tx_queues 41 malloc succ 
07-05 04:27:44.596 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118061da80 size 8.622894m n 4095 d 2048 for R_P0Q0_MBUF_1 
07-05 04:27:44.597 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11816fdf00 size 6.623383m n 4095 d 1536 for R_P0Q1_MBUF_2 
07-05 04:27:44.598 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11820fdf00 size 6.623383m n 4095 d 1536 for R_P0Q2_MBUF_3 
...

Some command output in case it's useful... (note - we create 16 vfio devices on each port)

imagine@tor-dl345-3:/opt/ZeniumUtils$ dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:83:01.0 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.1 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.2 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.3 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.4 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.5 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.6 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:01.7 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.0 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.1 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.2 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.3 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.4 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.5 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.6 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:02.7 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.0 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.1 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.2 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.3 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.4 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.5 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.6 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:11.7 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.0 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.1 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.2 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.3 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.4 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.5 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.6 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf
0000:83:12.7 'Ethernet Adaptive Virtual Function 1889' numa_node=0 drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:43:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' numa_node=0 if=ens22f0 drv=tg3 unused=vfio-pci *Active*
0000:43:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' numa_node=0 if=ens22f1 drv=tg3 unused=vfio-pci 
0000:43:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' numa_node=0 if=ens22f2 drv=tg3 unused=vfio-pci 
0000:43:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' numa_node=0 if=ens22f3 drv=tg3 unused=vfio-pci 
0000:83:00.0 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=ens3f0 drv=ice unused=vfio-pci *Active*
0000:83:00.1 'Ethernet Controller E810-XXV for SFP 159b' numa_node=0 if=ens3f1 drv=ice unused=vfio-pci *Active*
0000:c8:00.0 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' numa_node=0 if=ens21f0np0 drv=bnxt_en unused=vfio-pci 
0000:c8:00.1 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller 16d7' numa_node=0 if=ens21f1np1 drv=bnxt_en unused=vfio-pci 




sudo lshw -c network -businfo
Bus info          Device      Class          Description
========================================================
pci@0000:43:00.0  ens22f0     network        NetXtreme BCM5719 Gigabit Ethernet PCIe
pci@0000:43:00.1  ens22f1     network        NetXtreme BCM5719 Gigabit Ethernet PCIe
pci@0000:43:00.2  ens22f2     network        NetXtreme BCM5719 Gigabit Ethernet PCIe
pci@0000:43:00.3  ens22f3     network        NetXtreme BCM5719 Gigabit Ethernet PCIe
pci@0000:83:00.0  ens3f0      network        Ethernet Controller E810-XXV for SFP
pci@0000:83:00.1  ens3f1      network        Ethernet Controller E810-XXV for SFP
pci@0000:83:01.0              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.1              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.2              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.3              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.4              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.5              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.6              network        Ethernet Adaptive Virtual Function
pci@0000:83:01.7              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.0              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.1              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.2              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.3              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.4              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.5              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.6              network        Ethernet Adaptive Virtual Function
pci@0000:83:02.7              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.0              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.1              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.2              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.3              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.4              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.5              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.6              network        Ethernet Adaptive Virtual Function
pci@0000:83:11.7              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.0              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.1              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.2              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.3              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.4              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.5              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.6              network        Ethernet Adaptive Virtual Function
pci@0000:83:12.7              network        Ethernet Adaptive Virtual Function
pci@0000:c8:00.0  ens21f0np0  network        BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
pci@0000:c8:00.1  ens21f1np1  network        BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller

[cwhite102]

In case this is useful, I tailed /var/log/kern.log when starting the failing process.

This is a log of the MTL init starting up and failing. BDF for VF 2 is in a bad state. No other MTL processes were currently running for this log.

Jul  8 00:48:38 tor-dl345-3 kernel: [243212.410133] vfio-pci 0000:83:01.2: enabling device (0000 -> 0002)
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.654298] vfio-pci 0000:83:11.2: enabling device (0000 -> 0002)
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.668899] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.668907] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.768874] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.768883] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul  8 00:48:38 tor-dl345-3 kernel: [243212.952656] ice 0000:83:00.0: VF 2 granted request of 41 queues.
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.109493] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.109502] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.134110] ice 0000:83:00.1: VF 2 granted request of 41 queues.
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.239944] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.239960] ice 0000:83:00.1: PF failed to honor VF 2, opcode 24, error -22
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.303295] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.303305] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul  8 00:48:38 tor-dl345-3 kernel: [243213.342133] ice 0000:83:00.0: VF 2 granted request of 16 queues.
Jul  8 00:48:39 tor-dl345-3 kernel: [243213.385700] ice 0000:83:00.1: VF 2 granted request of 16 queues.

Contrast this to a successful startup using VF 1 ...

Jul  8 00:48:02 tor-dl345-3 kernel: [243177.015855] vfio-pci 0000:83:01.1: enabling device (0000 -> 0002)
Jul  8 00:48:02 tor-dl345-3 kernel: [243177.276604] vfio-pci 0000:83:11.1: enabling device (0000 -> 0002)
Jul  8 00:48:03 tor-dl345-3 kernel: [243177.576367] ice 0000:83:00.0: VF 1 granted request of 41 queues.
Jul  8 00:48:03 tor-dl345-3 kernel: [243177.795635] ice 0000:83:00.1: VF 1 granted request of 41 queues.
Jul  8 00:48:03 tor-dl345-3 kernel: [243178.066588] ice 0000:83:00.0: Unprivileged VF 1 is attempting to configure promiscuous mode
Jul  8 00:48:03 tor-dl345-3 kernel: [243178.067649] ice 0000:83:00.0: Unprivileged VF 1 is attempting to configure promiscuous mode
Jul  8 00:48:03 tor-dl345-3 kernel: [243178.348237] ice 0000:83:00.1: Unprivileged VF 1 is attempting to configure promiscuous mode
Jul  8 00:48:03 tor-dl345-3 kernel: [243178.349290] ice 0000:83:00.1: Unprivileged VF 1 is attempting to configure promiscuous mode

[tszumski]

Thanks for extended logs, it looks like there are some errors at ice-driver/firmware level, can you execute ethtool -i <interface> to get more information about software/firmware versions ?

[andremiszcz]

Can you also exeucte lspci -nn -vvv | grep Ethernet command to get subsystem ID details about network adapter?

[cwhite102]

Here's the output of those two commands as requested.
Please note - the system is NOT in a bad state at the moment, and the VFs have recently been recreated in trusted mode as per my investigation on issue #1182

ethtool -i ens3f0

driver: ice
version: Kahawai_1.14.9_20240613
firmware-version: 4.40 0x8001c96c 1.3534.0
expansion-rom-version: 
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

lspci -nn -vvv | grep Ethernet

43:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1590]
43:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1590]
43:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1590]
43:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1590]
83:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller E810-XXV for SFP [8086:159b] (rev 02)
	Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-2 [8086:0003]
		Product Name: Intel(R) Ethernet Network Adapter E810-XXVDA2
			[V1] Vendor specific: Intel(R) Ethernet Network Adapter E810-XXVDA2
83:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller E810-XXV for SFP [8086:159b] (rev 02)
	Subsystem: Intel Corporation Ethernet Network Adapter E810-XXV-2 [8086:0003]
		Product Name: Intel(R) Ethernet Network Adapter E810-XXVDA2
			[V1] Vendor specific: Intel(R) Ethernet Network Adapter E810-XXVDA2
83:01.0 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.1 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.2 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.3 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.4 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.5 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.6 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:01.7 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.0 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.1 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.2 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.3 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.4 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.5 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.6 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:02.7 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.0 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.1 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.2 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.3 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.4 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.5 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.6 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:11.7 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.0 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.1 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.2 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.3 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.4 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.5 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.6 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
83:12.7 Ethernet controller [0200]: Intel Corporation Ethernet Adaptive Virtual Function [8086:1889] (rev 02)
	Subsystem: Intel Corporation Ethernet Adaptive Virtual Function [8086:0000]
c8:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:1596]
c8:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:16d7] (rev 01)
	Subsystem: Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller [14e4:1596]

[tszumski]

Thanks for the update, I cannot reproduce your issue, as recommendation can you update ice-drivers to 1.16.3 and update firmware to latest (30.2) based on instruction here https://github.com/OpenVisualCloud/Media-Transport-Library/blob/main/doc/e810.md

[cwhite102]

Update...

ICE driver now at 1.16.3
Firmware hasn't been updated yet.

ethtool -i ens3f0

driver: ice
version: Kahawai_1.16.3_20250327
firmware-version: 4.40 0x8001c96c 1.3534.0
expansion-rom-version:
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Fresh reboot, first launch of 9 MTL processes (concurrently) went fine.
Stopped the processes.
Launched again, one failed to init (CH-E on VF 5).

kern.log

Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.109555] ice 0000:83:00.0: VF 8 granted request of 16 queues.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.127731] ice 0000:83:00.0: Promoting VSI 14 to GLOBAL LUT.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.185674] ice 0000:83:00.1: VF 8 granted request of 16 queues.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.205751] ice 0000:83:00.1: Promoting VSI 14 to GLOBAL LUT.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.297529] ice 0000:83:00.1: VF 2 granted request of 16 queues.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.319261] ice 0000:83:00.0: VF 2 granted request of 16 queues.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.479161] ice 0000:83:00.0: Promoting VSI 2 to GLOBAL LUT.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.582214] ice 0000:83:00.1: Promoting VSI 2 to GLOBAL LUT.
Jul 17 17:50:44 tor-dl345-3 kernel: [ 1259.605015] ice 0000:83:00.0: VF 0 granted request of 16 queues.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1259.869908] ice 0000:83:00.1: VF 0 granted request of 16 queues.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1259.870514] ice 0000:83:00.0: Promoting VSI 6 to GLOBAL LUT.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1260.030062] ice 0000:83:00.0: VF 6 granted request of 16 queues.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1260.215316] ice 0000:83:00.1: Promoting VSI 6 to GLOBAL LUT.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1260.552835] ice 0000:83:00.1: VF 6 granted request of 16 queues.
Jul 17 17:50:45 tor-dl345-3 kernel: [ 1260.670473] ice 0000:83:00.0: Promoting VSI 10 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1260.789995] ice 0000:83:00.0: VF 4 granted request of 16 queues.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1260.834524] ice 0000:83:00.0: Promoting VSI 12 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.017328] ice 0000:83:00.1: Promoting VSI 10 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.058484] ice 0000:83:00.0: VF 7 granted request of 16 queues.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.077514] ice 0000:83:00.0: Promoting VSI 16 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.097503] ice 0000:83:00.1: VF 4 granted request of 16 queues.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.156243] ice 0000:83:00.1: Promoting VSI 4 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.318012] ice 0000:83:00.1: Promoting VSI 8 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.318884] ice 0000:83:00.0: VF 5 granted request of 16 queues.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.377914] ice 0000:83:00.0: Promoting VSI 4 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.458330] ice 0000:83:00.1: Promoting VSI 12 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.517166] ice 0000:83:00.1: Promoting VSI 16 to GLOBAL LUT.
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.518952] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.518957] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 17 17:50:46 tor-dl345-3 kernel: [ 1261.679693] ice 0000:83:00.0: Promoting VSI 8 to GLOBAL LUT.
Jul 17 17:50:57 tor-dl345-3 kernel: [ 1272.309702] vfio-pci 0000:83:01.0: enabling device (0000 -> 0002)
Jul 17 17:50:57 tor-dl345-3 kernel: [ 1272.566883] vfio-pci 0000:83:11.0: enabling device (0000 -> 0002)
Jul 17 17:50:58 tor-dl345-3 kernel: [ 1272.888737] ice 0000:83:00.0: VF 0 granted request of 41 queues.
Jul 17 17:50:58 tor-dl345-3 kernel: [ 1273.070520] ice 0000:83:00.1: VF 0 granted request of 41 queues.
Jul 17 17:51:01 tor-dl345-3 kernel: [ 1276.263433] vfio-pci 0000:83:01.7: enabling device (0000 -> 0002)
Jul 17 17:51:01 tor-dl345-3 kernel: [ 1276.374374] vfio-pci 0000:83:01.3: enabling device (0000 -> 0002)
Jul 17 17:51:01 tor-dl345-3 kernel: [ 1276.482400] vfio-pci 0000:83:01.5: enabling device (0000 -> 0002)
Jul 17 17:51:01 tor-dl345-3 kernel: [ 1276.590405] vfio-pci 0000:83:02.0: enabling device (0000 -> 0002)
Jul 17 17:51:01 tor-dl345-3 kernel: [ 1276.698385] vfio-pci 0000:83:01.4: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1276.806366] vfio-pci 0000:83:11.7: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1276.914376] vfio-pci 0000:83:01.6: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.022373] vfio-pci 0000:83:01.1: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.130385] vfio-pci 0000:83:11.3: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.163328] ice 0000:83:00.0: VF 7 granted request of 41 queues.
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.238365] vfio-pci 0000:83:11.5: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.312171] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.312178] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.346384] vfio-pci 0000:83:12.0: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.395953] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.395958] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.454366] vfio-pci 0000:83:01.2: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.562373] vfio-pci 0000:83:11.4: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.656692] ice 0000:83:00.0: VF 3 granted request of 41 queues.
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.670378] vfio-pci 0000:83:11.6: enabling device (0000 -> 0002)
Jul 17 17:51:02 tor-dl345-3 kernel: [ 1277.699032] ice 0000:83:00.1: VF 7 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1277.778373] vfio-pci 0000:83:11.1: enabling device (0000 -> 0002)
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1277.856510] ice 0000:83:00.0: VF 5 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.018201] vfio-pci 0000:83:11.2: enabling device (0000 -> 0002)
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.076592] ice 0000:83:00.1: VF 3 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.100399] ice 0000:83:00.0: VF 8 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.198479] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.198485] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.307028] ice 0000:83:00.1: VF 5 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.408286] ice 0000:83:00.1: VF 8 granted request of 41 queues.
Jul 17 17:51:03 tor-dl345-3 kernel: [ 1278.507745] ice 0000:83:00.0: VF 4 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1278.808033] ice 0000:83:00.0: VF 6 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1278.930169] ice 0000:83:00.0: VF 1 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.010740] ice 0000:83:00.1: VF 4 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.026941] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.026954] ice 0000:83:00.1: PF failed to honor VF 5, opcode 24, error -22
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.030636] ice 0000:83:00.0: VF 2 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.311580] ice 0000:83:00.0: VF 5 granted request of 16 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.370649] ice 0000:83:00.1: VF 5 granted request of 16 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.735508] ice 0000:83:00.1: VF 6 granted request of 41 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.896472] ice 0000:83:00.1: VF 1 granted request of 41 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.940741] ice 0000:83:00.1: VF 2 granted request of 41 queues.

application log

07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: Logging thread started 
07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init(0), port_param: 0000:83:01.5 
07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init(1), port_param: 0000:83:11.5 
07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init, main_lcore: 40 
07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init, lcores: 41,
42,
43 
07-17 17:51:01.523 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_eal_init, wait eal_init_thread done 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL version: 25.2.0.DEV Thu Jul 17 12:20:51 2025 be6e413b gcc-11.4.0, dpdk version: DPDK 25.03.0 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init, MTL_HAS_USDT is defined for this build 
07-17 17:51:02.994 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Warn: mt_instance_init, connect to manager fail, assume single instance mode 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init(0), socket_id 0 port 0000:83:01.5 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mtl_init(1), socket_id 0 port 0000:83:11.5 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_stat_init, stat period 10s 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), use user ptp source 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), user request queues tx 11 rx 40 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), deprecated sessions tx 0 rx 0 
07-17 17:51:02.994 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: stat_thread, start 
07-17 17:51:03.413 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_config_port(0), tx_q(41 with 512 desc) rx_q (41 with 2048 desc) 
07-17 17:51:03.414 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11804fdf00 size 4.310394m n 2047 d 2048 for T_P0_SYS_0 
07-17 17:51:03.414 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_if_init_tx_queues(0), tx_queues 41 malloc succ 
07-17 17:51:03.415 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118061da80 size 8.622894m n 4095 d 2048 for R_P0Q0_MBUF_1 
07-17 17:51:03.416 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11816fdf00 size 6.623383m n 4095 d 1536 for R_P0Q1_MBUF_2 
07-17 17:51:03.417 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11820fdf00 size 6.623383m n 4095 d 1536 for R_P0Q2_MBUF_3 
07-17 17:51:03.418 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1182afdf00 size 6.623383m n 4095 d 1536 for R_P0Q3_MBUF_4 
07-17 17:51:03.419 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11834fdf00 size 6.623383m n 4095 d 1536 for R_P0Q4_MBUF_5 
07-17 17:51:03.420 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1183efdf00 size 6.623383m n 4095 d 1536 for R_P0Q5_MBUF_6 
07-17 17:51:03.421 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11848fdf00 size 6.623383m n 4095 d 1536 for R_P0Q6_MBUF_7 
07-17 17:51:03.422 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11852fdf00 size 6.623383m n 4095 d 1536 for R_P0Q7_MBUF_8 
07-17 17:51:03.423 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1185cfdf00 size 6.623383m n 4095 d 1536 for R_P0Q8_MBUF_9 
07-17 17:51:03.424 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11866fdf00 size 6.623383m n 4095 d 1536 for R_P0Q9_MBUF_10 
07-17 17:51:03.425 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11870fdf00 size 6.623383m n 4095 d 1536 for R_P0Q10_MBUF_11 
07-17 17:51:03.426 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1187afdf00 size 6.623383m n 4095 d 1536 for R_P0Q11_MBUF_12 
07-17 17:51:03.427 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11884fdf00 size 6.623383m n 4095 d 1536 for R_P0Q12_MBUF_13 
07-17 17:51:03.428 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1188efdf00 size 6.623383m n 4095 d 1536 for R_P0Q13_MBUF_14 
07-17 17:51:03.429 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11898fdf00 size 6.623383m n 4095 d 1536 for R_P0Q14_MBUF_15 
07-17 17:51:03.430 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118a2fdf00 size 6.623383m n 4095 d 1536 for R_P0Q15_MBUF_16 
07-17 17:51:03.431 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118acfdf00 size 6.623383m n 4095 d 1536 for R_P0Q16_MBUF_17 
07-17 17:51:03.432 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118b6fdf00 size 6.623383m n 4095 d 1536 for R_P0Q17_MBUF_18 
07-17 17:51:03.433 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118c0fdf00 size 6.623383m n 4095 d 1536 for R_P0Q18_MBUF_19 
07-17 17:51:03.434 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118cafdf00 size 6.623383m n 4095 d 1536 for R_P0Q19_MBUF_20 
07-17 17:51:03.435 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118d4fdf00 size 6.623383m n 4095 d 1536 for R_P0Q20_MBUF_21 
07-17 17:51:03.436 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118defdf00 size 6.623383m n 4095 d 1536 for R_P0Q21_MBUF_22 
07-17 17:51:03.437 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118e8fdf00 size 6.623383m n 4095 d 1536 for R_P0Q22_MBUF_23 
07-17 17:51:03.438 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118f2fdf00 size 6.623383m n 4095 d 1536 for R_P0Q23_MBUF_24 
07-17 17:51:03.439 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x118fcfdf00 size 6.623383m n 4095 d 1536 for R_P0Q24_MBUF_25 
07-17 17:51:03.440 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11906fdf00 size 6.623383m n 4095 d 1536 for R_P0Q25_MBUF_26 
07-17 17:51:03.441 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11910fdf00 size 6.623383m n 4095 d 1536 for R_P0Q26_MBUF_27 
07-17 17:51:03.442 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1191afdf00 size 6.623383m n 4095 d 1536 for R_P0Q27_MBUF_28 
07-17 17:51:03.443 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11924fdf00 size 6.623383m n 4095 d 1536 for R_P0Q28_MBUF_29 
07-17 17:51:03.444 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1192efdf00 size 6.623383m n 4095 d 1536 for R_P0Q29_MBUF_30 
07-17 17:51:03.445 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11938fdf00 size 6.623383m n 4095 d 1536 for R_P0Q30_MBUF_31 
07-17 17:51:03.446 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11942fdf00 size 6.623383m n 4095 d 1536 for R_P0Q31_MBUF_32 
07-17 17:51:03.447 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1194cfdf00 size 6.623383m n 4095 d 1536 for R_P0Q32_MBUF_33 
07-17 17:51:03.448 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11956fdf00 size 6.623383m n 4095 d 1536 for R_P0Q33_MBUF_34 
07-17 17:51:03.449 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11960fdf00 size 6.623383m n 4095 d 1536 for R_P0Q34_MBUF_35 
07-17 17:51:03.450 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1196afdf00 size 6.623383m n 4095 d 1536 for R_P0Q35_MBUF_36 
07-17 17:51:03.450 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11974fdf00 size 6.623383m n 4095 d 1536 for R_P0Q36_MBUF_37 
07-17 17:51:03.451 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1197efdf00 size 6.623383m n 4095 d 1536 for R_P0Q37_MBUF_38 
07-17 17:51:03.452 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11988fdf00 size 6.623383m n 4095 d 1536 for R_P0Q38_MBUF_39 
07-17 17:51:03.453 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x11992fdf00 size 6.623383m n 4095 d 1536 for R_P0Q39_MBUF_40 
07-17 17:51:03.454 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_create_by_ops(0), succ at 0x1199cfdf00 size 6.623383m n 4095 d 1536 for R_P0Q40_MBUF_41 
07-17 17:51:03.454 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: dev_if_init_rx_queues(0), rx_queues 41 malloc succ 
07-17 17:51:03.454 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), port_id 0 port_type 1 drv_type 4 
07-17 17:51:03.454 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), dev_capa 0x0, offload 0x119fbf:0x9666f queue offload 0x0:0x0, rss : 0x3ffc 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), system_rx_queues_end 1 hdr_split_rx_queues_end 1 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), sip: 10.12.168.12 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), netmask: 255.255.255.0 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), gateway: 0.0.0.0 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(0), mac: d2:36:ae:4e:ab:70 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(1), use user ptp source 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(1), user request queues tx 11 rx 40 
07-17 17:51:03.455 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_dev_if_init(1), deprecated sessions tx 0 rx 0 
07-17 17:51:04.322 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: dev_config_port(1), rte_eth_dev_configure fail -1 
07-17 17:51:04.322 WARNING Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: Error: mt_dev_if_init(1), dev_config_port fail -1 
07-17 17:51:04.322 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_free, free mempool R_P0Q0_MBUF_1 
07-17 17:51:04.322 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_free, free mempool R_P0Q1_MBUF_2 
07-17 17:51:04.323 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_free, free mempool R_P0Q2_MBUF_3 
07-17 17:51:04.324 INFO Intel2110Config_GLOBAL [Intel2110_Logging_BG-1]: /: MTL: mt_mempool_free, free mempool R_P0Q3_MBUF_4 
...

[cwhite102]

Another data point...
If I try and launch the failed process again right away, it will fail again.
But after some time, maybe several minutes, I've seen it successfully launch.

The failure above was able to launch 9 minutes later (without doing anything else... the other processes were still running)

kern.log (starting from the failure above, showing the new launch later)

Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.026941] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.026954] ice 0000:83:00.1: PF failed to honor VF 5, opcode 24, error -22
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.030636] ice 0000:83:00.0: VF 2 granted request of 41 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.311580] ice 0000:83:00.0: VF 5 granted request of 16 queues.
Jul 17 17:51:04 tor-dl345-3 kernel: [ 1279.370649] ice 0000:83:00.1: VF 5 granted request of 16 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.735508] ice 0000:83:00.1: VF 6 granted request of 41 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.896472] ice 0000:83:00.1: VF 1 granted request of 41 queues.
Jul 17 17:51:05 tor-dl345-3 kernel: [ 1279.940741] ice 0000:83:00.1: VF 2 granted request of 41 queues.
Jul 17 18:00:06 tor-dl345-3 kernel: [ 1820.874635] vfio-pci 0000:83:01.5: enabling device (0000 -> 0002)
Jul 17 18:00:06 tor-dl345-3 kernel: [ 1821.133946] vfio-pci 0000:83:11.5: enabling device (0000 -> 0002)
Jul 17 18:00:06 tor-dl345-3 kernel: [ 1821.429708] ice 0000:83:00.0: VF 5 granted request of 41 queues.
Jul 17 18:00:06 tor-dl345-3 kernel: [ 1821.607497] ice 0000:83:00.1: VF 5 granted request of 41 queues.

[soopel]

Thanks for update Craig. Please let us know if the issue persist as well with latest FW.

[cwhite102]

Updated firmware. Looks like the issue still persists.

ethtool -i ens3f0

driver: ice
version: Kahawai_1.16.3_20250327
firmware-version: 4.71 0x80020038 1.3755.0
expansion-rom-version:
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

kern.log

Jul 18 18:53:10 tor-dl345-3 kernel: [ 5063.181432] vfio-pci 0000:83:01.0: enabling device (0000 -> 0002)
Jul 18 18:53:10 tor-dl345-3 kernel: [ 5063.417662] vfio-pci 0000:83:11.0: enabling device (0000 -> 0002)
Jul 18 18:53:10 tor-dl345-3 kernel: [ 5063.675441] ice 0000:83:00.0: VF 0 granted request of 41 queues.
Jul 18 18:53:11 tor-dl345-3 kernel: [ 5063.762441] ice 0000:83:00.1: VF 0 granted request of 41 queues.
Jul 18 18:53:14 tor-dl345-3 kernel: [ 5067.399273] vfio-pci 0000:83:02.0: enabling device (0000 -> 0002)
Jul 18 18:53:14 tor-dl345-3 kernel: [ 5067.503918] vfio-pci 0000:83:01.3: enabling device (0000 -> 0002)
Jul 18 18:53:14 tor-dl345-3 kernel: [ 5067.611935] vfio-pci 0000:83:01.1: enabling device (0000 -> 0002)
Jul 18 18:53:14 tor-dl345-3 kernel: [ 5067.719960] vfio-pci 0000:83:01.4: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5067.827942] vfio-pci 0000:83:01.7: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5067.935942] vfio-pci 0000:83:01.6: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.043942] vfio-pci 0000:83:01.5: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.151942] vfio-pci 0000:83:01.2: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.259945] vfio-pci 0000:83:12.0: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.367927] vfio-pci 0000:83:11.3: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.475930] vfio-pci 0000:83:11.1: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.507408] ice 0000:83:00.0: VF 8 granted request of 41 queues.
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.583945] vfio-pci 0000:83:11.4: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.588358] ice 0000:83:00.1: VF 8 granted request of 41 queues.
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.623999] ice 0000:83:00.0: VF 3 granted request of 41 queues.
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.691933] vfio-pci 0000:83:11.7: enabling device (0000 -> 0002)
Jul 18 18:53:15 tor-dl345-3 kernel: [ 5068.711745] ice 0000:83:00.1: VF 3 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.759440] ice 0000:83:00.0: VF 1 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.799930] vfio-pci 0000:83:11.6: enabling device (0000 -> 0002)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.801806] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.801815] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.835446] ice 0000:83:00.0: VF 4 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.903903] ice 0000:83:00.1: VF 1 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.907942] vfio-pci 0000:83:11.5: enabling device (0000 -> 0002)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.915833] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.915840] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.952304] ice 0000:83:00.0: VF 7 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5068.977963] ice 0000:83:00.1: VF 4 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.015925] vfio-pci 0000:83:11.2: enabling device (0000 -> 0002)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.099585] ice 0000:83:00.1: VF 7 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.143911] ice 0000:83:00.0: VF 6 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.188323] ice 0000:83:00.0: VF 5 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.219649] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.219656] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.235264] ice 0000:83:00.1: VF 6 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.263454] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.263468] ice 0000:83:00.1: PF failed to honor VF 6, opcode 24, error -22
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.278080] ice 0000:83:00.0: VF 2 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.309696] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.309701] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.319975] ice 0000:83:00.0: VF 6 granted request of 16 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.343331] ice 0000:83:00.1: VF 6 granted request of 16 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.384308] ice 0000:83:00.1: VF 5 granted request of 41 queues.
Jul 18 18:53:16 tor-dl345-3 kernel: [ 5069.420862] ice 0000:83:00.1: VF 2 granted request of 41 queues.
Jul 18 18:53:17 tor-dl345-3 kernel: [ 5069.963258] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:53:17 tor-dl345-3 kernel: [ 5069.963266] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:53:21 tor-dl345-3 kernel: [ 5074.054695] ice 0000:83:00.0: VF 2 granted request of 16 queues.
Jul 18 18:53:21 tor-dl345-3 kernel: [ 5074.077215] ice 0000:83:00.1: VF 2 granted request of 16 queues.

Trying to relaunch just the failed channel.... (others still running)

Jul 18 18:59:10 tor-dl345-3 kernel: [ 5423.412314] vfio-pci 0000:83:01.6: enabling device (0000 -> 0002)
Jul 18 18:59:10 tor-dl345-3 kernel: [ 5423.641829] vfio-pci 0000:83:11.6: enabling device (0000 -> 0002)
Jul 18 18:59:10 tor-dl345-3 kernel: [ 5423.644933] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:10 tor-dl345-3 kernel: [ 5423.644942] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.751579] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.751588] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.905901] ice 0000:83:00.0: VF 6 granted request of 41 queues.
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.986339] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.986347] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5423.996094] ice 0000:83:00.1: VF 6 granted request of 41 queues.
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.021149] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.021163] ice 0000:83:00.1: PF failed to honor VF 6, opcode 24, error -22
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.065285] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.065292] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.075003] ice 0000:83:00.1: VF 6 granted request of 16 queues.
Jul 18 18:59:11 tor-dl345-3 kernel: [ 5424.093775] ice 0000:83:00.0: VF 6 granted request of 16 queues.
Jul 18 18:59:12 tor-dl345-3 kernel: [ 5424.879623] ice 0000:83:00.1: Cannot set RSS lut, err -5 aq_err ICE_AQ_RC_EACCES
Jul 18 18:59:12 tor-dl345-3 kernel: [ 5424.879631] ice 0000:83:00.1: set_rss_lut failed, type=2, size=512, error: -5 (-EIO)

[tszumski]

Thanks, it looks like ICE driver or firmware issue, can you share more logs with ICE drivers debug logs?

To enable dynamic debug during runtime operation of the driver, use
this command:

echo "module ice +p" > /sys/kernel/debug/dynamic_debug/control