Sometimes VM fails to start due to out of memory error, even though qmemman did freed (supposedly) enough

[marmarek]

How to file a helpful issue

Qubes OS release

R4.3

Brief summary

Sometimes VM fails to start with internal error: libxenlight failed to create new domain message. libxl logs shows it's about out of memory.

Steps to reproduce

Not sure exactly. Happens from time to time during integration tests, I think more often when starting two VMs at once.

Expected behavior

VM starts normally

Actual behavior

VM fails to start
libxl-driver.log contains

2024-08-23 20:39:24.619+0000: xc: panic: xg_dom_boot.c:119: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
2024-08-23 20:39:24.619+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
2024-08-23 20:39:24.638+0000: libxl: libxl_create.c:1753:domcreate_rebuild_done: Domain 58:cannot (re-)build domain: -3

I suspect the calculation how much free memory is needed to start a VM needs an update.

This started happening after update to Xen 4.19 (from Xen 4.17).

[marmarek]

With some extra logging I've collected the following:

1. qubesd requested 422 MB from qmemman
2. Before starting the VM there is 526 MB free (according to xl info free_memory)
3. After starting the VM (still paused) - 92 MB free

So, it used 434MB, 12MB more than calculated. I need to collect more data for the new formula...

[marmarek]

While the formula may be inaccurate, it looks like the issue is somewhere else. One of the failed log contains:

DEBUG:vm.test-inst-vm2:mem required with overhead: 459538432.0
DEBUG:vm.test-inst-vm2:free mem before start: 119

So, it requested 438MB from qmemman, but then only 119MB was freed - way too little for starting 400MB VM.