test: Fix scheduler for CPU/RAM starvation

[martinpitt]

Hopefully reproduces this mess from #22373. No changes for the first run, I want to see it fail.

We are seeing these problems even in PRs without affected "expensive" tests.

[martinpitt]

This initial run and a forced retry both have a few retries due to VM boot failures.

[martinpitt]

first run with fixed scheduler looks promising -- two real flakes, plus check-testlib failures which are a real bug (fixing now), but no VM boot failures!

[martinpitt]

ugh, that was worse again, the fine-tuning with measured variants was apparently too aggressive

[martinpitt]

This run was using 13 parallel spots, as I was refining the memory estimates. Turns out they are very good now!

RAM analysis:

• Budgeted: 28.0 GB for 14 tests
• Actual peak: 27.3 GB consumed (min available was 2.9 GB of 30.2 GB total)
• Overestimation: Only 0.7 GB (2%)

So that confirms my recent suspicion that with 8 parallel tests we are not RAM bound, as I always thought. Instead, booting many parallel VMs is just too slow and times out, hence we are CPU/load bound. The recent commit already throttled the destructive tests by load, but it wasn't enough yet, and the 13 parallel ND tests are definitively too much.

Load Analysis:

• Max load: 31.49 on 16 CPUs (almost 2× overload!)
• Load at ND start: 0.93 → climbed to 12.3 during ND phase → peaked at 31.49 during destructive phase
• Problem: 1-min loadavg is a trailing average - it started low (0.93), so all 14 VMs launched. By the time load climbed to 31.49, tests were already running.

The load throttling from the current top commit a264683 didn't trigger at all.

Analysis:

Peak loads:

• During ND phase (14 VMs): Load peaked at 31.49 🔥
• During destructive phase: Max load was only 15.67 (never hit the 16.0 threshold)

Load by VM count during destructive:

• 7-8 VMs: avg ~9.5, max 15.67
• 9 VMs: avg 4.91 (only briefly)

[martinpitt]

for $deity's sake, can this get green at all? Putting back the load test in addition to running_procs, and lowering the numbers even more. This is getting ridiculously inefficient, although surprisingly it only took 64 mins -- not really longer than the previous approaches.