Intermittent cross-comp errors when many builds are queued

[MattSturgeon]

e.g. from this build:

cannot build on 'ssh-ng://[email protected]': error: failed to start SSH connection to '[email protected]'
Failed to find a machine for remote build!
derivation: sjilr3pnp4l1mcl6pfry3yaz4nxyam08-plugins-utils-dashboard.drv
required (system, features): (aarch64-linux, [])
2 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 80, [benchmark, big-parallel, gccarch-armv8-a, kvm, nixos-test], [])
([aarch64-darwin, x86_64-darwin], 8, [big-parallel], [])
error: a 'aarch64-linux' with features {} is required to build '/nix/store/sjilr3pnp4l1mcl6pfry3yaz4nxyam08-plugins-utils-dashboard.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

This is frustrating, because it usually happens when the load is high, and the only solution is to attempt a re-build of the entire nix-eval, wasting resources further.

[MagicRB]

This seems like a infra/nix issue not buildbot-nix issue. There may be a way to detect this class of failures and to retry automatically though.

[MattSturgeon]

This seems like a infra/nix issue not buildbot-nix issue.

Thanks for the pointer, I've opened nix-community/infra#1416.

I'll close this for now, unless you think it's worth pursuing a workaround in buildbot-nix?

[MagicRB]

I think as a long term thing, handling this case would be nice. Computers suck, stuff happens, sometimes temporarily, would be nice to not involve the human, let the computer clean up its own mess if you ask me.