CI actions: is the test suite ever run?

[bruceoberg]

do any of the offical github actions run the full test suite from the tests directory? the test.yml workflow does an init --switch command, but i can't see where any of the tests from the tests directory are run.

i'm trying to fix #6323 and am new to nix/home-manager development. the problem i'm fixing would have been caught by an appropriate unit test, so i'm trying to get the tests running. for various reasons (see below), i'd like to run the test suite via github actions. i was expecting test.yml to do this, but it doesn't appear to.

the contributing/tests.md page mentions running something like nix build --reference-lock-file flake.lock --option allow-import-from-derivation false ./tests#test-all. none of the machines i have acess to can run this command. my linux boxes are all in the cloud and don't have the needed nested virtualization features for the NMT test suite. my home machine is an M2 mac, and it cannot run tests due to x86_64 v aarch64 and darwin v linux issues.

[rycee]

Hi! Yeah, the tests are run for PRs but not as GitHub actions but by the Nix Community buildbot: https://buildbot.nix-community.org/#/projects/29

For changes in the command line it is probably better to work with integration tests, though, and unfortunately those are not run for PR jobs. They can be run (at least on Linux) using nix run .#tests -- -t.

I think @khaneliman is using mac so perhaps he has some insight into running tests on macs?

[khaneliman]

I believe I added some integration tests to run in buildbot, though. But, yes, I've been running them through nix run .#tests utility myself. Unfortunately, I don't think we can run the integration tests on darwin. I've had to run those on my nixos desktop.

[bruceoberg]

thanks folks. yes i'd like to amend the basic integration tests to verify that home-manager option works as desired. i've even written the tests.

i've found a linux box in the cloud that i believe should run the home-manager integration tests (a digital ocean droplet). it has a minimal nixos system (25.05) and a clean clone of the latest home-manager source. running nix run .#tests -- -t integration-test-standalone-standard-basics fails on the first subtest ("Home Manager installation").

the failure is that several external hosts (cache.nixos.org, ftp.gnu.org) cannot be reached at all from within the NMT qemu VM. this fails some downloads and eventually the test itself.

i thought this was a qemu networking issue, but claude convinced me that NMT intends the qemu VM to be sandboxed with no network access at all.

so i'm confused about why my clean linux box with an offical clone of the home-manager source can't run the single integration-test-standalone-standard-basics test. is there something i'm missing about how the host and VM nix stores are supposed to interact? is there some sort of build i should manually do before running the test?

[fwiw, running this test within an aarch64-linux VM on a mac fails in exactly the same way... downloading croaks.]