`nh os switch --build-host <HOST>` still builds locally

[ejpcmac]

Context

I’m trying to replace nixos-rebuild switch --target-host <TARGET> --build-host <TARGET> with nh os switch --target-host <TARGET> --build-host <TARGET>, the target and build host being the same.

Problem description

When I pass --build-host to nh os switch, I properly see that a --builder parameter is passed to nix build. However, it builds everything locally, and then copies the data to my target host, as if --build-host was not passed.

Expected behaviour

When using nixos-rebuild switch with --build-host being the same as --target-host, only the evaluation is performed locally. Everything else (downloading and building the derivation outputs) is done on the target, avoiding unnecessary network copies.

[cdo256]

I believe the issue stems from its interaction with --no-nom.

I'll open a PR with a fix shortly.

Could you please confirm whether you were running with --no-nom, so I know this caused your issue.

The issue is in commands::Build::run():

        let exit = if self.nom {
            let cmd = {
                Exec::cmd("nix")
                    .arg("build")
                    .args(&installable_args)
                    .args(&["--log-format", "internal-json", "--verbose"])
                    .args(&match &self.builder {
                        Some(host) => {
                            vec!["--builders".to_string(), format!("ssh://{host} - - - 100")]
                        }
                        None => vec![],
                    })
                    .args(&self.extra_args)
                    .stderr(Redirection::Merge)
                    .stdout(Redirection::Pipe)
                    | Exec::cmd("nom").args(&["--json"])
            }
            .stdout(Redirection::None);
            debug!(?cmd);
            cmd.join()
        } else {
            let cmd = Exec::cmd("nix")
                .arg("build")
                .args(&installable_args)
                .args(&self.extra_args)
                .stderr(Redirection::Merge)
                .stdout(Redirection::None);

            debug!(?cmd);
            cmd.join()
        };

[cdo256]

The other thing you may be running into is:

warning: Ignoring the client-specified setting 'builders', because it is a restricted setting and you are not a trusted user

In which case you need to set yourself as a trusted user in your NixOS config:

{
  nix.settings.trusted-users = [ "yourusername" ];
}

[ejpcmac]

@cdo256, thanks for looking into this! I was not using --no-nom by the way. The exact command I’ve been using is:

nh os switch --ask <path-to-config> --hostname <hostname> --target-host <target> --build-host <target> -- --keep-going

My nix.settings.trusted-users has the default value "*", thus I’m not seeing either the warning you are mentioning.

I’ll try using the latest nh next time I do a deployment and let you know.

[MaxSilvesterQWE]

This is still broken. When I run nh os switch --target-host user@host --build-host user@host -H host it outputs Could not open a connection to your authentication agent. twice and then just build locally / with my usual remote builders. (I did test the latest version using nix shell github:nix-community/nh)

I think this is because it is implemented using --builders, which doesn't seem to be able to ask for a password interactively.

In nixos-rebuild (and nixos-rebuild-ng) it asks for the password interactively and then copies the derivation using nix-copy-closure and then builds remotely using something like ssh user@host nix-build /nix/store/hash-system-derivation.

[ejpcmac]

I did not retry yet, but I must add that I’m using a SSH key provided by a hardware security key via gpg-agent to connect to the remote host. It did not even tried to unlock the key when starting the build, contrary to nixos-rebuild, which connects to the remote host right after the evaluation.

[ejpcmac]

I can confirm this does still builds locally with nh from commit 003dd72.

A few more details:

• as @MaxSilvesterQWE, I am specifying the --target-host and --build-host as user@host,
• there were no warning at any time during the execution contrary to him (it just silently builds locally instead of on the remote),
• gpg-agent runs as my user, not nix-builder, which may be the issue. However this is working perfectly fine with nixos-rebuild.

It seems I don’t have the right to reopen the issue. Should I open a new one?

[cdo256]

I currently don't have a setup to test this, but it definitely seems like my initial fix was for something other than this bug.

@viperML I think this should be reopened.

[NotAShelf]

I don't have the time (nor the desire) to test this, but given this is reportedly not fixed I'm re-opening the issue. A detailed report, or a PR would be appreciated but I'll try to look into this within the upcoming weeks.

[culxttes]

Hi,

I'm trying to use nh os switch with a remote build host, but it seems like the --build-host flag is being ignored, all builds still occur locally, and there's no SSH connection initiated to the specified host.

Here’s the command I ran:

nh os switch \
  --hostname sisyphe \
  --build-host culottes@sisyphe.sagbot.com \
  --target-host culottes@sisyphe.sagbot.com

Output:

┣━━━ Builds              │ Downloads         │ Host                                 
┃    ⏵ 8 │ ✔ 107 │       │     │       │     │ localhost                            
┃        │       │       │ ↓ 1 │ ↓ 546 │     │ [1]: https://cache.nixos.org         
┃        │       │       │     │ ↓   8 │     │ [2]: https://nix-community.cachix.org
┗━ ∑ ⏵ 8 │ ✔ 107 │ ⏸ 103 │ ↓ 1 │ ↓ 554 │ ⏸ 0 │ ⏱ 3m11s

I also tried just:

nh os switch \
  --build-host culottes@sisyphe.sagbot.com

But the output still shows only local builds:

┣━━━ Builds           │ Downloads       │ Host                                 
┃        │ ✔ 20 │     │     │     │     │ localhost                            
┃        │      │     │     │ ↓ 3 │     │ [1]: https://cache.nixos.org         
┃        │      │     │     │ ↓ 1 │     │ [2]: https://hyprland.cachix.org     
┃        │      │     │     │ ↓ 2 │     │ [3]: https://nix-community.cachix.org
┗━ ∑ ⏵ 0 │ ✔ 20 │ ⏸ 0 │ ↓ 0 │ ↓ 6 │ ⏸ 0 │ Finished at 15:11:30 after 48s    

There are no visible build processes or SSH connections to the remote host, indicating that the builds are happening entirely on my local machine. Let me know if I'm missing something or if this is a bug

[culxttes]

As an additional note, if I add the --no-nom flag, I get the following error:

cannot build on 'ssh://culottes@sisyphe.sagbot.com': error: failed to start SSH connection to 'culottes@sisyphe.sagbot.com': culottes@sisyphe.sagbot.com: Permission denied (publickey,keyboard-interactive).

But despite this error, the build still proceeds locally, which suggests that the remote host is completely ignored instead of causing a hard failure or fallback mechanism.

[culxttes]

For context, I'm using an SSH key of type sk-ssh-ed25519@openssh.com (a FIDO2 security key), and it's properly set up and works with manual SSH connections as well as with nixos-rebuild using the same remote host.