Nix daemon crashes with assertion failure when building multiple mlibc cross-compiled packages

[siraben]

Describe the bug

Nix daemon crashes with assertion failure when building multiple cross-compiled packages for aarch64-linux-mlibc target in parallel. The crash occurs in the BinaryCacheStore's callback mechanism with the error: Assertion '!prev' failed.

Steps To Reproduce

1. Run the following command:

   nix build --no-link -L github:NixOS/nixpkgs/pull/371092/merge#{pkgsCross.aarch64-linux-mlibc.ed,pkgsCross.aarch64-linux-mlibc.vim,pkgsCross.aarch64-linux-mlibc.tmux,pkgsCross.aarch64-linux-mlibc.git}

2. Observe the error:

   error: Nix daemon disconnected unexpectedly (maybe it crashed?)
   

3. Check system logs to see the assertion failure and stack trace

Expected behavior

The packages should build successfully without crashing the nix daemon. Cross-compilation for mlibc targets should work reliably when building multiple packages in parallel.

Metadata

nix-env (Nix) 2.28.3

Additional context

Stack Trace

nix-daemon: /nix/store/XHD38P32PVBZQMI0ZQ1WICLJY5B7YQDV-nix-util-2.28.3-dev/include/nix/util/callback.hh:45: void nix::Callback<T>::rethrow(const std::__exception_ptr::exception_ptr&) [with T = std::optional<std::__cxx11::basic_string<char> >]: Assertion `!prev' failed.

Stack trace:
 0# 0x000000000048F74B in nix-daemon
 1# 0x00007FD49D8BC20A in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
 2# 0x00007FD49D8BC275 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
 3# 0x00007FD49E092358 in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 4# nix::Store::queryPathInfo(nix::StorePath const&, nix::Callback<nix::ref<nix::ValidPathInfo const> >) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 5# nix::Store::queryPathInfo(nix::StorePath const&) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 6# nix::Store::querySubstitutablePathInfos(std::map<nix::StorePath, std::optional<nix::ContentAddress>, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, std::optional<nix::ContentAddress> > > > const&, std::map<nix::StorePath, nix::SubstitutablePathInfo, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::SubstitutablePathInfo> > >&) in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 7# 0x00007FD49E1D4FBE in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 8# 0x00007FD49E1DC712 in /nix/store/rcjm767qc0mb6qv6inn4vgx8sqwbach1-nix-store-2.28.3/lib/libnixstore.so
 9# nix::ThreadPool::doWork(bool) in /nix/store/319hpn6dffzslywqhvjcwhs8xj822rdl-nix-util-2.28.3/lib/libnixutil.so
10# 0x00007FD49D8E86D3 in /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
11# 0x00007FD49D672D02 in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
12# 0x00007FD49D6F23AC in /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6

Key observations:

• Crash occurs specifically when building multiple aarch64-linux-mlibc cross-compiled packages in parallel
• Building individual packages works fine
• Crash happens in BinaryCacheStore::queryPathInfoUncached when querying substitutable paths
• Possible race condition in the callback handling mechanism
• Issue discovered while testing nixpkgs PR mlibc: init at 6.1.1 nixpkgs#371092

Workarounds:

1. Build packages individually instead of all at once
2. Use --option substituters "" to disable substituters
3. Build with --max-jobs 1 (though this doesn't always prevent the crash)

System information:

• Platform: Linux x86_64
• NixOS version: unstable
• The crash is reproducible and happens immediately when the command is run

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[xokdvium]

Randomly ran into the same assertion failure on master, so at least >= 2.28, <= master are affected.

[siraben]

@xokdvium can you elaborate on how you triggered it? And provide a stack trace if possible.

[xokdvium]

@xokdvium can you elaborate on how you triggered it

I think it was rather random, just when rebuilding the system.

Unfortunately the coredump isn't available, so I don't have a stacktrace atm. Will try to reproduce.