Support the MacOS sandbox for larger projects

[j-baker]

Motivation

Fixes #482.

MacOS has trouble with derivations which (directly or transitively) have many buildInputs.

Crane at present creates a build structure in which a given cargo command will transitively depend on numCrates nix store paths. This means that Crane fails to build projects with over about 600 crate dependencies on MacOS if the sandbox is enabled.

This MR utilises a tiering approach to improve this.

Each registry is assigned to a shard based on the hash of the crate name. If there are <32 crates in a registry there is one shard, if <2048 there are 16 shards, otherwise 256. Crates are directly extracted into these shard derivations rather than symlinking.

What this means is:

1. Crane will not create a vendoring derivation with many inputs unless a project has a truly crazy number of dependencies.
2. No downstream cargo derivation will have many inputs either.

Checklist

• added tests to verify new behavior
• added an example template or updated an existing one
• updated docs/API.md (or general documentation) with changes
• updated CHANGELOG.md

[ipetkov]

Hi @j-baker thanks for the PR!

One bit of hesitation I have with the current direction is that it ends up pessimizing dependency handling for the "default" case (of not building inside of a macOS sandbox)... The approach to symlink crate sources in the final vendor directory means that each unique crate-version-source is unpacked only once in the Nix store (regardless of whether Nix store optimization/hard-linking is enabled) so that derivations with completely different sets of dependencies don't end up unpacking sources multiple times (each which uses up some CPU and disk space).

I'm not familiar with how the macOS sandbox operates, but based on your comment #482 (comment) it sounds like each derivation is getting its own sandbox, and the vendor derivation's dependencies along with the real build's deps pushes it over the edge. Another potential (and maybe simpler?) solution is to introduce some kind of vendorInstallMode variable to vendorCargoDeps (similar to installCargoArtifactsMode) which results in a deep copy of the source (instead of a symlink which would remain the default). That way one can opt-into this "more expensive" behavior as necessary.

Thoughts?