Add support for distributing prebuilds as platform-specific packages

[kriszyp]

A promising technique for distributing prebuilt binaries is to use a set of platform-specific packages for each platform's prebuilt binaries. Then each platform-specific package specifies the required platform and architecture, and the parent package specifies all the platform packages as optionalDependencies. When installed, NPM will then only install the optional dependency with the platform/architecture matching the current machine.

This provides both the efficiency of only needing to download the binary (or binaries) needed for the current platform, but also the key benefit of the prebuildify system in that binaries are downloaded as part of the normal NPM install process (no install scripts for extra downloads are necessary).

The new parcel css bundler demonstrates this technique:
https://cdn.jsdelivr.net/npm/@parcel/[email protected]/package.json

And this has also been suggested by isaacs himself for platform specific distribution:
npm/rfcs#120

As it turns out, this technique actually works really nicely, and with relatively little effort/changes, to the prebuildify system. I attempted to add some minimal modifications to add a flag to enable creating these optional platform-specific packages in this PR.

A few other things would need to be done to finish this:

• I also have a PR that I will file with node-gyp-build to find and use these platform specific packages (after searching builds/Debug, builds/Release, and prebuilds/)
• Some documentation, of course (if you are interested in proceeding with this).
• It might be nice to have a script or automate the addition/update of the optionalDependencies versions with all the platform packages to match the current package's version (or maybe this is more appropriate in prebuildify-ci, I am not sure)
• Could also have a script for publishing the platform packages, but I think that's probably outside the scope of prebuildify

[mafintosh]

Awesome, some CI failures above if you can take a look. So with the optional deps, all builds are installed but then discarded at install time if they are irrelevant?

[kriszyp]

some CI failures above if you can take a look

I'm a little stumped on this. The CI failures seem like they are due to Visual Studio being missing or out-of-date on the CI target machines, and unrelated to any changes I made. I verified that the npm test runs successfully on my local Windows machine, and I pushed a branch that is an exact replica of the prebuildify's master (without any of my PR) and it also fails:
https://github.com/kriszyp/prebuildify/actions/runs/2077970145

So with the optional deps, all builds are installed but then discarded at install time if they are irrelevant?

Basically yes, although there is actually some slight variation in how this is handled. In NPM v6 and Yarn, all the platform packages are download (into the cache), and the platform compatibility is checked and only the matching/compatible package gets installed (others are discarded). NPM v7+ improves on this, and only downloads the package metadata first, then checks the platform compatibility and then only downloads the matching/compatible platform package. But in all cases, only the compatible/matching platform package gets installed.

If you want to see this all in action, I've built and published a beta version of my msgpackr-extract package with this updated system, and you can try it out (with full details so you can see the requests that npm makes):

npm install [email protected] --verbose

[vweevers]

The Windows failure is a mismatch between github runners and node-gyp version, indeed unrelated to this PR. I've seen it before, will have a look. Either need to downgrade github runner or bump node-gyp.

[vweevers]

@kriszyp I fixed windows CI on the master branch; rebasing should do it

[Julusian]

This pr looks promising.

Should there be some tooling to help generate the package.json optionalDependencies block? I can very easily see myself forgetting to update it when publishing new versions, resulting in a broken publish of the main package.

I would suggest it best to require the platform packages to be scoped packages, to prevent potential issues with package name squatting.

[kriszyp]

Should there be some tooling to help generate the package.json optionalDependencies block?

That would be nice, but I don't think it belongs here. At least with how I have used prebuildify, it is run in a separate container/server for each platform, so it is more focused on the builds for that platform, without knowledge of other platforms that will be built. I use prebuildify-ci to download all the builds (from github), and I think it would be reasonable to set the optionalDependencies there. However, I am not sure if everyone is using that (or prebuildify-cross?), and it seemed like the easiest part of this to do separately from the prebuildify tools (here is the script I use). But I'd be happy to put together a PR for prebuildify-ci if that helps. The one other thing that requires a little bit of manual effort is setting up the mass publishing, but again, it is not too hard to set up.

I would suggest it best to require the platform packages to be scoped packages

Oh yeah, scoped packages are good idea for this. Now I feel bad I haven't been using that. So I think this raises a few questions about configuring and requirements. First, do we include a setting to indicating the scope, or infer it? For unscoped packages, do we assume the platform builds use a scope with the same name as the package, like my-package has linux build @my-package/linux-x64? And for already scoped packages, just continue to prefix like we are doing now, so @my-org/my-package has linux build @my-org/my-package-linux-x64?

Or are the some cases where might be preferable to publish under a scope that isn't used by the package? And if so would it be better to have a separate argument for this, or have a single optional-packages-scope argument? And is it too much to require this since it imposes a little extra setup/configuration?

[Julusian]

Should there be some tooling to help generate the package.json optionalDependencies block?

That would be nice, but I don't think it belongs here. At least with how I have used prebuildify, it is run in a separate container/server for each platform, so it is more focused on the builds for that platform, without knowledge of other platforms that will be built. I use prebuildify-ci to download all the builds (from github), and I think it would be reasonable to set the optionalDependencies there. However, I am not sure if everyone is using that (or prebuildify-cross?), and it seemed like the easiest part of this to do separately from the prebuildify tools (here is the script I use). But I'd be happy to put together a PR for prebuildify-ci if that helps. The one other thing that requires a little bit of manual effort is setting up the mass publishing, but again, it is not too hard to set up.

Yeah that makes sense. I have not actually usedprebuildify for anything myself yet, due to lack of cmake-js support so I dont have any real knowledge on workflows. I would really like to start using prebuildify or something similar though (and have been for over a year now)..

I would suggest it best to require the platform packages to be scoped packages

Oh yeah, scoped packages are good idea for this. Now I feel bad I haven't been using that. So I think this raises a few questions about configuring and requirements. First, do we include a setting to indicating the scope, or infer it? For unscoped packages, do we assume the platform builds use a scope with the same name as the package, like my-package has linux build @my-package/linux-x64? And for already scoped packages, just continue to prefix like we are doing now, so @my-org/my-package has linux build @my-org/my-package-linux-x64?

Or are the some cases where might be preferable to publish under a scope that isn't used by the package? And if so would it be better to have a separate argument for this, or have a single optional-packages-scope argument? And is it too much to require this since it imposes a little extra setup/configuration?

I think the optional-packages-prefix would be good to have. For already scoped packages, it should be optional. For packages which arent scoped it should be required and running prebuildify should fail.
To me that gives a good level of flexibility and forces users to avoid potential pitfalls of not using scoped packages.

[kriszyp]

Perhaps something like this?
prebuildify --optional-packages

• @my-org/my-package -> @my-org/my-package-linux-x64
• my-package -> @my-package/linux-x64 (to me this seems like a reasonable default to encourage and enforces scoping)

prebuildify --optional-packages-prefix=@some-org/some-package

• @my-org/my-package (or anything) -> @some-org/some-package-linux-x64

prebuildify --optional-packages-prefix=@some-org

• @my-org/my-package (or anything) -> @some-org/linux-x64

prebuildify --optional-packages-prefix=my-package (if you really don't want scoping, can explicitly avoid using perhaps?)

• @my-org/my-package (or anything) -> my-package-linux-x64

[vweevers]

If possible, I'd like to avoid configurable prefixes, as it makes discovery (by node-gyp-build and other tooling) more difficult. The naming should be simple and predictable.

[Julusian]

In node-gyp-build you would make it take matching properties to tell it look for packages and what name to use. Doesnt sound much more difficult to me. You would have to change the method signature for it, but doing that would greatly help with other current issues (such as prebuild/node-gyp-build#22 or #57). And it would let you make this package flow be opt-in on the node-gyp-build side too

The problem I can see with forced prefixes, is that if you publish my-lib along with my-lib-linux-x64, then anyone (either maliciously or accidentally) could publish my-lib-linux-arm64. Then if you decide to publish that build, you are a bit stuck. How would you resolve that? Or someone might add that version someone else published as a dependency themselves to their project hoping it might fix the lack of prebuilds for their platform, and then your node-gyp-build will happily try to load this incorrect/malicious module.

Or the more concerning thought, of that if you make the scanning for this libs be enabled by default, then that feels like a dll injection attack waiting to happen. I suppose this isnt very likely because you would need to still have the rogue module explicitly installed onto the machine, but it could happen

[kriszyp]

I was thinking about that as well. However, I was wondering if it would be better if node-gyp-build actually scanned the list optionalDependencies from the package.json, and then matched on the platform names there. This would also provide a way of dealing with the combined architecture builds that macos supports (and once a package from optionalDependencies was matched that clearly indicates the package to load). But that being said, I am not opposed to simple and predictable :).

[Julusian]

node-gyp-build actually scanned the list optionalDependencies from the package.json

I would rather it didnt do that.. dynamic import of files is the exact way to make libraries not work with webpack or other bundlers..

I have a project where I am playing with bundling in webpack, and all the native modules have had to be kept outside of that bundle (which then includes all their dependencies), often because the clever searching for .node files doesnt work prebuild/node-gyp-build#22

Making it an explicit call when invoking node-gyp-build is the best way to make it consistently work everywhere

[vweevers]

I think supporting only this will work in 99% of cases:

prebuildify --optional-packages

• @my-org/my-package -> @my-org/my-package-linux-x64
• my-package -> @my-package/linux-x64 (to me this seems like a reasonable default to encourage and enforces scoping)

The --optional-packages-prefix examples seem unrealistic, I don't see a use case for separate scopes (@my-org/my-package -> @some-org/*). That just leaves the --optional-packages-prefix=my-package example, i.e. unscoped optional packages, which I agree should be discouraged. We might as well take that off the table altogether, and only support scoped optional packages.

[kriszyp]

Sounds good to me. So to be clear, no extra arguments, the only change is making unscoped packages map from my-package -> @my-package/linux-64 (and scoped packages will just use the platform as a suffix like it currently does, @my-org/my-package -> @my-org/my-package-linux-64). I can go ahead make that change to this PR and the one for node-gyp-build.

I guess one other thing we might consider is if we want a little more descriptive package names for the scoped platform names than just "linux-x64", like maybe mapping to @my-package/prebuild-linux-x64?

I think the only collision hazard would be if someone has a package with the same name as an existing org (or username) (I think they are packages and orgs are different namespaces), but seems reasonable to cross that bridge if it ever actually happens.

[vweevers]

I guess one other thing we might consider is if we want a little more descriptive package names for the scoped platform names than just "linux-x64", like maybe mapping to @my-package/prebuild-linux-x64?

Or even @my-package/my-package-linux-x64? A bit ugly, but descriptive and consistent. Also leaves room for other packages under the scope (that might be another unrealistic case, but this time the only cost is smurf naming).

[CMCDragonkai]

Then each platform-specific package specifies the required platform and architecture, and the parent package specifies all the platform packages as optionalDependencies. When installed, NPM will then only install the optional dependency with the platform/architecture matching the current machine.

This means the parent package's code is still installed right? Which would mean that the platform-specific dependency should only contain the native-addon binary. Or would it be better to have the js-bindings and native-addon binary in the platform-specific package?

[kriszyp]

This means the parent package's code is still installed right?

Yes

Which would mean that the platform-specific dependency should only contain the native-addon binary.

Basically yes, although it also needs (and auto-creates) a simple package.json, which is necessary for publishing, and an empty index.js, which is necessary to get node to resolve the package path).

Or would it be better to have the js-bindings and native-addon binary in the platform-specific package?

You could, but then the js bindings get duplicated to each platform package, and since some package managers download all the platform packages, would be a little more wasted bytes, less efficient. Also, prebuild's node-gyp-build is set up to just dynamically load the binary, and this PR was intended to have minimal changes to prebuild's system.

[CMCDragonkai]

I see so you have 2 options:

1. Put empty index.js and minimal package.json for then native package, and keep the js binding in the parent package, which can also switch behaviour depending on platform differences.
2. Put the js binding using node-gyp-build in each platform package, and then the package.json will require node-gyp-build. This could be useful in case the js-bindings also had to change behaviour due to platform-differences. I wonder how the js-binding would know what platform they are running in.

[kriszyp]

If it helps at all, you can see this PR (and the node-gyp-build PR) in use with my https://www.npmjs.com/package/msgpackr-extract package, and platform packages at https://www.npmjs.com/org/msgpackr-extract .

[CMCDragonkai]

Question, how does npm know which optional dependency to install? The docs for npm says that it just means installation failure of an optional dependency will not prevent npm to proceed.

Does that mean there are meant to be some sort of constraint either via an install or preinstall script inside the optional dependencies that fail to install if not on the right platform?

I just saw the os and cpu constraints. This makes sense.

But what about for things like android. Nobody is compiling something on android to target android right? I'm thinking of situations where you have a native package that depends on another native package that uses this optional packages structure. What you need all the platform-specific packages and you want to ignore the os/cpu constraints?

[kriszyp]

@CMCDragonkai The parent project includes optional dependencies for all the platforms, and each platform package specifies an os and cpu so that npm rejects all but the matching platform dependency package. If there is no matching platform, the parent project has "scripts": { "install": "node-gyp-build" } in the package.json, which is a script that verifies if there is an existing binary that is available (tries to load it), and if there isn't one, runs the standard node-gyp compilation process (and hopes and prays that a build tool chain exists), which should hopefully get it to build on "other" platforms. Of course if the build tools aren't there, then the binaries can't be loaded or compiled, and the package installation will fail. Of course if there is some way that a dependent package doesn't really require the native package and it can tolerate its absence, it can likewise use an optionalDependency (the example package msgpackr-extract is actually used by msgpackr in this way).