Binary mirroring not working since node-pre-gyp to prebuild-install migration

[Fiouz]

How frequently does the bug occur?

All the time

Description

Context

• Closed network environment (intranet), where build machines have no IP routing to the internet (and especially https://static.realm.io)
• Build machines have IP routing to an HTTP mirroring machine (e.g. Sonatype Nexus / JFrog Artifactory), so they can still access content from public repositories, by using internal URLs (as long as mirroring rules have been provisioned) such as https://corporate.mirror.lan/repository/static.realm.io
• Build machines NOT provisioned for native binary compilation
• Build machines configured to redirect realm binary download HTTP request to the internal mirror npm config set realm_binary_host_mirror https://corporate.mirror.lan/repository/static.realm.io, according to https://github.com/mapbox/node-pre-gyp/blob/v1.0.9/README.md#download-binary-files-from-a-mirror (for older versions of realm) / https://github.com/prebuild/prebuild-install/blob/v7.1.1/README.md#custom-binaries (for newer versions of realm)

Command

npm --loglevel=verbose install [email protected]

Expected behavior

npm install should success, and since our build machines are not able to compile native components, success would imply that the binary was successfully downloaded from the configured HTTP mirror.

Since the mirror setting between node-pre-gyp and prebuild-install has the same spelling, we expect a working mirroring for older versions to also work for newer versions of realm.

See also: #2501

Actual behavior

When prebuild-install constructs the mirror URL, the URL has a different shape compared to the "original" URL, making it unable to download anything from the mirror.

Generated URL for mirroring:

<mirror_prefix>/v10.20.0-beta.5/realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz

Original URL (without mirroring):

<original_prefix>/realm-js-prebuilds/10.20.0-beta.5/realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz

(URL obtained by manually executing the prebuild-install --verbose command from the node_modules/realm directory)

This only affects latest versions, previous versions that relied on node-pre-gyp (e.g. 6.1.8) are properly mirrored, i.e. the following works fine with binary mirroring enabled:

npm --loglevel=verbose install [email protected]

Stacktrace & log output

npm verb stack Error: command failed
npm verb stack     at ChildProcess.<anonymous> (C:\Users\Fiouz\AppData\Local\Temp\realm\prefix\node_modules\npm\node_modules\@npmcli\promise-spawn\lib\index.js:63:27)
npm verb stack     at ChildProcess.emit (node:events:527:28)
npm verb stack     at maybeClose (node:internal/child_process:1092:16)
npm verb stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm verb pkgid [email protected]
npm verb cwd C:\Users\Fiouz\AppData\Local\Temp\realm\project
npm verb Windows_NT 10.0.19043
npm verb node v16.15.1
npm verb npm  v8.12.2
npm ERR! code 1
npm ERR! path C:\Users\Fiouz\AppData\Local\Temp\realm\project\node_modules\realm
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c prebuild-install -r napi || cmake-js build
npm ERR! prebuild-install info begin Prebuild-install version 7.1.1
npm ERR! prebuild-install info looking for local prebuild @ prebuilds\realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz
npm ERR! prebuild-install info looking for cached prebuild @ C:\Users\Fiouz\AppData\Local\Temp\realm\cache\_prebuilds\0b863b-realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz
npm ERR! prebuild-install http request GET https://static.realm.io/v10.20.0-beta.5/realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz
npm ERR! prebuild-install http 404 https://static.realm.io/v10.20.0-beta.5/realm-v10.20.0-beta.5-napi-v4-win32-x64.tar.gz
npm ERR! prebuild-install warn install No prebuilt binaries found (target=4 runtime=napi arch=x64 libc= platform=win32)
npm ERR! 'cmake-js' is not recognized as an internal or external command,
npm ERR! operable program or batch file.

Can you reproduce the bug?

Yes, always

Reproduction Steps

1. Make sure local machine does not have native compilation capability
2. Setup an HTTP mirror for https://static.realm.io
3. Configure the realm_binary_host_mirror npm variable with the relevant mirror URL (npm config set realm_binary_host_mirror https://mirror.host/static.realm.io)
   Since it may be impractical to setup an HTTP mirror just for the sake of reproducing the issue, you can just use the official URL as a mirror URL, and see that this results in a broken URL, as shown in the above reported failure log:

npm config set realm_binary_host_mirror https://static.realm.io

1. Clean <npm_config_cache>/_prebuilds
2. npm --loglevel=verbose install [email protected], and see that it fails
3. Try an older version that still uses node-pre-gyp npm --loglevel=verbose install [email protected], and see that it succeeds

Note: details about mirroring the npm registry, and making it work with custom X.509 certificates are omitted from the reproduction steps for brevity

Version

10.20.0-beta.5

What SDK flavour are you using?

Local Database only

Are you using encryption?

No, not using encryption

Platform OS and version(s)

Windows / Linux

Build environment

[email protected]
[email protected]

Cocoapods version

No response

[tomduncalf]

Thanks for the report @Fiouz. This isn't something I'm familiar with but it sounds like you probably have some in depth knowledge of working with this setup – do you have any idea of what we might need to change to make this work, or an example of another npm module which uses prebuild-install and can successfully be installed in this environment?

[Fiouz]

From what I can gather, we need the following branches to behave the same:

• https://github.com/prebuild/prebuild-install/blob/4a1ed436676a6b8b0611272440f0ce5fd0b798bf/util.js#L47
• https://github.com/prebuild/prebuild-install/blob/4a1ed436676a6b8b0611272440f0ce5fd0b798bf/util.js#L51-L57

After some quick testing, it seems possible by combining a change on the project, and a change on the build machine:

• We need to discard the {tag_prefix} (it is forcefully added when mirroring, but omitted with custom host), by changing

  realm-js/package.json

  Line 56 in 8aaf11c

  "install": "prebuild-install -r napi || cmake-js build",

  into

    "install": "prebuild-install --tag-prefix= -r napi || cmake-js build",

• It seems prebuild-install has a different design decision compared to node-pre-gyp: node-pre-gyp always takes into consideration the remote_path parameter, whereas prebuild-install ignores it when using a mirror URL. Thankfully, we can exploit the fact that prebuild-install allows for an additional setting spelling {module_name}_binary_host which takes precedence over the shared spelling with node-pre-gyp {module_name}_binary_host_mirror. We could instruct users with the following (wording is sub-optimal):

If you previously used the npm config set realm_binary_host_mirror https://mirror.lan/realm, you now need to configure npm config set realm_binary_host https://mirror.lan/realm/realm-js-prebuilds (same mirror, additional suffix)

[tomduncalf]

Thanks for the research @Fiouz! I've created a branch of Realm JS with the change you suggested, named td/binary-mirroring-test. Are you able to try this out to see if it works for you?

[Fiouz]

Yes, it works thank you. As you may already feel, such setup looks janky, so I raised prebuild/prebuild-install#183 to have the point of view of prebuild-install maintainers.

[tomduncalf]

OK great, I'll run this past the team to check they are happy with this change as I don't have much experience with this part of the build. If they are then we can merge it in

[tomduncalf]

@Fiouz We have just released Realm JS 10.9.3 which should fix this issue, please let us know if you are still having problems.

[Fiouz]

Thank you @tomduncalf, it is indeed working with 10.19.3. Would you have a timeline for a > 10.20.0-beta.5 version?

[tomduncalf]

We are actively working on getting a new beta release out now. We have some test failures we are investigating but hopefully we can resolve these soon. I can't commit to a timeline but we will get it out as soon as possible!