HTTP2 performance and multiple concurrency

[gunta]

Hi!

I'm currently using an internal npm registry.

Do you think it would be worth exploring the possibility of making this HTTP2 enabled?
This may allow for infinite concurrency with no penalty.

For example we could use this for server/client: https://github.com/molnarg/node-http2

[rstacruz]

That'd be interesting. We're currently using got, but not by much, so it might be easy to switch to http2 with possible http fallback.

If there's a package that already handles this, lemme know (I don't know of any!). PR's are also welcome.

[gunta]

Unfortunately there is no package that wraps this.

I've created an Issue for GOT here:
sindresorhus/got#167

Otherwise, we would have to use is-http2 along with node-http2.
https://github.com/stefanjudis/is-http2

Or create a new package.

[gunta]

Ok seems that its gonna be easier to fix on the GOT side.

However we might add here the spdy module as a dependency in pnpm to have HTTP2 support enabled by default.

[zkochan]

@gunta we now use the same package that npm uses to make http requests (https://github.com/npm/npm-registry-client). Do you think this issue is still relevant?

[gunta]

@zkochan Still relevant. They use request which doesn't support http2 neither.

Since it doesn't looks too difficult, we could hack something like this in the meantime: http://stackoverflow.com/questions/37653110/is-there-a-request-like-package-using-http2

[zkochan]

I see. We can try to make a pull request to npm-registry-client. If they won't accept it, we can fork it and use the fork.

[DanielRuf]

Is this still relevant? Node 10 (and probably earlier releases) supports http2 internally (at least "experimental").

https://nodejs.org/api/http2.html#http2_http_2

[aparajita]

@zkochan Do you still want to consider this? Since node-fetch doesn't seem to support http2, it would mean switching to something like fetch-h2.

[zkochan]

sure, would be nice to see if we get a performance boost with that library.

[dmichon-msft]

I've been doing some deep exploration of dependency installation, focusing on the typical CI scenario (lockfile, no cache, no node_modules), and switching to HTTP/2 (using the built-in "node:http2" module) reduced the amount of cumulative CPU time that was spent on network requests by about 70%.

In particular, for the private registry I use (an Azure DevOps Artifacts Feed), each tarball download is actually two network requests: the registry query, which always results in a 303 redirect to an authenticated blob URL, and the actual download, which is across sharded domains and therefore not really a bottleneck.

Some numbers using (cumulative time for async network operations, these numbers get a little weird):
Total packages: ~7000

HTTP/1.1:
Authentication Redirect: 2087s
Download: 188s

HTTP/2:
Authentication Redirect: 614s
Download: 140s

Numbers from a Windows machine with 64 cores, co-located in the same region as the registry.

Edit: On further inspection, HTTP/2 isn't significantly faster than HTTP/1.1 when the latter is configured correctly, unless one schedules a large number of concurrent streams on each connection (higher than the number of sockets that would be used with HTTP/1.1). The 70% reduction was compared to HTTP/1.1 code that had a critical bug that was preventing socket reuse.