[FR] Improve how we resolve packages between different hosts

[AlexV525]

I'm using our cn mirror (pub.flutter-io.cn) when developing some projects that don't use the mirror and have a lock file with pub.dev resolved.

Now, say we have package:a that has version 1.0.0 and 1.1.0 that can both be resolved with the current project, the lock file specified package:a resolved as version 1.0.0 from pub.dev host. When I run pub get and after it finishes, the resolved version becomes 1.1.0 along with the changed host pub.flutter-io.cn.

Before:

a:
  dependency: "direct main"
  description:
    name: a
    sha256: abcdef
    url: "https://pub.dev"
  source: hosted
  version: "1.0.0"

After:

a:
  dependency: "direct main"
  description:
    name: a
    sha256: fedcba
    url: "https://pub.flutter-io.cn"
  source: hosted
  version: "1.1.0"

I'd raise at least 3 unexpected behaviors:

1. Changing hosts should not lead to version changes, unless the previous ref cannot be resolved in the new host. I wouldn't expect to have an upgrade behavior when I was changing the host.
2. package > description > url should be removed or only present when explicitly specified. The sha256 has already covered the identification of the resolved package, we don't need a host to identify. There is a related improvement that takes place in other popular package management tools: pnpm. See https://github.com/pnpm/pnpm/releases/tag/v10.0.0
3. You don't need package > description > name, it's redundant nowadays.

Expected results

My idea about pubspec.lock:

Host-based ref

pubspec.yaml	pubspec.lock
a: ^1.0.0	a: dependency: "direct main" description: sha256: abcdef source: hosted version: "1.0.0"
a: hosted: https://pub.flutter-io.cn version: ^1.0.0	a: dependency: "direct main" description: sha256: abcdef url: https://pub.flutter-io.cn source: hosted version: "1.0.0"

[jonasfj]

Changing hosts should not lead to version changes, unless the previous ref cannot be resolved in the new host. I wouldn't expect to have an upgrade behavior when I was changing the host.

Changing hosts is the same as changing package names.

The same package name on two different hosts should always be treated as unrelated.

package > description > url should be removed or only present when explicitly specified.

We could consider this. I think if we should do proper mirror support we should do it in a manner that is secure.

---

The fundamental problem here is that pub doesn't have built-in support for mirroring a pub server.

In the ideal world, when using a pub.dev mirror you would specify it using the environment variable PUB_HOSTED_URL, but instead configure it globally on your computer.
The configuration would specify:

• The URI of the server being mirrored https://pub.dev.
• The URI of the mirror https://pub.flutter-io.cn.
• (optionally) public root key for the original server (if not trust-on-first-use, or baked into the Dart SDK, like pub.dev could be).

We could imagine:

• (A) A command like dart pub mirrors add https://pub.dev --mirror https://pub.flutter-io.cn, which write a config file on your machine; OR;
• (B) That the mirror is configured in a mirrors section in pubspec.yaml.

And pubspec.lock would then look the same whether you used the mirror or not.

However, I think allowing such mirror logic is dangerous without some security features in place, specifically we should be able to prove:

• (i) Mirrored packages are the same.
• (ii) Mirrored packages are up-to-date.
• (iii) All packages from the original server are present on the mirror.

(i) aims to ensure that the mirror isn't modifying packages. (ii) aims to ensure that the mirror is not used to facilitate replay-attacks by only serving old versions that have known security vulnerabilities.

I think that (i) requires that pub server have a mechanism for signing packages.
And (ii) requires a signature has a short expiration, or that there is a separate role for signing meta-data indicating when the package was published. The Update Framework has some decent ideas for how do this.

A simple approach would ofcourse be to simply sign the versioning listing response, and let the signature expire within 24 hours. I'm quite sure how scalable such a solution is, but it's possibly an option.

But signing probably requires a bit of work to standup, notably we'd have to make sure that we can have an offline root key, to replace whatever key pub.dev then uses. And probably some very complicated procedures for handling such root key, so we don't loose it 🙈

[jonasfj]

Please upvote this issue, if this is important.

IMO, proper mirror support needs to be able to prove that the mirror produces the same resolutions as the mirrored package repository.

[AlexV525]

However, I think allowing such mirror logic is dangerous without some security features in place

Totally agreed. But isn't the sha256 enough to prove their equality? We may raise warnings that require further user input or exceptions when two packages are not identical.

The fundamental problem here is that pub doesn't have built-in support for mirroring a pub server.

We have some offline discussions in our groups, and we think it would be a bit overkill, but if you want to achieve the goal once for all, it sounds like a better choice, that might decrease the priority however.

[sigurdm]

Yes, for the case where you have an existing lockfile sha256 is enough.

If you use sed to replace the hostnames and then do pub get --enforce-lockfile you should get all the same package archive bits.

What @jonasfj is discussing if a way to guarantee that a new resolution is the same as what you would get on pub.dev

Not sure we want to build extra conveniences around using a mirror for only the purpose of --enforce-lockfile.

[AlexV525]

If you use sed to replace the hostnames and then do pub get --enforce-lockfile you should get all the same package archive bits.

Okay, that sounds like a workaround. But you won't see it from editors AFAIK, and you don't always use cli in these modern years. :)

I'm not sure if pub should take the responsibility to make sure a mirror is valid, or verified as a safe. There are already some warnings when you've specified the host environment (like in Flutter), I'm also not sure if any of the package management tools have done that.

Regardless of switching between hosts, I think the structure of pubspec.lock can be tailored appropriately.