[POC] make it possible to _not_ have a baseline for git registries

[strega-nil-re]

RFC: Registries without a Baseline

Inside reMarkable, we've been using vcpkg for around five years now for
external, open source libraries, as well as for some internal libraries.
However, we are now working on using vcpkg for more internal libraries, with
significantly more inter-dependencies. This has resulted in our discovering
serious issues with how the baseline works in a world where vcpkg acts more like
a Cargo than an apt.

The Current Experience

Our current private registry set up is very similar to the public registry; we
keep the baseline up to date, we test all of the libraries at head against each
other, etc. This generally works okay, but it starts to fall apart when we have
dependencies between libraries. It is also deeply confusing to people; overrides
have especially caused significant confusion.

Example 1 – Breaking Changes are Difficult

We use vcpkg to download, build and install our dependencies for all of our
libraries. This means that when there is a breaking change in a
deep-in-the-stack library, it is very frustrating to make that change. As our
example, the deep-in-the-stack library would be scene, and the dependent library
would be gfx; our application is called xochitl.

1. Make the breaking change in scene.
2. Using overlay ports, test the breaking change with gfx; fix the issues.
3. Using overlay ports, test the breaking change and gfx with xochitl.
4. Open a PR (S#1) into scene's git repository
5. Open a PR (V#1) into the vcpkg private registry referencing S#1; ignore the
   CI failures from gfx
6. Open a PR (G#1) into gfx, referencing V#1 as the baseline in the
   vcpkg-configuration.json, so that we can test in gfx's CI
7. Open V#2 on top of V#1 with the changes to gfx from G#1
8. Open a PR (X#1) into xochitl, referencing V#2 (now with G#1 changes), so we
   can test in xochitl's CI
9. Merge S#1
10. Merge V#1, so now we can get the real baseline for G#1
11. Update G#1 with the real baseline from V#1 being merged; merge it once the
    CI passes
12. Update V#2 with the new G#1; merge it once the CI passes
13. Update X#1 with the baseline from V#2 being merged

A lot of this complexity is inherent, but a lot of it is because of baselines.
In my ideal world, we would be using versions, and the result would end up
looking more like:

1. Make the breaking change in scene.
2. Using overlay ports, test the breaking change with gfx; fix the issues.
3. Using overlay ports, test the breaking change and gfx with xochitl.
4. Open a PR (S#1) into scene's git repository
5. Open a PR (G#1) into gfx, depending on the new version of scene, so we can
   test it in CI (note: no baseline change!)
6. Open a PR (V#1) into the vcpkg private registry referencing S#1 and G#1
7. Open a PR (X#1) into xochitl, referencing V#1, so we can test in xochitl's
   CI
8. Merge S#1, G#1, V#1, and X#1

You will notice that without having to update all the baselines all the time,
there's no need to ignore CI so many times, and there's no need to keep updating
PRs with the "correct" baseline.

Example 2 - Overrides/Baselines are confusing

Now, this example might be more of an issue of training, but I think it really
shows how the baseline is confusing. The assumption by many people inside
reMarkable, before I came along and made it more clear how it actually worked,
was that the baseline must be moved forward when updating versions. This means
that whenever there was a new version of one library, the baseline moved the
versions forward for everyone.

We had an issue recently where a team released version 1.3.0 of scene (an
internal library); this was intended as a prerelease version, and was broken
with the version of gfx that was in the registry, but because we had the idea
that "the baseline must always be the latest version", the baseline was also
moved forward. The team involved attempted to "fix" the issue by setting an
override in gfx, "pushing back" the version used to 1.2.4; this, of course, did
not work since overrides are not transitive.

My desired outcome would instead be to use the version>= feature that we have.
My hope is that we would update versions when we need to. This would fit much
more closely with the mental model that people actually have.

This would work inside the current vcpkg, but the insistence on baselines does
result in serious confusion for people; I personally believe that allowing one
to completely remove the baseline would allow us to avoid the confusion
completely.

Example 3 - Baselines are Confusing (part 2. Electric Boogaloo)

Added on 14 October since it happened today.

It is very confusing to people that baselines are not equivalent to version sets.
Just today, someone pushed forward the baseline "to get access" to a version
of our qt port, instead of just updating the "version>=". This happens All The Time.

How to Work Inside the Current Baseline World

The way to make this work in the current world is very simple; you have a
baseline which is never updated past the first version of any library. This
results in the minimum version for everything, and only when one updates
the version>= will you get new versions.

This has two pain points, one minor and one major. The minor pain point is, of
course, when one adds a new port; that means one must also update all the
baselines to one which contain that port. This is frustrating, but can be worked
around.

The major pain point is significantly worse – this means we do not get any
warning or errors when adding a dependency without a version. It just gets to be
the old version without an error.

How I Want it to Work

I want to be able to depend on a registry, and require every single port to be
versioned, including internally to that registry. I want there to be an error if
the baseline doesn't exist, and the port is unversioned.

I do not mind if I have to explicitly ask for no baseline, for example, by doing
"baseline": null, but I want to be able to actually do this.

[BillyONeal]

@vicroms @JavierMatosD @AugP and I discussed this today. However, the folks present had not had time to completely internalize what the impact of this change would be. We're going to discuss next week as well. The observations below are mine, not speaking as the team.

---

• Make the breaking change in scene.
• Using overlay ports, test the breaking change with gfx; fix the issues.
• Using overlay ports, test the breaking change and gfx with xochitl.
• Open a PR (S#1) into scene's git repository
• Open a PR (G#1) into gfx, depending on the new version of scene, so we can
  test it in CI (note: no baseline change!)
• Open a PR (V#1) into the vcpkg private registry referencing S#1 and G#1
• Open a PR (X#1) into xochitl, referencing V#1, so we can test in xochitl's
  CI
• Merge S#1, G#1, V#1, and X#1

Why can't you do this now? As long as the current baseline is older than any of the PRs the suggested version>= constraints will work correctly.

because we had the idea that "the baseline must always be the latest version"

That's not correct though. It's true that baselines are linearized but it is not true that the versions in baselines are always increasing.

The major pain point is significantly worse – this means we do not get any
warning or errors when adding a dependency without a version. It just gets to be
the old version without an error.

This problem is not actually fixed by this proposal as long as there is anything in the transitive closure of dependencies with a version>=.

The minor pain point is, of course, when one adds a new port; that means one must also update all the baselines to one which contain that port.

It's possible that we would be happy to change this behavior without dropping the baselines requirement entirely.

[strega-nil-re]

@BillyONeal

It's possible that we would be happy to change this behavior without dropping the baselines requirement entirely.

If the latter thing was to be an option, we could certainly accept that also.

Why can't you do this now? As long as the current baseline is older than any of the PRs the suggested version>= constraints will work correctly.

This is definitely more of a training and pit of success thing. I think the versioning situation I am requesting is a better situation for internal libraries, and I hope more people use it. I think that giving documentation and help to this situation (and with stuff like depend-a-bot, external support) is beneficial. I am certainly willing to do a decent amount of work here, of course.

This problem is not actually fixed by this proposal as long as there is anything in the transitive closure of dependencies with a version>=.

That is unfortunate.

That's not correct though. It's true that baselines are linearized but it is not true that the versions in baselines are always increasing.

While true, this is an incredibly confusing thing for people in my world. Baselines are consistently the most confusing thing we deal with with regard to vcpkg.

[BillyONeal]

I'm taking some notes from a discussion between @AugP @vicroms @ras0219-msft (partially) @JavierMatosD @MahmoudGSaleh and I. The notes as written are my words though, please take them for the paraphrasing that they are.

I note that while we have said we do think linearizing versions as the baseline mechanism does is how we believe versions should be used by most customers, we don't actually have much feedback from folks pushing the envelope with versioning so I think we should be willing to consider this even though we have rejected similar requests in the past now that this has been out for a while.

If you try something like this internally with Remarkable as a case study that would be helpful in making this be acceptable to us.

@vicroms observes:

• This does not actually change our versioning model so we would be more amenable to it than something like "null baseline means always latest".
• As proposed this will effectively require overrides rather than version>= given the current implementation of versioning. We aren't sure exactly what the implications of trying to change the solver to fix this are.
• [Victor] would be happy with a design requiring overrides for all versions if no baseline is provided.

@ras0219-msft notes that we don't fully understand how this proposal interacts with the:

[...] the result would end up looking more like: [...]

As far as we understand it in step 5,

5. Open a PR (G#1) into gfx, depending on the new version of scene, so we can
   test it in CI (note: no baseline change!)

it isn't clear how the new version of scene you want to test with would be found as it is not yet present in the registry. It seems you would need an overlay port for this, and if that's the case there's no reason to make a baseline change.

Just today, someone pushed forward the baseline "to get access" to a version
of our qt port, instead of just updating the "version>=". This happens All The Time.

Perhaps the problem is more a naming issue and "baseline" should be "version>=".

@BillyONeal observes:

At a minimum we're going to want explicit tests of this feature's behavior, and proposed docs changes to github.com/MicrosoftDocs/vcpkg-docs before we would push the merge button. However given that I think you were asking if we would reject this out of hand before doing that additional work, I hope the replies above make some sense. My main concerns remain the parts of the scenarios you want to work that aren't actually made to work by this change.

@vicroms will add a reply under this one with additional thoughts

[strega-nil-re]

I would love to hear from @vicroms, but I did want to say one thing:

As proposed this will effectively require overrides rather than version>= given the current implementation of versioning. We aren't sure exactly what the implications of trying to change the solver to fix this are.

We're hoping to also make all of our internal libraries use version>=, so given this implementation you do not need overrides. Note that the public registry is still going to have a baseline.