Please keep around old versions of dune

[RalfJung]

It seems old versions of dune are now being (automatically?) removed from the repo. This has caused us some head-scratching and even now that we know this is happening, has adverse side-effects for our CI setup: we'd like to pin an old version of dune, since dune gets updated a lot and each time it does, the world needs to be rebuilt, thus costing a lot of CI time. (Or, at least coq/rocq needs to be rebuilt, which is the one thing we actually use opam for.) But with old versions of dune being pruned from the repo, this has now become harder -- we may have to resort to copying the dune package definition into our own repo or so, to avoid these slow rebuilds.

[hannesm]

Dear @RalfJung,

thanks for opening this issue. For now, we restored the dune 3.17.2 package in the opam-repository, so your builds will succeed again. That version of dune is also marked to not being removed from the opam-repository, so subsequent archivals based on the maintenance-intent will leave it untouched.

There is still a question in my mind: The archival was announced on https://discuss.ocaml.org/t/opam-repository-archival-phase-3/ - but it seems this didn't catch your attention. What could be done to improve this? An idea from @raphael-proust is that we should include the steps needed to test the opam-repository after archival in the announcement (would that catch your attention, would you spend time to test it?). Should we announce such archivals elsewhere (for example on the caml-list?)? Should there be tooling taking a .opam.locked file and adding the x-maintained field to these (including a comment about where it is used)?

The idea behind the archival process is to automatically (based on the x-maintenance-intent) archive every few months opam packages, but of course we would like to not break any of your CI systems (see https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md for the plan, based on the discussions in #23789 and various video meetings). So we're curious what can we do to improve the communication? We already planned for the possibility to mark packages with x-maintained: true so they're not being removed (for the exact use case, that these are used by external/private organizations/CI systems that pin to specific versions.

In the meantime, how to fix this:

• One path is to add the archive repository to your systems (but of course then you won't benefit from reduced solver times) (opam repo add archive git+https://github.com/ocaml/opam-repository-archive);
• another is to pin to a specific commit of the opam-repository (opam repo set-url default git+https://github.com/ocaml/opam-repository.git#$commit-hash);
• or take the time to update your lock file / version pins.

//cc @liyishuai and @Lysxia who worked on the rocq issue; also @7h3kk1d and @cyrus- for hazel, and @Blaisorblade for their internal systems, @LasseBlaauwbroek and @bertrand-sb may as well have input

[Zimmi48]

Should we announce such archivals elsewhere (for example on the caml-list?)?

If you want the announcements to be noticed by the Rocq community, the best choice would be to announce them as well on the Rocq Discourse (https://discourse.rocq-prover.org/). That being said, I don't know if we would have realized the impact on the Docker-Rocq images.

The maintainers of Docker-Rocq are @erikmd and @himito, so perhaps they will be able to tell which solution they prefer.

[bertrand-sb]

Hello, it's not a big deal overall. I think I understand the rationale behind the change here and I now have a workaround. That being said, maybe I can bring some perspective as a user. By the way, obviously I'm very thankful for the work all the maintainers are doing to keep the repository working and make it better.

The update caught us off guard. Our CI relies on a few outdated packages, one of which is dune-configurator < 0.17.0. The reason is mainly for compatibility with older build environments like RHEL 8, but mostly engineering resources that would be required to upgrade that we don't currently have. Of course, ideally we wouldn't rely on those older versions, but that's the reality I live in.

As a user, the breakage was unexpected in two ways:

• No easy-to-find announcement. I can't really keep up with what's going on in https://discuss.ocaml.org/ unfortunately.
  • To this day, there doesn't seem to be easy-to-find official instructions on how to download those older versions of OCaml packages.
• In general, it's unusual for versions of packages to disappear from a package registry. When it happens (e.g. in Python or Rust), it's usually for security reasons (e.g. a malicious package) or because the author pushed a broken version, so they yank it from the registry to push another version. But never have I seen such archiving being done in any other registry before.
  • I'm not suggesting the opam registry should work differently, but rather highlighting that users, especially those familiar with other languages, may be even more surprised by the archival.

With that said, maybe some of these could improve the situation:

• An announcement and workaround on https://opam.ocaml.org/blog/.
• Official instructions on how to access older packages (in the manual? or built in opam?)
  • And I mean user instructions, not package maintainer instructions.
• More ambitious: In the future, keep more packages in the main registry but somehow reduce their impact on opam resolution and OCaml CI?
  • Really unsure about that. I just know that the quality of resolution and tests in the registry is quite high and thus costly, but maybe we don't need that quality for 100% of packages.

(It would have been ideal if opam could have warned us ahead of time that the packages we depend on are about to be removed from the registry, but that would have been non-trivial to implement in opam and it's too late for that anyway.)

I hope this helps!

[RalfJung]

Thanks for the long and thoughtful reply!

There is still a question in my mind: The archival was announced on https://discuss.ocaml.org/t/opam-repository-archival-phase-3/ - but it seems this didn't catch your attention. What could be done to improve this?

That's a very fair question. I have basically no contact with the ocaml community and do not frequent the ocaml forum or mailing list, I am just a Rocq user. However, I do happen to have https://opam.ocaml.org/blog/feed.xml subscribed, so an announcement there would have gotten my attention.

But this may be an interesting data point for you: there is some number of opam users that are not ocaml users.

What would have aided in debugging is if opam didn't just say
"[ERROR] Package dune has no known version 3.13.1 in the repositories"
but somehow pointed out that the version has been archived, with a url providing context. I don't know whether there is a practical way to achieve this.
(Due to the way our CI works, the command that fails this way is an opam pin of the specific version. So it may also be the case that there would have been a better error if the missing version would have surfaced in a different way, e.g. via a missing dependency.)

That being said, I don't know if we would have realized the impact on the Docker-Rocq images.

To be clear, we are not using the docker-rocq images, we have our own CI. I was not aware that docker-rocq was also affected -- in fact, I was thinking maybe we should finally migrate to docker-rocq to avoid such issues...

In general, it's unusual for versions of packages to disappear from a package registry. When it happens (e.g. in Python or Rust), it's usually for security reasons (e.g. a malicious package) or because the author pushed a broken version, so they yank it from the registry to push another version. But never have I seen such archiving being done in any other registry before.

Yes, this has probably also contributed to my initial confusion. Also, the versions still showed up locally after they were missing on CI, and I thought I had done an opam update, but maybe I did not, or maybe the update took a bit longer to propagate to my local mirror if that's a thing (I am traveling and hence geographically quite far away from the CI machine right now).

[7h3kk1d]

also @7h3kk1d and @cyrus- for hazel may as well have input

For now we've just added the archive repository.

There conflicts due to the archived packages that seem to require maintainer help to resolve. e.g. ppx_deriving_qcheck requires ppx_lib >= 0.36.0 while ppx_js_style requires ppx_lib < 0.36.0. In the archive we use ppx_deriving_qcheck 0.6 which allows for ppx_lib < 0.36.0. By only keeping the most recent versions of packages there doesn't seem to be enough time for upates to propagate through the system like this.

The dependency resolution algorithm was also a bit noisy to track down the actual issue. It seemed to eagerly lock down some versions making the actual conflict hard to spot.

opam install output

[ERROR] Package conflict!
  * No agreement on the version of ocaml-base-compiler:
    - (invariant) → ocaml-base-compiler = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-base-compiler (< 3.07+1 | = 3.08.0 | = 3.08.1
      | = 3.08.2 | = 3.08.3 | = 3.08.4 | = 3.09.0 | = 3.09.1 | = 3.09.2 | =
      3.09.3 | = 3.10.0 | = 3.10.1 | = 3.10.2 | = 3.11.0 | = 3.11.1 | = 3.11.2 |
      = 3.12.0 | = 3.12.1 | = 4.00.0 | = 4.00.1 | = 4.01.0 | = 4.02.0 | = 4.02.1
      | = 4.02.2 | = 4.02.3 | = 4.04.0 | = 4.04.1 | = 4.04.2 | = 4.05.0 | =
      4.06.0 | = 4.06.1 | = 4.07.0 | = 4.07.1 | = 4.08.0 | = 4.08.1 | = 4.09.0 |
      = 4.09.1)
    You can temporarily relax the switch invariant with `--update-invariant'
  * No agreement on the version of ocaml:
    - (invariant) → ocaml-base-compiler = 5.2.0 → ocaml = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml < 4.09.0
  * No agreement on the version of ocaml-base-compiler:
    - (invariant) → ocaml-base-compiler = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → ocaml-base-compiler (=
      4.10.0 | = 4.10.1 | = 4.10.2 | = 4.11.0 | = 4.11.1 | = 4.11.2)
  * No agreement on the version of ppxlib:
    - deps-of-hazel → ppx_deriving_qcheck → ppx_deriving >= 6.1.0 → ppxlib >=
      0.36.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0
  * Incompatible packages:
    - (invariant) → ocaml-base-compiler = 5.2.0
    - (invariant) → ocaml-system = 5.2.0
  * Incompatible packages:
    - (invariant) → ocaml-system = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-base-compiler (< 3.07+1 | = 3.08.0 | = 3.08.1
      | = 3.08.2 | = 3.08.3 | = 3.08.4 | = 3.09.0 | = 3.09.1 | = 3.09.2 | =
      3.09.3 | = 3.10.0 | = 3.10.1 | = 3.10.2 | = 3.11.0 | = 3.11.1 | = 3.11.2 |
      = 3.12.0 | = 3.12.1 | = 4.00.0 | = 4.00.1 | = 4.01.0 | = 4.02.0 | = 4.02.1
      | = 4.02.2 | = 4.02.3 | = 4.04.0 | = 4.04.1 | = 4.04.2 | = 4.05.0 | =
      4.06.0 | = 4.06.1 | = 4.07.0 | = 4.07.1 | = 4.08.0 | = 4.08.1 | = 4.09.0 |
      = 4.09.1)
  * Incompatible packages:
    - (invariant) → ocaml-base-compiler = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-variants
  * Incompatible packages:
    - (invariant) → ocaml-system = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-variants
  * Incompatible packages:
    - (invariant) → ocaml-base-compiler = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → dkml-base-compiler
  * Incompatible packages:
    - (invariant) → ocaml-system = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → dkml-base-compiler
  * Incompatible packages:
    - (invariant) → ocaml-system = 5.2.0
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml < 4.13 → ocaml-base-compiler (=
      4.10.0 | = 4.10.1 | = 4.10.2 | = 4.11.0 | = 4.11.1 | = 4.11.2)
  * Missing dependency:
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml-migrate-parsetree → ocaml-variants (= 4.08.0+beta2 | =
      4.08.0+beta3)
    no matching version
  * Missing dependency:
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-variants → ocaml-beta
    unmet availability conditions: 'enable-ocaml-beta-repository'
  * Missing dependency:
    - deps-of-hazel → ppx_yojson_conv >= v0.17.0 → ppx_js_style >= v0.17 → ppxlib <
      0.36.0 → ocaml < 4.10 → ocaml-variants → system-msvc
    unmet availability conditions: 'os = "win32"'

No solution found, exiting

With regards to notification. It would be nice if we had a warning in our builds showing that some of the packages we depended on were up for archiving.

[hannesm]

@RalfJung interesting -- so do you use another dune version you'd like to keep? I thought the rocq-base would be the thing you're using as well.

[pi8027]

Should we announce such archivals elsewhere (for example on the caml-list?)?

If you want the announcements to be noticed by the Rocq community, the best choice would be to announce them as well on the Rocq Discourse (https://discourse.rocq-prover.org/). That being said, I don't know if we would have realized the impact on the Docker-Rocq images.

The maintainers of Docker-Rocq are @erikmd and @himito, so perhaps they will be able to tell which solution they prefer.

It's not too late to make such an announcement on the Rocq Discourse. I guess that many Rocq users are still not aware of the issue and will see similar issues in the coming weeks. Also, it would be nice if the announcement included a recommended way to fix it (either opening an issue to unarchive some packages or locally adding opam-repository-archive as an OPAM repository).