Marking Package Compatibility With Dune Package Management

[rgrinberg]

Some opam packages are incompatible with dune's package management. Specifically, the OCaml compiler itself is a notable example due to some issues in the configure script and the lack of relocation. As well as in general, opam allows package builds to do things will always be prohibited by dune. Therefore we need to have a way to work around such temporary and sometimes permanent limitations. Our solution is to introduce our own opam repository overlay that will be available by default to user's of dune's package management. The overlay should give us the following two capabilities:

• Apply patches to existing packages to make them compatible with dune builds.
• Mark packages as being incompatible with dune's package management.

[emillon]

I agree that there needs to be a way to override the definitions from opam, but I don't think that this is something that should be done by dune itself by default.
If the end goal is to have working build plans for all packages (and I think we want to get there), then overrides are as you said just temporary things.

Overlays are not easily expressed as an extra opam-repository, as we discovered while developing opam-monorepo. The reason is that because opam merges all repositories in one, with priority only used when there's a conflict. This model is has two problems:

• it is a bad idea to use the same version number for different metadata, because version numbers name this metadata. So we want to avoid conflicts anyway. This is why overlays for opam-monorepo used +dune version numbers.
• because of the flat namespace, recent versions (incompatible with the tool) are preferred to older versions. For example, 0.24 < 0.24+dune < 0.24+dune2 < 0.25`. This makes locking unreliable. A different encoding could be used (for example using epochs) but in general, the set of version compatible with the tool isn't convex.

There are good ideas to be taken in the existing ecosystems:

• In Ruby, it's possible to express "use the default sources for all packages, but for this dependency, use this repository". In our case, instead of mapping to repository merging, we would remove all upstream versions of the overridden package, and only have the ones from the "overlays" (to get back to the example before: opam would only see 0.24+dune and 0.24+dune2, and pick the latter, not 0.25).
• In Nix, it's possible to directly express changes to build instructions without referring to a hosted metadata file. Concretely, today with opam it is necessary to create an opam repository and host it in a git repository somewhere. In practice, we could have shortcuts to embed the contents of the overridden opam file directly in a file in the workspace source code, or a way to apply patch file, or even the equivalent of nix's substituteInPlace.

Regarding the last point: for dune-universe/opam-overlays, we worked under the assumption that we would centralize all patches in a single point and maintain that from the community. This works once the patches are in the state where we agree that there's the right ones, and that it's clear that they won't make their way upstream. But it makes it a lot harder to iterate on that. In the meantime, I believe that it's useful to have a way to to embed patches.

[rgrinberg]

I agree that there needs to be a way to override the definitions from opam, but I don't think that this is something that should be done by dune itself by default.

What's the alternative to doing it by default? Some packages don't build with dune and we need a way to avoid them otherwise users will generate build plans that don't work and conclude that dune is buggy.

If the end goal is to have working build plans for all packages (and I think we want to get there), then overrides are as you said just temporary things.

Yes, that's the goal, but goals have priorities and varying levels of difficulty. The fact is that dune's definition of a valid package build is by necessity stricter than opam's. This higher standard is going to disqualify a subset of packages. The important ones we'll need to address (hence this issue) while the less important ones we can just mark as broken and move on to more important work.

Overlays are not easily expressed as an extra opam-repository, as we discovered while developing opam-monorepo. The reason is that because opam merges all repositories in one, with priority only used when there's a conflict. This model is has two problems:

I think your experience with duniverse is possibly over prescribing what our overlays should look like. We do not necessarily need to rely on opam's priority model, we could create our own metadata scheme if it works better. Here are the only properties that we need:

1. The metadata has to be updated without releasing dune.
2. The metadata has to come from some source that can be shared and tracked (i.e. a git repo).
3. We need a way of marking existing packages as broken with dune
4. We need a way to augment existing packages with patches to make them work with dune

We do not necessarily need to introduce our own versions of packages. Any scheme that addresses the above should work.

In Ruby, it's possible to express "use the default sources for all packages, but for this dependency, use this repository". In our case, instead of mapping to repository merging, we would remove all upstream versions of the overridden package, and only have the ones from the "overlays" (to get back to the example before: opam would only see 0.24+dune and 0.24+dune2, and pick the latter, not 0.25).

Yes, I think that we'll need something that accomplishes this.

In Nix, it's possible to directly express changes to build instructions without referring to a hosted metadata file. Concretely, today with opam it is necessary to create an opam repository and host it in a git repository somewhere. In practice, we could have shortcuts to embed the contents of the overridden opam file directly in a file in the workspace source code, or a way to apply patch file, or even the equivalent of nix's substituteInPlace.

We already have the features to do something like that. The issue is that we want these things to be available by default without any intervention from the user. By default things should just work.

Regarding the last point: for dune-universe/opam-overlays, we worked under the assumption that we would centralize all patches in a single point and maintain that from the community. This works once the patches are in the state where we agree that there's the right ones, and that it's clear that they won't make their way upstream.

I'm working under the same assumptions for dune, except that the set of packages and patches we'll need to maintain will be small and will hopefully shrink with time. But I also except some tail of very rarely used packages will permanently remain broken with dune so we'll just need a way to avoid those.

[rgrinberg]

This was discussed with the opam team, and the main goal of the overlay is to mark packages as incompatible with opam packages. According to @kit-ty-kate, such metadata can be pushed upstream to the opam-repository right away. Thus, we pivot this ticket to designing the metadata and the process needed for this.