How to make "`b.opam` depends on external `foo` package which depends on `a.opam`" work?

[sir4ur0n]

Describe the bug
We have a repo with 2 packages a and b, where b depends transitively on a via an intermediary package foo.
We don't manage to provide a Nix shell with opam.nix.

To Reproduce

• in repo bar, have 2 opam files:
  • a.opam
  • b.opam which depends on foo
• in repo foo, have 1 opam file foo.opam which depends on a
• use opam-nix to provide a Nix shell in repo bar for both a and b

I tried to add this overlay:

final: prev: {
      a = prev.a.overrideAttrs
        (a: {
          src = builtins.fetchGit ./.;
          doCheck = false;
        });
    }

This makes the Nix shell succeed to evaluate, however dune build fails with something like

Error: Conflict between the following libraries:
- "a" in _build/default/a/lib
- "a" in
  /nix/store/q1c4vacvhql0vc3ry9gvr3xjnqrzml2f-a-0.19.6/lib/ocaml/5.2.0/site-lib/a
  -> required by library "foo" in
     /nix/store/1i05pbq3j8swqz5s9pxhkzs4yf03dlqd-foo-1.1.6/lib/ocaml/5.2.0/site-lib/foo
  -> required by library "b" in
     _build/default/b/lib
-> required by _build/default/b/tests/b.exe
-> required by alias b/tests/all
-> required by alias b/default
-> required by alias default in dune:20

Expected behavior
Some way to get a nix-shell for both a and b

[greedy]

Does this strategy work with regular opam? I would think you’d have the same problem with that setup. In other words I don’t think this would be a opam-nix specific issue.

That said,your best bet is probably to have dune building a, b, and foo together. You can check out foo inside your a+b tree and optionally set it as a vendored directory or have foo in a sibling directory and use a dune-workspace file in the common parent to set the workspace root so dune sees a, b, and foo together.

[balsoft]

I think another solution would be to filter out the opam file for a.opam when you're building the b package somehow.

[sir4ur0n]

Does this strategy work with regular opam?

@greedy yes it does: as much as I dislike this setup, this is the current setup we have on some of our projects, and devs who use opam make it work.

That said,your best bet is probably to have dune building a, b, and foo together. You can check out foo inside your a+b tree and optionally set it as a vendored directory or have foo in a sibling directory and use a dune-workspace file in the common parent to set the workspace root so dune sees a, b, and foo together.

The problem is that it changes the setup for non-nix users. A key benefit of using opam-nix for us is precisely to have non-invasive Nix integration.

I think another solution would be to filter out the opam file for a.opam when you're building the b package somehow.

@balsoft if I correctly understood your suggestion: the problem is that we would like to have a single nix shell to build all packages. With your suggestion, for nix shells, we would need to have 2 nix shells (or a flag parameter to decide which shell we want to land in), right?

For now we use a combination of cleanSourceWith + gitignore.lib.gitignoreFilterWith to ignore the problematic b.opam (and we manually change the value passed to cleanSourceWith when we need to work on package b), but we would have liked a single shell without manual changes to work seamlessly and be able to just run dune build on the whole repo.

[balsoft]

@balsoft if I correctly understood your suggestion: the problem is that we would like to have a single nix shell to build all packages. With your suggestion, for nix shells, we would need to have 2 nix shells (or a flag parameter to decide which shell we want to land in), right?

Not necessarily

For now we use a combination of cleanSourceWith + gitignore.lib.gitignoreFilterWith to ignore the problematic b.opam (and we manually change the value passed to cleanSourceWith when we need to work on package b), but we would have liked a single shell without manual changes to work seamlessly and be able to just run dune build on the whole repo.

I think I'm proposing the inverse: to override the b package's source to exclude the source files for a entirely (as it should be provided by opam-nix as a dependency). Something like b = b.overrideAttrs (oa: { src = lib.fileset.toSource { root = ./.; fileset = oa.filesets.difference ./. ./a; }; });

[sir4ur0n]

I tried

b = b.overrideAttrs (oa: { src = lib.fileset.toSource { root = ./.; fileset = lib.fileset.difference ./. ./a; }; });

Notice this diff with what you suggested, otherwise I had an error that oa did not have field filesets:

- oa.filesets.difference
+ lib.fileset.difference

In any case it doesn't work, I have the same error as in the original issue:

error: builder for '/nix/store/ng5lv10h9a3fahjz5wi96jiqdm25qy12-a-0.20.1.drv' failed with exit code 1;
       last 12 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/nf6br8mcd00nc4hjllnnn0jclbchbnkk-4x0xpyackj6jrq3473ivnyw24kah9msn-source
       > source root is 4x0xpyackj6jrq3473ivnyw24kah9msn-source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > patching script interpreter paths in .
       > Running phase: buildPhase
       > File ".", line 1, characters 0-0:
       > Error: There is no dune-project file in the current directory, please add one
       > with a (name <name>) field in it.
       > Hint: 'dune subst' must be executed from the root of the project.
       For full logs, run 'nix log /nix/store/ng5lv10h9a3fahjz5wi96jiqdm25qy12-a-0.20.1.drv'.
error: 1 dependencies of derivation '/nix/store/96rxcrjm2l8mxs5nlj1l3csqv2455v0c-foo-1.1.6.drv' failed to build
error: 1 dependencies of derivation '/nix/store/av9p8gg9s1mqfn7hq32ahfrvdjls7rdg-nix-shell-env.drv' failed to build

[balsoft]

In any case it doesn't work, I have the same error as in the original issue:

Ah, you may also want to add your original override

final: prev: {
      a = prev.a.overrideAttrs
        (a: {
          src = builtins.fetchGit ./.;
          doCheck = false;
        });
    }

[sir4ur0n]

Then I still get the error Conflict between the following libraries 😞

[balsoft]

Hey, sorry, I forgot about this :) If you're still interested, I can work on this a bit more if you provide a simple reproducer which I can test on.

[sir4ur0n]

Hey, no worries!
In the meantime the person responsible for this design exploded it (because it also breaks with the new dune pkg), so we no longer have this scenario (thus limitation).
If you want to close this issue, feel free!

[balsoft]

I'll keep this issue open for posterity; if someone else needs a similar setup, send me a reproducer and I will try to fix it. Although I also feel that it's a pretty hacky way to set things up all things considered.