Installing 1.3.3 fails with opam

[Kakadu]

But 1.3.2 works fine. Could it be a dune issue?

➜  ~ opam --version                                      
2.1.2
➜  ~ opam install easy-format.1.3.3                         
The following actions will be performed:
  ↗ upgrade easy-format 1.3.2 to 1.3.3

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved easy-format.1.3.3  (cached)
[ERROR] The compilation of easy-format.1.3.3 failed at "dune install -p easy-format --create-install-files easy-format".

#=== ERROR while compiling easy-format.1.3.3 ==================================#
# context     2.1.2 | linux/x86_64 | ocaml-option-flambda.1 ocaml-variants.4.13.1+options | https://opam.ocaml.org#16ff1304
# path        /media/mand/.opam/4.13.1+flambda/.opam-switch/build/easy-format.1.3.3
# command     /media/mand/.opam/opam-init/hooks/sandbox.sh build dune install -p easy-format --create-install-files easy-format
# exit-code   1
# env-file    /media/mand/.opam/log/easy-format-70972-8bc676.env
# output-file /media/mand/.opam/log/easy-format-70972-8bc676.out
### output ###
# /usr/bin/opam var prefix > /opam-tmp/dune_9cfe24_output
# [ERROR] /media/mand/.opam exists, but does not appear to be a valid opam root. Please remove it and use `opam init', or specify a different `--root' argument



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build easy-format 1.3.3
└─ 
╶─ No changes have been performed

[bschommer]

I stumbled upon a similar issue.

[mjambon]

It looks like an opam issue:

# /usr/bin/opam var prefix > /opam-tmp/dune_9cfe24_output
# [ERROR] /media/mand/.opam exists, but does not appear to be a valid opam root. Please remove it and use `opam init', or specify a different `--root' argument

• is the install location /media/mand/.opam intentional? Is there anything weird about it?
• are you able to install other packages than easy-format? (for example opam install dum should be fast, without special dependencies, and easy to undo)

[Kakadu]

It looks like an opam issue: # /usr/bin/opam var prefix > /opam-tmp/dune_9cfe24_output # [ERROR] /media/mand/.opam exists, but does not appear to be a valid opam root. Please remove it and use `opam init', or specify a different `--root' argument - is the install location /media/mand/.opam intentional? Is there anything weird about it?

Yes, it's intentional.

- are you able to install other packages than easy-format? (for example opam install dum should be fast, without special dependencies, and easy to undo)

Yes, many other packages are fine. The previous version of easy-format is fine too.

Message ID: ***@***.***>

My guess is either you use dune in a weird manner, or new dune is weird itself.

[mjambon]

What version of opam and dune are you all using? Here's what I have, which worked fine for upgrading easy-format from 1.3.2 to 1.3.3:

$ opam --version
2.1.0
$ dune --version
3.0.3

[Kakadu]

➜   opam --version
2.1.2
➜   dune --version
3.1.1

[eponier]

We had the problem in Jasmin's CI as easy-format is one of our dependencies. The problem is that the sandbox that opam uses for the building of packages hides from the build process most of the filesystem. /home is always available in the sandbox, so in most systems this means that the opam root is available too. But if the location of the opam root is non standard, it is hidden by the sandbox. And therefore if opam is called inside the sandbox, it does not recognize a valid opam root and fails. Apparently, easy-format.1.3.3 is one of the few packages calling opam inside its build process, thus its installation fails as soon as the location of the opam root is not standard.

As a quick fix, you can use the environment variable OPAM_USER_PATH_RO to make other folders available in the sandbox. For instance OPAM_USER_PATH_RO=$OPAM_USER_PATH_RO:$OPAM_ROOT opam install easy-format should work. But not on nix systems (see the link above). I don't know what change could be implemented in easy-format itself to circumvent this problem.

Note that the future opam 2.2 solves the problem completely since nearly all the filesystem will be available by default.

[mjambon]

@eponier thanks for the explanation. I haven't touched easy-format in a while myself but it's an ordinary ocaml library so it shouldn't be tricky to fix. I'm taking a look.

[mjambon]

I suspect it has to do with the opam file that dune generates from the dune-project file and ends up opam-repository. Here's the diff from the previous version of the opam package:

/opam-repository/packages/easy-format $ diff -u easy-format.1.3.2/opam easy-format.1.3.3/opam
--- easy-format.1.3.2/opam	2021-06-03 15:44:28.897293329 -0700
+++ easy-format.1.3.3/opam	2022-05-12 14:58:32.493030829 -0700
@@ -1,17 +1,4 @@
 opam-version: "2.0"
-build: [
-  ["dune" "subst"] {dev}
-  ["dune" "build" "-p" name "-j" jobs]
-  ["dune" "runtest" "-p" name "-j" jobs] {with-test}
-  ["dune" "build" "-p" name "@doc"] {with-doc}
-]
-maintainer: ["martin@mjambon.com" "rudi.grinberg@gmail.com"]
-authors: ["Martin Jambon"]
-bug-reports: "https://github.com/mjambon/easy-format/issues"
-homepage: "https://github.com/mjambon/easy-format"
-doc: "https://mjambon.github.io/easy-format/"
-license: "BSD-3-Clause"
-dev-repo: "git+https://github.com/mjambon/easy-format.git"
 synopsis:
   "High-level and functional interface to the Format module of the OCaml standard library"
 description: """
@@ -32,15 +19,40 @@
 Atoms represent any text that is guaranteed to be printed as-is. Lists can model
 any sequence of items such as arrays of data or lists of definitions that are
 labelled with something like "int main", "let x =" or "x:"."""
+maintainer: ["martin@mjambon.com" "rudi.grinberg@gmail.com"]
+authors: ["Martin Jambon"]
+license: "BSD-3-Clause"
+homepage: "https://github.com/mjambon/easy-format"
+doc: "https://mjambon.github.io/easy-format/"
+bug-reports: "https://github.com/mjambon/easy-format/issues"
 depends: [
-  "dune" {>= "1.10"}
-  "ocaml" {>= "4.02.3"}
+  "dune" {>= "2.9"}
+  "ocaml" {>= "4.08"}
+  "odoc" {with-doc}
 ]
+build: [
+  ["dune" "subst"] {dev}
+  [
+    "dune"
+    "build"
+    "-p"
+    name
+    "-j"
+    jobs
+    "--promote-install-files=false"
+    "@install"
+    "@runtest" {with-test}
+    "@doc" {with-doc}
+  ]
+  ["dune" "install" "-p" name "--create-install-files" name]
+]
+dev-repo: "git+https://github.com/mjambon/easy-format.git"
 url {
   src:
-    "https://github.com/mjambon/easy-format/releases/download/1.3.2/easy-format-1.3.2.tbz"
+    "https://github.com/mjambon/easy-format/releases/download/1.3.3/easy-format-1.3.3.tbz"
   checksum: [
-    "sha256=3440c2b882d537ae5e9011eb06abb53f5667e651ea4bb3b460ea8230fa8c1926"
-    "sha512=e39377a2ff020ceb9ac29e8515a89d9bdbc91dfcfa871c4e3baafa56753fac2896768e5d9822a050dc1e2ade43c8967afb69391a386c0a8ecd4e1f774e236135"
+    "sha256=eafccae911c26ca23e4ddacee3eaa54654d20f973b8680f84b708cef43adc416"
+    "sha512=611b3124f6a0ec6406b7bda8018a94c9c4a9da9d22495a5c34a6312bf7f0f0607a9529b276f7039ce3f3b15a955dac413d6d1229a55d5ac291302a3ddd5807e5"
   ]
 }
+x-commit-hash: "56c57e69ef067d1cc4e31029d31e77e55b46be95"

The dune install command is new in there. Running it with strace shows that it calls two opam commands:

[pid 1863215] execve("/usr/local/bin/opam", ["/usr/local/bin/opam", "--version", "--color=never"], 0x56064e006aa0 /* 61 vars */ <unfinished ...>
[pid 1863218] execve("/usr/local/bin/opam", ["/usr/local/bin/opam", "var", "prefix"], 0x56064e026b60 /* 68 vars */ <unfinished ...>

Running opam var prefix with a valid opam setup works fine, as expected. If I rename my opam root (which was ~/.opam), I get the dreaded error:

$ opam var prefix
[ERROR] Opam has not been initialised, please run `opam init'

So, that's the command invoked by dune install during the installation of the opam package that results in the error.

We probably can get away with making the opam file in opam-repository not call dune install like for easy-format.1.3.2. However it won't solve the problem for all the other packages that also derive their public opam packages from dune-project. This is a question for dune, opam, and opam-repository maintainers.

For easy-format users who need to get unstuck, try @eponier's suggested workaround (OPAM_USER_PATH_RO=$OPAM_USER_PATH_RO:$OPAM_ROOT opam install easy-format) or pin easy-format 1.3.2 which should work if you use ocaml < 5.

[mjambon]

dune issue: ocaml/dune#5699
opam-repository issue: ocaml/opam-repository#21364

[Kakadu]

@mjambon Thanks for your investigation. Do you think we should add upper bound for dune for easy-format 1.3.3 in the main opam repo?

[mjambon]

@Kakadu I don't know if downgrading dune would be enough. What you could try is:

• clone the easy-format repo
• run manually the build/install commands as found in the opam file (the one in opam-repository)
• same thing with the commands from opam file for 1.3.2: if this works, then we can edit the opam file for 1.3.3 accordingly.

I suspect the issue with dune install isn't new, it's just that we weren't using it before. I could be wrong.

[Kakadu]

It looks like it should be fixed on the opam's side...

[rgrinberg]

Should be fixed in dune 3.2.0