Jumping to `Core.Md5` fails with `Several source files in your path have the same name`

[voodoos]

Initially reported by @ddickstein.

In the source file:

module M = Core.Md5

Running ocamlmerlin single locate -position 2:18 -filename bin/main.ml <bin/main.ml | jq'.value' returns:

"Several source files in your path have the same name, and merlin doesn't know which is the right one: /.../.opam/5.3.0/lib/core/md5.ml, /.../opam/5.3.0/lib/core/md5.mli, /.../.opam/5.3.0/lib/bin_prot/md5.ml, /.../.opam/5.3.0/lib/bin_prot/md5.mli"

Full log

# 0.03 locate - reconstructed identifier
Core.Md5
# 0.03 locate - from_string
inferred context: module path
# 0.03 locate - from_string
looking for the source of 'Core.Md5' (prioritizing .mli files)
# 0.04 locate - uid_of_path
Unaliased declaration uid: Core.88 -> Core__Md5
# 0.04 locate - find_loc_of_comp_unit
Got the uid of a compilation unit: Core__Md5
# 0.04 locate - find_file_with_path
Try find "Core__Md5"
# 0.04 locate - find_file_with_path
Trying "Core__Md5.cmti"
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/tmp/repro-core/_build/default/bin/.main.eobjs/byte/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/variantslib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/typerep/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/time_now/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/stdio/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/splittable_random/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/sexplib0/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/sexplib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_log/types/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/config/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/parsexp/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/jane-street-headers/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/int_repr/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/gel/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/fieldslib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/validate/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/univ_map/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/heap_block/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/filename_base/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/command/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/base_for_tests/Core__Md5.cmti
# 0.04 locate - load_cmt
Found "Core__Md5" at path "/Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti"
# 0.04 locate - File_switching.move_to
file: /Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti
digest: 897cbc74854e265574e70c752c09cc57
# 0.04 locate - find_source
attempt to find "core/src/md5.pp.mli"
# 0.04 locate - find_source
initial path: "/Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti"
# 0.04 locate - find_all_in_path_uncap
Looking for file "Md5.mli" in path:
/Users/ulysse/tmp/repro-core/bin/Users/ulysse/.opam/5.3.0/lib/ocaml
/Users/ulysse/.opam/5.3.0/lib/base
/Users/ulysse/.opam/5.3.0/lib/base/base_internalhash_types
/Users/ulysse/.opam/5.3.0/lib/base/md5
/Users/ulysse/.opam/5.3.0/lib/base/shadow_stdlib
/Users/ulysse/.opam/5.3.0/lib/base_bigstring
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck/ppx_quickcheck/runtime
/Users/ulysse/.opam/5.3.0/lib/bin_prot
/Users/ulysse/.opam/5.3.0/lib/bin_prot/shape
/Users/ulysse/.opam/5.3.0/lib/core
/Users/ulysse/.opam/5.3.0/lib/core/base_for_tests
/Users/ulysse/.opam/5.3.0/lib/core/command
/Users/ulysse/.opam/5.3.0/lib/core/filename_base
/Users/ulysse/.opam/5.3.0/lib/core/heap_block
/Users/ulysse/.opam/5.3.0/lib/core/univ_map
/Users/ulysse/.opam/5.3.0/lib/core/validate
/Users/ulysse/.opam/5.3.0/lib/fieldslib/Users/ulysse/.opam/5.3.0/lib/gel
/Users/ulysse/.opam/5.3.0/lib/int_repr
/Users/ulysse/.opam/5.3.0/lib/jane-street-headers
/Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel
/Users/ulysse/.opam/5.3.0/lib/parsexp
/Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps
/Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax
/Users/ulysse/.opam/5.3.0/lib/ppx_log/types
/Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness
/Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime
/Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff
/Users/ulysse/.opam/5.3.0/lib/sexplib/Users/ulysse/.opam/5.3.0/lib/sexplib0
/Users/ulysse/.opam/5.3.0/lib/splittable_random
/Users/ulysse/.opam/5.3.0/lib/stdio/Users/ulysse/.opam/5.3.0/lib/time_now
/Users/ulysse/.opam/5.3.0/lib/typerep
/Users/ulysse/.opam/5.3.0/lib/variantslib
# 0.04 locate - find_all_in_path_uncap
Looking for file "Md5.rei" in path:
/Users/ulysse/tmp/repro-core/bin/Users/ulysse/.opam/5.3.0/lib/ocaml
/Users/ulysse/.opam/5.3.0/lib/base
/Users/ulysse/.opam/5.3.0/lib/base/base_internalhash_types
/Users/ulysse/.opam/5.3.0/lib/base/md5
/Users/ulysse/.opam/5.3.0/lib/base/shadow_stdlib
/Users/ulysse/.opam/5.3.0/lib/base_bigstring
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck/ppx_quickcheck/runtime
/Users/ulysse/.opam/5.3.0/lib/bin_prot
/Users/ulysse/.opam/5.3.0/lib/bin_prot/shape
/Users/ulysse/.opam/5.3.0/lib/core
/Users/ulysse/.opam/5.3.0/lib/core/base_for_tests
/Users/ulysse/.opam/5.3.0/lib/core/command
/Users/ulysse/.opam/5.3.0/lib/core/filename_base
/Users/ulysse/.opam/5.3.0/lib/core/heap_block
/Users/ulysse/.opam/5.3.0/lib/core/univ_map
/Users/ulysse/.opam/5.3.0/lib/core/validate
/Users/ulysse/.opam/5.3.0/lib/fieldslib/Users/ulysse/.opam/5.3.0/lib/gel
/Users/ulysse/.opam/5.3.0/lib/int_repr
/Users/ulysse/.opam/5.3.0/lib/jane-street-headers
/Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel
/Users/ulysse/.opam/5.3.0/lib/parsexp
/Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps
/Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax
/Users/ulysse/.opam/5.3.0/lib/ppx_log/types
/Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness
/Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime
/Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff
/Users/ulysse/.opam/5.3.0/lib/sexplib/Users/ulysse/.opam/5.3.0/lib/sexplib0
/Users/ulysse/.opam/5.3.0/lib/splittable_random
/Users/ulysse/.opam/5.3.0/lib/stdio/Users/ulysse/.opam/5.3.0/lib/time_now
/Users/ulysse/.opam/5.3.0/lib/typerep
/Users/ulysse/.opam/5.3.0/lib/variantslib
# 0.04 locate - find_source(Md5)
multiple matches in the source path : /Users/ulysse/.opam/5.3.0/lib/core/md5.ml , /Users/ulysse/.opam/5.3.0/lib/core/md5.mli , /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.ml , /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.mli
# 0.04 locate - find_source
... trying to use source digest to find the right one
# 0.04 locate - find_source
Source digest: 897cbc74854e265574e70c752c09cc57
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/core/md5.ml (5c07d21953176a8bb5701f0361525a6b)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/core/md5.mli (e400638f83e6028dfd761ba11c9373f7)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.ml (83bb88d9053d76a01ca15ae73e8588c6)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.mli (4073e45e18be5ef6d3f65954a6d8a912)
# 0.04 locate - find_source
... found no files with matching digest
# 0.04 locate - find_source
... using heuristic to select the right one
# 0.04 locate - find_source
we are looking for a file named core/src/md5.pp.mli in /Users/ulysse/.opam/5.3.0/lib/core

[voodoos]

I think this is due to the fact that the files are preprocessed. The cmt for core__Md5 actually refers to core/src/md5.pp.ml and it's digest. Since this file is not part of the installed files Merlin cannot find it and has no way to infer the correct one.

[voodoos]

Archeology moment:

This issue was already raised in: #894

And @ddickstein proposed a fix in #1219

It looks like Merlin has become stricter in matching digests since then, which broke the fix in #1219 ?

edit: or more probably merlin fails to find the "exact match" to make a digest here:
https://github.com/ocaml/merlin/blob/main/src/analysis/locate.ml#L319

[voodoos]

Same issue as #1821

[voodoos]

I think I found the issue: we rely on the build directory stored in the cmt file to look for the exact path of an original .ml file when we are looking for its corresponding .pp.ml file. But today this information is non-sensical: When in a test it litterally contains $TESTCASE_ROOT/_build/default and when in a normal project it literrally contains /workspace_root. These obviously do not refer to existing paths and this prevent the heuristic proposed in #1219 from working.

This is due to the fact that OCaml honors the BUILD_PATH_PREFIX_MAP ocaml/ocaml#1515.
Dune use it to sanitize paths, this is enabled since Dune 3.0.

https://dune.readthedocs.io/en/stable/reference/dune-project/map_workspace_root.html

I am not sure yet what is the best way to work around that, especially for installed artifacts.

[voodoos]

I was going to suggest, for monorepos a workaround using (map_workspace_root false) in the dune-project file, but that setting doesn't impact vendored directories...

An option might be to add the build root as a configuration directive sent by dune to Merlin. An other one is to do some parsing based on the cmt paths, but that's not very satisfying.

[liam923]

If I understand this issue correctly, I think the heuristic from #1882 would help here. That heuristic only kicks in when there are multiple files with a matching digest. But maybe it should also kick in if no files have a matching digest?

Concretely, I'm proposing replacing the heuristic on line 444 with the score_file heuristic.

(I have not tested whether this actually works for this case.)

If you agree this would be a good idea, I'm happy to open a pr to do so.

[rgrinberg]

An option might be to add the build root as a configuration directive sent by dune to Merlin. An other one is to do some parsing based on the cmt paths, but that's not very satisfying.

I quite like this approach and perhaps it could be generalized for merlin to be able to reverse all the mangling done with BUILD_PATH_PREFIX_MAP. At least the alternative is far worse: (map_workspace_root false) makes the dune cache completely useless. Note that ocamldebug has the same problem too.

[liam923]

Internally at Jane Street, we're planning on changing the build rules so that the compiler records the relative path to the source file from the project root. And then Merlin can use the SOURCE_ROOT directive (which dune already includes for us internally) to reconstruct the full path.

[voodoos]

perhaps it could be generalized for merlin to be able to reverse all the mangling done with BUILD_PATH_PREFIX_MAP. At least the alternative is far worse: (map_workspace_root false) makes the dune cache completely useless. Note that ocamldebug has the same problem too.

I agree that this sounds like a sane fix for that issue.

Meanwhile I proposed a workaround (in e4cfc06) that simply re-roots the _build folder to the SOURCE_ROOT when possible. It should work in much more cases than the current version (which never does).

@liam923 I am not sure your proposition would fix the issue here, but I happy to try a POC if you have one :-)

[liam923]

@liam923 I am not sure your proposition would fix the issue here, but I happy to try a POC if you have one :-)

Yes I agree, I've realized that I had misunderstood the issue.