Running Dev Tools in Dune

[rgrinberg]

We’ve done good work on the automatic building and installation of dev tools, but I’d like to see a better design for how editor integration is going to look like. The proposed approach of prefixing every dev tool with dune isn’t going to scale at all.

Editors, various scripts, and other dev tools all expect the dev tools to exist in PATH when they’re running. We don’t want to update every single one of them to have a mode when running under dune’s package management.

We should think about how to make it easier for various dev-tools to integrate with each other rather than creating new barriers. Eventually, the end goal should be that someone should be able to create a dev tool and have dune manage it without modifying dune at all. This is the way to go to encourage the creation of better tooling.

If you’re out of ideas on what to do concretely, I suggest you just follow along in esy steps and provide an explicit dev environment that editors are expected to inherit and dev tools need to run in.

[edwintorok]

FWIW here is how I configured helix in ~/.config/helix/languages.toml:

[language-server.ocamllsp]
command="dune"
args=["tools","exec","ocamllsp"]

As you can see I had to split it up, because the command needs to be an actual executable.

Would be good if I could somehow get ocamllsp into my $PATH. Helix' support for OCaml is otherwise excellent: you don't even need to configure anything, it just works if you've installed ocaml-lsp-server with opam and you've used eval $(opam env) or equivalent. Would be good if dune's editor support is similarly easy to use with zero config changes in the editors themselves.

It would be good if multiple mechanisms were supported:

• dune tools exec .. can be useful if you open multiple files/projects in the same editor and they use different compiler versions: you don't need to relaunch your editor to pick up different env vars, the plugin can be taught to invoke this
• source _build/activate (similar to how you'd source venv/bin/activate with Python. No editor changes needed.
• dune env or dune tools env that prints PATH and other needed settings, similar to how opam env works

[gridbugs]

Would be good if I could somehow get ocamllsp into my $PATH.

In helix do you need to run eval $(opam env) before launching the editor in order for the default ocamllsp config to work? In vim I've found that I need to configure it to launch ocamllsp with the command opam exec ocamllsp so that it picks up ocamllsp exe from the current opam switch as I often forget to run eval $(opam env) before starting the editor. If helix's ocaml support is smart enough to locate the correct ocamllsp in the current opam switch then I think it's reasonable to expect similar support for running ocamllsp via dune to eventually be supported by the editor (or a plugin).

provide an explicit dev environment that editors are expected to inherit and dev tools need to run in.

I don't understand how this will work for editors that are not launched from a terminal. E.g. in vscode and emacs the editor is usually started once and doesn't get the chance to inherit the environment of the project that's currently being edited. I like @rgrinberg's idea from here of making it so that dune tools exec -- <command> would run the command in an environment with all the tools in PATH. Editors could then launch tools with dune tools exec -- <command> and the tools would run in an environment where other tools would be in PATH allowing tools to invoke other tools.

[edwintorok]

In helix do you need to run eval $(opam env) before launching the editor in order for the default ocamllsp config to work?

Yes, my shell config does that automatically.

In vim I've found that I need to configure it to launch ocamllsp with the command opam exec ocamllsp so that it picks up ocamllsp exe from the current opam switch as I often forget to run eval $(opam env) before starting the editor

For Neovim 'mason' supports installing 'ocamllsp' via opam automatically: https://github.com/mason-org/mason-registry/blob/2024-09-30-urban-rocket/packages/ocaml-lsp/package.yaml, so I guess it should be taught about 'dune tools exec' as an alternative 'package manager'.
Although it is a bit awkard because I think it used its own opam switch, so if you want to switch compiler versions you have to override the one it has installed internally, so I'd rather it didn't support opam at all, and let me set up the correct $PATH before invoking it.

I don't understand how this will work for editors that are not launched from a terminal. E.g. in vscode and emacs the editor is usually started once and doesn't get the chance to inherit the environment of the project that's currently being edited

Emacs currently does something like this:

(opam-switch-set-switch (tuareg-opam-current-compiler)))

Calling https://github.com/ocaml/tuareg/blob/master/tuareg-opam.el

So we'll need to figure a way to co-exist with it (depending on whether opam or dune is being used as a package manager for a project you want to use either opam or dune to set the environment). Perhaps the presence of the dune lock dirs can be used for that?

@rgrinberg's idea from here of making it so that dune tools exec -- <command> would run the command in an environment

Yes, that'd be useful too.

Perhaps opam could be taught to emit env settings compatible with dune if it detects dune >= 3.17 and the present of dune lock dirs (similar to how it detects local opam switches by the presence of _opam I guess).
If it detected the presence of dune as a package manager, then opam switch show -s could output __dune__ and opam env --switch __dune__ could invoke dune tools env (or a similar command once it exists).
This would then almost transparently support dune as a package manager without having to update any editor plugins (and of course once the plugins/editors are updated to support dune directly, without opam then that'll work too), and would also support 'opam' and 'dune' to co-exist nicely on the same system, with some projects using only 'opam' and some using 'dune' package management directly.

[gridbugs]

Emacs currently does something like this:

(opam-switch-set-switch (tuareg-opam-current-compiler)))

Calling https://github.com/ocaml/tuareg/blob/master/tuareg-opam.el

So we'll need to figure a way to co-exist with it (depending on whether opam or dune is being used as a package manager for a project you want to use either opam or dune to set the environment). Perhaps the presence of the dune lock dirs can be used for that?

In emacs I currently configure ocamllsp to start from my current opam switch with:

(setq lsp-ocaml-lsp-server-command '("opam" "exec" "--" "ocamllsp"))

This goes within the configuration for the lsp-mode package (see here for the full config). In dune's present state it's already possible to change this to run dune tools exec -- ocamllsp instead and it works, however ocamllsp can't be used to format the file with ocamlformat as ocamllsp can't find the ocamlformat dev tool binary (which would be fixed by having dune tools exec run commands with a modified PATH).

On the topic of co-existing with opam, one use case I'm keeping in mind is using dune to edit a project that doesn't use dune to manage dependencies (or even doesn't use dune at all). So any editor configs where you explicitly start a tool via dune need to still work in cases where ocamllsp/ocamlformat/etc are installed in an opam switch or in your PATH (e.g. system-wide, installed with nix, managed manually).

[rgrinberg]

I don't understand how this will work for editors that are not launched from a terminal. E.g. in vscode and emacs the editor is usually started once and doesn't get the chance to inherit the environment of the project that's currently being edited.

It would be enough if we had a comment such as dune tools env that would return an environment that editors could to set to correctly spawn all tools. In vscode, we already have something that's very similar to this for both opam and esy. Adding similar support to dune should be easy enough.

Both vim and emacs should have similar support via something like direnv for example.

On the topic of co-existing with opam, one use case I'm keeping in mind is using dune to edit a project that doesn't use dune to manage dependencies (or even doesn't use dune at all). So any editor configs where you explicitly start a tool via dune need to still work in cases where ocamllsp/ocamlformat/etc are installed in an opam switch or in your PATH (e.g. system-wide, installed with nix, managed manually).

I think having this feature is fine, but I've yet to understand why dune should care. Ultimately, the choice whether to use dune or opam for developer tools is up to the user to express via their editor. I imagine the user may want the following behaviors:

1. Only use opam
2. Only use dune
3. Prefer dune but fallback to opam
4. Prefer opam but fallback to dune
5. Use nix to manage dev deps
6. Use the system package manager for dev deps

I can see how some of these options should be common among editors, but it really shouldn't be up to date to centralize this logic. If there was a library that every editor plugin used, it would make sense to put such logic there. Our job is just to have a solid contract editors can use to launch dev tools installed by dune.

[edwintorok]

It would be enough if we had a comment such as dune tools env that would return an environment that editors could to set to correctly spawn all tools

Agreed, this is probably the simplest and most flexible (assuming you meant "command", not comment).

[gridbugs]

Providing dune tools env and requiring users to launch terminal-based editors within the correct environment and graphical editors to modify the environment used to launch tools sounds like a reasonable way to fall back to non-dune installations of tools outside of projects using dune package management. It also makes it easy for tools to call each other as their executables will already be in PATH.

However there remains the issue of how the tools will be installed. Currently when you run dune tools exec <tool> dune will lock, build, and install the tool inside the project's build directory, before running it, with any extra arguments passed to the tool itself. If editors are going to invoke (say) ocamllsp by just running ocamllsp from the correct environment then dune won't have the opportunity to install ocamllsp in the current project before running it.

The choices seem to be:

1. Launch dev tools by setting up the environment with dune tools env and then running the command like normal. This requires no extra work to fall back to non-dune instances of tools, and it makes it easy for dev tools to invoke each other as the tools will be in PATH, however it prevents the tools from being installed automatically in the process of running the tools. We would need to provide some other way of installing dev tools in the current project, such as introducing a dune tools install <tool> command.
2. Launch dev tools by running the explicitly via dune with dune tools exec <tool>. This would require fallback behaviour to be built into dune, and makes it more complicated for dev tools to call each other, however it allows dev tools to be transparently installed in the process of invoking them so it becomes harder to accidentally run the wrong version of ocamllsp for example.

Of these two options I have a mild preference for option 1 as it feels conceptually simpler to me, at the cost of requiring the user to explicitly install tools. It also avoids the current issue with dev tools where starting ocamllsp for the first time from your editor causes it to hang for a minute or so while dune compiles ocamllsp which is not great ux.

[Leonidas-from-XIV]

I personally quite dislike the way the PATH is getting messed with and it seems like an endless source of issues (unless you use opam exec -- <tool> which isn't really practical for interactive use). This has spawned a whole cottage-industry of tools managing environments with direnv, the opam shell hook, Python's virtual envs etc and the moment something doesn't remove itself from the environment things go wrong: get fetched to wrong places, installed in wrong places, using wrong compilers and causing confusing bugs etc.

I would much prefer to call a command and then the command knows what environment it is supposed to run in. Unless that tool needs to launch subprocesses there is no need to even modify its environment, and even then just adding it to the environment when launching the tool is a lot less error prone than giving the users some config that they need to correctly pass to their shell somehow.

[rgrinberg]

Launching subprocesses is the norm, not the exception:

1. ocamllsp needs to launch ocamlformat
2. ocamllsp/merlin needs to launch ppx's inside the build environment
3. ocamllsp/merlin need to run dune to get the build configuration

In general, dev processes talking to each other is a good thing. It means the tools are working together.

I think I've injected a bit of confusion into the discussion. In principle, I have no issues with either dune tools env and dune tools exec. They just need to be launched in the correct environment. To summarize, here's my position:

1. Introduce dune tools exec foo. This will launch a dev tool in an environment where it can see all other dev tools and whatever else it may need to communicate with dune.
2. Introduce dune tools env to allow editors/direnv to setup their environment in a way that they can use their existing process launching code.
3. dune tools commands should not have any fallback behavior whatsoever. All fallback should is highly user specific and any kind of attempt to guess the user's intention will be often wrong. If a dune tools command fails to do something, it should just give the user/editor a proper error message and error code so that the caller can decide what the right fallback is.