`NoSuchFileException` on `bloop compile`

[harpocrates]

Here is the error message at the CLI running bloop compile (slightly redacted because this is from a closed source enterprise codebase):

Compiling project1 (77 Scala sources)
[W]  [E-1] one deprecation (since 23.31); re-run with -deprecation for details
Compiled project1 (18432ms)
[E] Unexpected error when compiling project2: java.nio.file.NoSuchFileException: /tmp/.bloop/project2/project2/out/bloop-bsp-clients-classes/classes-CustomBloopClient-GO6XrTalT5GbywKxOhpJew==/com/project2/myPackage/MyClass$$anonfun$$lessinit$greater$42.class
[E]     at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
[E]     at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
[E]     at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[E]     at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:248)
[E]     at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
[E]     at java.base/java.nio.file.Files.delete(Files.java:1152)
[E]     at bloop.Compiler$$anon$3.$anonfun$trigger$18(Compiler.scala:699)
[E]     at bloop.Compiler$$anon$3.$anonfun$trigger$18$adapted(Compiler.scala:691)
[E]     at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[E]     at bloop.Compiler$$anon$3.$anonfun$trigger$17(Compiler.scala:691)
[E]     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[E]     at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81)
[E]     at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[E]     at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[E]     at monix.eval.internal.TaskShift$Register$$anon$1.run(TaskShift.scala:65)
[E]     at monix.execution.internal.InterceptRunnable.run(InterceptRunnable.scala:27)
[E]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[E]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[E]     at java.base/java.lang.Thread.run(Thread.java:840)
[E] Failed to compile 'project2'

This is almost certainly related to the fact that we have a second Bloop client (CustomBloopClient) also issuing sometimes concurrent compilation requests. My understanding is that Bloop should handle that sort of concurrent request just fine, so this should be a bug.

AFAICT, seems like this is a case where Bloop is expecting to copy some compiled class out of the class directory configured for the other client in order to reuse it in the latest CLI-requested compilation, but the file doesn't actually exist in the output directory of the other client. The other custom client is just communicating over BSP with Bloop, basically only really requesting compilation and then making classloaders to hot-load the compiled outputs.

[wiacekm]

Hi @harpocrates,

I've started looking into this issue.
In order to better understand and investigate the problem, I’d appreciate your help with a few clarifying questions:

1. How frequently does this error occur? Is it intermittent, or does it happen consistently under certain conditions?

2. Do you have any steps or context that could help us reproduce this failure locally? For example, specific commands, project state, or actions leading up to the issue.

3. When you refer to the “second Bloop client (CustomBloopClient),” do you mean that multiple clients (e.g. CLI and a BSP client) might be triggering compilation concurrently for the same project? Or are they working on separate subprojects?

4. Is it possible that some of the consecutive compilations are either failing and then succeeding (or vice versa)? This could help determine whether there's a timing or cleanup issue between runs.

Any additional details would help a lot. If any of my question is unclear Let me know.

[harpocrates]

1. How frequently does this error occur? Is it intermittent, or does it happen consistently under certain conditions?

It is intermittent. Across a user-group of about 10 people, it probably occurs about once every couple of days that we know of. It might be more often, but we're often missing a good detection mechanism to diagnose.

2. Do you have any steps or context that could help us reproduce this failure locally?

Unfortunately no - because it triggers so rarely and we've not been able to get anything more specific here. It occurs across different projects. AFAICT, usually the lead up to this occurring is regular Scala development during which the compile calls will be made to the BSP by both clients.

3. When you refer to the “second Bloop client (CustomBloopClient),” do you mean that multiple clients (e.g. CLI and a BSP client) might be triggering compilation concurrently for the same project? Or are they working on separate subprojects?

Yes, I mean concurrent triggering compilation for the same project. Although they go through different codepaths, I think both the Metals and the custom bloop client indirectly trigger compile on save actions, so they will almost always start around the same time.

4. Is it possible that some of the consecutive compilations are either failing and then succeeding (or vice versa)? This could help determine whether there's a timing or cleanup issue between runs.

I'm not sure I understand the question...

Since this is generally occurring as part of a project development, I would probably expect that the project is periodically flipping back and forth between compiling and non-compiling states.

If you have ideas for other things we could watch for, we could probably add telemetry. Right now, part of the problem is that usually when the issue occurs, the user doesn't notice - they're typing away in their IDE and only gradually notice that the Metals experience is degrading. By the time I get to seeing the issue, I'll run something like bloop compile and notice the bad state.

[wiacekm]

Thanks for clarification.
I have two more questions:

1. what is the size of the project? ex. numer of modules, classes, objects, etc.
2. do you use best-effort mode for this project

[harpocrates]

1. what is the size of the project? ex. numer of modules, classes, objects, etc

It should be on the order of a hundred to several hundred source files, each file usually has just an object/class pair, though there are considerably more classfiles (the one that is missing here maps onto a closure generated by a macro). I'll collect more information next time this is hit (different projects have hit it)

2. do you use best-effort mode for this project

Not sure if the setting is enabled in the Bloop config, but this is compiling against Scala 2.12.17, so is best-effort mode even a thing for that Scala version?