Slowness in NeoVim

[vinistock]

Context

We have some reports of slowness and maxing CPU for NeoVim users in #2446. I'm splitting this into a separate issue since the root cause of #2446 was incorrect handling of UTF-8 code unit positions and that was fixed. The root cause for whatever is happening now must be different.

How to help diagnose

Tracing

The initial step for diagnosing is figuring out when the slowness happens. In other words, which actions lead to the LSP being slow.

Enabling verbose or messages trace is very helpful because it shows all messages exchanged between editor and server and how long each one took. This can help us understand which request is causing the LSP to take longer or which one is putting it in a bad state.

I don't know how to turn it on for NeoVim though. In VS Code, it's the "ruby lsp.trace.server": "messages" setting, which is automatically created by the official LSP client package.

Custom diagnose state request

The Ruby LSP exposes a custom LSP request called rubyLsp/diagnoseState, which will return some internal information, like the current backtrace of the worker thread and the internal representation of each document.

1. If the backtrace is constantly in a place that isn't here, that is indication that the worker might be getting stuck. The linked spot in the code does not indicate that the worker is stuck because it is simply waiting for jobs to be pushed into the work queue. If there aren't any jobs to be processed, it will hang in pop until there's something to do
2. If the document representation does not match what you currently have in the editor, then we have a bug that's causing document corruption. The most important information here is understanding how to get the documents corrupted. For example, does it involve inserting or replacing multibyte characters?