Allow running tests without a bundle

[vinistock]

Running for default frameworks, such as Minitest or Test Unit, is not working if you don't have a Gemfile.

[vinistock]

This problem is not so easy to fix. We don't automatically index gems that aren't in the Gemfile and so we have no way to discover test items because we never indexed Minitest::Test or Test::Unit::TestCase.

We may need to make some different decisions to improve support for tests without a bundle, although I do feel like requiring people to add their test framework to the Gemfile is a fair requirement.

[jconley88]

Is there an appetite for a general solution to determine dependencies without a Gemfile? For instance, this could be done by parsing require statements, filtering non-gems, and then installing the bundle locally so as to emulate version matching without Bundler. I almost always use Gemfiles, but occasionally I will have simple projects/scripts that are at most a handful of files with tests. I don't feel the need to add a Gemfile for these, especially when it's a single-file script. Having better standalone dependency analysis would still be useful and provide a more consistent experience. As a bonus, stdlib requires could be identified and added to the Dependencies tree view.

It seems as if test files aren't currently indexed in the same way as the rest of the project, if at all. If that's the case, then for a general solution to solve this issue, some minimal parsing would need to be introduced for test files.

If ruby-lsp has a strong requirement for a Gemfile, at least at the moment, then disabling code lens test runners entirely, or having an obvious error message when clicked, would improve the current experience.

If you just want this issue to be solved, then how about making a quick, configurable guess as to the test framework by grepping the test directory and appending the determined framework to the composed bundle?

[vinistock]

I think it would be great if we could determine dependencies based on requires, but I'm afraid it might be quite challenging. You have the dynamic aspect of it (like dynamic requires and autoload) and the fact that requires are global. This makes it challenging to figure out the order in which requires happen and the entry points.

Consider a Rails application, where the entry point is define in a gem (the Rails executable that boots the server) and everything is autoloaded dynamically through Zeitwerk. It's quite challenging to build an accurate tree of requires. If you have an idea on how to model this, I'd love to hear it!

Concerning this specific issue, I think we can try a simpler solution first. The reason this fails is because we try to check if the test inherits from Minitest::Test or Test::Unit::TestCase and it ends up failing because we haven't indexed those classes in the first place.

We can probably ignore the fact that we've never seen the classes and just assume that anything inheriting from those are tests that we need to consider. It would not cover the case of a custom parent class, but it would cover simple scripts that have tests with no bundle.

[jconley88]

Thanks for your feedback on requires! I do have some more thoughts on that, but I'll table them here and possibly open a discussion later.

What about adding both minitest and test-unit to the bundled Gemfile if they're available locally? Does that introduce too much complexity for a general solution or cause other issues? To be honest, your suggestion above to check the superclass does seem like it might be easier to implement.

Regardless, if you're not looking for more community input, I'd be happy to attempt a fix.

[vinistock]

I put together a PR with what I meant in #3689. It has the intermediate parent class limitation, but it works for the common case.

I'd avoid adding more gems to the composed Gemfile since that process ends up being the most brittle part of activation. We could end up causing some sort of conflict with the main lockfile and I'm a bit wary of doing so.

Regardless, if you're not looking for more community input, I'd be happy to attempt a fix.

We're always looking for community input! And I appreciate your feedback.