Add `solargraph profile` command using vernier #1071

lekemula · 2025-09-06T19:16:02Z

Hi,

I thought of adding a profiling command for convenience, to avoid the hassle of configuring the IDE, switching gem versions, restarting, and all that:

❯ solargraph --help profile
Usage:
  solargraph profile [FILE]

Options:
  -d, [--directory=DIRECTORY]                     # The workspace directory
                                                  # Default: .
  -o, [--output-dir=OUTPUT_DIR]                   # The output directory for profiles
                                                  # Default: ./tmp/profiles
  -l, [--line=N]                                  # Line number (0-based)
                                                  # Default: 4
  -c, [--column=N]                                # Column number
                                                  # Default: 10
  -m, [--memory], [--no-memory], [--skip-memory]  # Include memory usage counter
                                                  # Default: true

Profile go-to-definition performance using vernier

Example Output

		❯ solargraph profile
		Parsing and mapping source files...
		Mapping libraries
		[WARN] Failed to load plugin 'solargraph-rspec'
		Notification: $/progress - {:token=>"189f5c5e-fdae-4bdc-99e0-923af5768255", :value=>{:kind=>"begin", :cancellable=>false, :title=>"Mapping workspace", :message=>"0/355 files", :percentage=>0}}
		... trimmed for brevity
		Notification: $/progress - {:token=>"189f5c5e-fdae-4bdc-99e0-923af5768255", :value=>{:kind=>"report", :cancellable=>false, :message=>"355/355 files", :percentage=>100}}
		Notification: $/progress - {:token=>"189f5c5e-fdae-4bdc-99e0-923af5768255", :value=>{:kind=>"end", :cancellable=>false, :message=>"done"}}
		Building the catalog...
		[WARN] Failed to load plugin 'solargraph-rspec'
		Profiling go-to-definition for /Users/lekemula/Projects/solargraph/lib/solargraph/api_map/cache.rb
		Position: line 4, column 10
		Processing go-to-definition request...
		Result: [{:uri=>"file:///Users/lekemula/Projects/solargraph/lib/solargraph/api_map/cache.rb", :range=>{:start=>{:line=>4, :character=>4}, :end=>{:line=>107, :character=>7}}}]
		
		=== Timing Results ===
		Parsing & mapping: 1103.74ms
		Catalog building: 7.7ms
		Go-to-definition: 5742.56ms
		Total time: 6854.0ms
		
		Profiles saved to:
		  - /Users/lekemula/Projects/solargraph/tmp/profiles/parse_benchmark.json.gz
		  - /Users/lekemula/Projects/solargraph/tmp/profiles/catalog_benchmark.json.gz
		  - /Users/lekemula/Projects/solargraph/tmp/profiles/definition_benchmark.json.gz
		
		Upload the JSON files to https://vernier.prof/ to view the profiles.
		Or use https://rubygems.org/gems/profile-viewer to view them locally.

By default, the solargraph profile will run against the current directory, and will find the first Ruby file with the default line/number.

When testing new local changes, one could use the bundle exec solargraph --directory=/path/to/a/real-project ..other params

Here's a quick demo: here

I hope you find it useful.

Future Work

Maybe as a next step, we could select some large open-source Rails projects that we can use to profile and track the merged changes in main through a CI step. Any suggestions for such projects? Personally, I've used https://github.com/gitlabhq/gitlabhq in numerous cases, but https://github.com/mastodon/mastodon also come to mind as good candidates. WDYT?

lekemula · 2025-09-06T19:34:27Z

I'm not sure how my changes could lead to such breaking of the typechecking step here.

Run SOLARGRAPH_ASSERTS=on bundle exec solargraph typecheck --level typed
bundler: failed to load command: solargraph (/opt/hostedtoolcache/Ruby/3.4.5/x64/lib/ruby/gems/3.4.0/bin/solargraph)
/home/runner/work/solargraph/solargraph/lib/solargraph.rb:75:in 'Solargraph.assert_or_log': source not provided -   Solargraph::Pin::Namespace (RuntimeError)

@apiology do you recognise anything obvious there by any chance?

lekemula · 2025-09-06T19:36:51Z

lib/solargraph/shell.rb

+      directory = File.realpath(options[:directory])
+      FileUtils.mkdir_p(options[:output_dir])
+
+      host = Solargraph::LanguageServer::Host.new


I have based the host setup, based on this spec:

solargraph/spec/language_server/message/text_document/definition_spec.rb

Line 2 in 872431c

it 'finds definitions of methods' do

Is there any critical step missing that should be included in the profile?

lekemula · 2025-09-06T19:37:53Z

lib/solargraph/shell.rb

+      def host.send_notification method, params
+        puts "Notification: #{method} - #{params}"
+      end


I thought I'd never find a use-case for this Ruby feature, but I finally found one 😍

Hey! This is the line that triggered that assert - looks like Solargraph has never run through that code in its tests or typechecking in CI, either!

https://github.com/apiology/solargraph/blob/master/lib/solargraph/parser/parser_gem/node_processors/defs_node.rb creates some pins, but doesn't pass the 'source' kwarg in. If you throw "source: :parser" into that "Solargraph::Pin::Namespace.new" call, it should get past that error.

Thanks again, @apiology, for coming to the rescue! 🙇‍♂️

For some reason, I thought the typechecker does not cover the shell area.

apiology · 2025-09-08T13:01:22Z

Maybe as a next step, we could select some large open-source Rails projects that we can use to profile and track the merged changes in main through a CI step. Any suggestions for such projects? Personally, I've used https://github.com/gitlabhq/gitlabhq in numerous cases, but https://github.com/mastodon/mastodon also come to mind as good candidates. WDYT?

I've been thinking about doing the same thing as a functionality test. @AaronC81 has some interesting work here in Sord that we could probably import with credit: https://github.com/AaronC81/sord/blob/master/Rakefile#L121-L138

apiology · 2025-09-08T13:05:14Z

solargraph.gemspec

  s.add_development_dependency 'undercover', '~> 0.7'
  s.add_development_dependency 'overcommit', '~> 0.68.0'
  s.add_development_dependency 'webmock', '~> 3.6'
+  s.add_development_dependency 'vernier'


Suggested change

s.add_development_dependency 'vernier'

s.add_development_dependency 'vernier', '~> 1.8'

This will keep a 2.0 release from breaking our build so we can update on our own timeline, and make sure weird things don't happen if a user has an old version installed.

Unfortunately both '~> 1.8' and '~> 1' failed, so it looks like we need < 2.

apiology · 2025-09-09T15:23:24Z

Thanks again for this work! I was able to track down a big perf hit in #1064 and #1006 using it.

castwide · 2025-09-10T20:40:27Z

My main concern with integrating Vernier into the codebase is that it doesn't compile on Windows. A similar issue with EventMachine used to cause a lot of headaches.

lekemula · 2025-09-11T21:06:45Z

My main concern with integrating Vernier into the codebase is that it doesn't compile on Windows. A similar issue with EventMachine used to cause a lot of headaches.

@castwide vernier gem is added as a development dependency, so normal users should not be affected by that. Or are you also actually concerned about potential contributors on Windows? Based on the Rails 2024 Survey, the number of devs using Windows/WSL2 amounts to~4% - so that can be negligible? IIUC, WSL2 is basically "Linux on Windows," so potentially the number of affected potential contributors might be less than 4%.

I was not aware of this bthw - the main reason why I kept it as an optional dependency is to not include unnecessary dependencies to end-users. If someone needs to profile it with the latest version, they can install it manually (see lib/solargraph/shell.rb:256).

We can always remove it as a development dependency, too, and keep it as an optional dependency in both contexts.

lekemula · 2025-09-11T21:14:37Z

Thanks again for this work! I was able to track down a big perf hit in #1064 and #1006 using it.

Happy to hear that, we're seeing benefits out of this already 🙌. I look forward to having this as a CI step, where we can formally track performance changes. Thanks for the suggestion here, that looks like a good start!

lekemula · 2025-09-11T21:58:30Z

@apiology, sorry to harass you again, but do you know why there are discrepancies between overcommit ❌ and typechecking ✅ step?

Are they using different levels of typechecking? Or a different Ruby/rbs version, maybe?

UPDATE:

I ran it locally (=* ruby-3.3.1 [ arm64 ]), and it gave me a different list of issues, which confuses me a little:

❯ bundle exec overcommit --run --diff upstream/master
Running pre-commit hooks
Check for "token" strings.....................................[FixMe] OK
Analyze with RuboCop........................................[RuboCop] OK
Typecheck with Solargraph................................[Solargraph] FAILED
Errors on modified lines:
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:268 - Missing @return tag for host.send_notification
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:268 - Missing @param tag for method on host.send_notification
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:268 - Missing @param tag for params on host.send_notification
Errors on lines you didn't modify:
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:69 - Unresolved call to each on Class<Gem::Specification>
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:144 - Unresolved call to to_a on Class<Gem::Specification>
/Users/lekemula/Projects/solargraph/lib/solargraph/shell.rb:145 - Unresolved call to count on Class<Gem::Specification>

I fixed them here, so let's see..

UPDATE:

One difference that I see between these CI steps, is caching:

solargraph/.github/workflows/linting.yml

Lines 42 to 52 in 453a6e9

    
                 - name: Restore cache of gem annotations 
        
                   id: dot-cache-restore 
        
                   uses: actions/cache/restore@v4 
        
                   with: 
        
                     key: | 
        
                       2025-06-26-09-${{ runner.os }}-dot-cache-${{ hashFiles('Gemfile.lock') }} 
        
                     restore-keys: | 
        
                       2025-06-26-09-${{ runner.os }}-dot-cache 
        
                       2025-06-26-09-${{ runner.os }}-dot-cache- 
        
                     path: | 
        
                       /home/runner/.cache/solargraph

vs

solargraph/.github/workflows/typecheck.yml

Line 31 in 453a6e9

bundler-cache: false

Could that have a role?

apiology · 2025-09-12T00:33:59Z

My main concern with integrating Vernier into the codebase is that it doesn't compile on Windows. A similar issue with EventMachine used to cause a lot of headaches.

Will it work if we leave it out altogether, add a require inside the function within a try / rescue block, and tell people to install the gem themselves?

We could also mark the command as hidden (like my PR for the pin command)

apiology · 2025-09-12T00:35:05Z

Are they using different levels of typechecking? Or a different Ruby/rbs version, maybe?

Yep! The overcommit uses the strong level for the bits you change, the rest is done at 'typed' level.

apiology · 2025-09-12T00:37:31Z

The errors you're seeing at strong level in overcommit will be fixed in #1059 - you can either merge that branch in or just ignore the errors entirely.

lekemula · 2025-09-12T08:47:56Z

My main concern with integrating Vernier into the codebase is that it doesn't compile on Windows. A similar issue with EventMachine used to cause a lot of headaches.

Will it work if we leave it out altogether, add a require inside the function within a try / rescue block, and tell people to install the gem themselves?

We already do that except for development.

We could also mark the command as hidden (like my PR for the pin command)

So I question how worth it that is here. I'd prefer more people know about this command so they can report issues they notice on their side.

Co-authored-by: Vince Broz <[email protected]>

lekemula · 2025-09-27T10:08:18Z

The errors you're seeing at strong level in overcommit will be fixed in #1059 - you can either merge that branch in or just ignore the errors entirely.

Thanks for the heads-up @apiology. I would prefer to keep the PR clean, and suggest that we ignore the overcommit for shell.rb for the time being.

lekemula force-pushed the profile-command branch from a2695f0 to 872431c Compare September 6, 2025 19:24

lekemula commented Sep 6, 2025

View reviewed changes

apiology reviewed Sep 8, 2025

View reviewed changes

lekemula force-pushed the profile-command branch 3 times, most recently from 6be19ba to a13322a Compare September 11, 2025 21:39

lekemula force-pushed the profile-command branch 2 times, most recently from 5af206e to 453a6e9 Compare September 11, 2025 22:48

lekemula and others added 4 commits September 27, 2025 12:02

Add solargraph profile command

b4c8b99

Add memory usage counter

d67d580

Fix CI (typechecking) and pin the gem version

c0455fa

Co-authored-by: Vince Broz <[email protected]>

Make overcommit happy?

431dd32

lekemula force-pushed the profile-command branch from 453a6e9 to 431dd32 Compare September 27, 2025 10:03

apiology mentioned this pull request Oct 6, 2025

2025-09-30 dev branch apiology/solargraph#14

Draft

	s.add_development_dependency 'vernier'
	s.add_development_dependency 'vernier', '~> 1.8'

Uh oh!

Add solargraph profile command using vernier #1071

Are you sure you want to change the base?

Add solargraph profile command using vernier #1071

Uh oh!

Conversation

lekemula commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Future Work

Uh oh!

lekemula commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lekemula Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

lekemula Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

apiology Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

lekemula Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

apiology commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apiology Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

lekemula Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

apiology commented Sep 9, 2025

Uh oh!

castwide commented Sep 10, 2025

Uh oh!

lekemula commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lekemula commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lekemula commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apiology commented Sep 12, 2025

Uh oh!

apiology commented Sep 12, 2025

Uh oh!

apiology commented Sep 12, 2025

Uh oh!

lekemula commented Sep 12, 2025

Uh oh!

lekemula commented Sep 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `solargraph profile` command using vernier #1071

Add `solargraph profile` command using vernier #1071

lekemula commented Sep 6, 2025 •

edited

Loading

lekemula commented Sep 6, 2025 •

edited

Loading

apiology commented Sep 8, 2025 •

edited

Loading

lekemula commented Sep 11, 2025 •

edited

Loading

lekemula commented Sep 11, 2025 •

edited

Loading

lekemula commented Sep 11, 2025 •

edited

Loading