Skip to content

pyroscope.ebpf: otel profiler implementation #2920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 68 commits into from
Jul 14, 2025
Merged

pyroscope.ebpf: otel profiler implementation #2920

merged 68 commits into from
Jul 14, 2025

Conversation

korniltsev
Copy link
Contributor

@korniltsev korniltsev commented Mar 6, 2025 •
edited
Loading

PR Description

This PR changes the implementation of pyroscope.ebpf compoenent from grafana/pyroscope/ebpf to opentelemetry-ebpf-profiler - a fork of opentelemetry-ebpf-profiler.
The notable changes of the fork: native frame symbolization, dynamic profiling policy, target discovery labels.

The fork uses a rust library for symbol extraction. See https://github.com/open-telemetry/opentelemetry-ebpf-profiler/tree/main/rust-crates.

In this PR I put all pyroscope.ebpf code behind a pyroscope_ebpf go tag.

Some component arguments became no-op as there are no corresponding knobs in the new implementations. These args a deprecated

// deprecated
	PidCacheSize int `alloy:"pid_cache_size,attr,optional"`
	// deprecated
	BuildIDCacheSize int `alloy:"build_id_cache_size,attr,optional"`
	// deprecated
	SameFileCacheSize int `alloy:"same_file_cache_size,attr,optional"`
	// deprecated
	CacheRounds int `alloy:"cache_rounds,attr,optional"`
	// deprecated
	GoTableFallback bool `alloy:"go_table_fallback,attr,optional"`
	// deprecated
	SymbolsMapSize int `alloy:"symbols_map_size,attr,optional"`

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

  • CHANGELOG.md updated
  • Documentation added
  • Tests updated
  • Config converters updated

@korniltsev korniltsev force-pushed the korniltsev/otel-profiler branch from d9663da to 3779c4e Compare March 6, 2025 12:21
korniltsev
korniltsev commented Mar 6, 2025
View reviewed changes
korniltsev
korniltsev commented Mar 6, 2025
View reviewed changes
korniltsev
korniltsev commented Mar 6, 2025
View reviewed changes
@korniltsev korniltsev force-pushed the korniltsev/otel-profiler branch from 3779c4e to 7c540d2 Compare March 6, 2025 12:32
@korniltsev korniltsev force-pushed the korniltsev/otel-profiler branch from 7c540d2 to ca84882 Compare March 6, 2025 12:53
@korniltsev korniltsev mentioned this pull request Mar 6, 2025
Merged
4 tasks
korniltsev
korniltsev commented Mar 6, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 10, 2025 •
edited
Loading

💻 Deploy preview deleted.

@korniltsev korniltsev marked this pull request as ready for review March 11, 2025 08:21
@korniltsev korniltsev mentioned this pull request Jun 18, 2025
Merged
4 tasks
@korniltsev korniltsev marked this pull request as ready for review June 24, 2025 06:41
@korniltsev korniltsev mentioned this pull request Jun 25, 2025
Closed
marcsanmi
marcsanmi reviewed Jun 26, 2025
View reviewed changes
Copy link
Contributor

@marcsanmi marcsanmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👏 , just left some nits!

docs/sources/reference/components/pyroscope/pyroscope.ebpf.md Outdated
| `python_enabled` | `bool` | A flag to enable or disable python profiling. | `true` | no |
| `ruby_enabled` | `bool` | A flag to enable or disable Ruby profiling. | `true` | no |
| `same_file_cache_size` | `int` | Deprecated (no-op), previously controlled the size of the elf file -> symbols table LRU cache. | `8` | no |
| `sample_rate` | `int` | How many times per second to collect profile samples. | `97` | no |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the new default is 19?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, updating

docs/sources/reference/components/pyroscope/pyroscope.ebpf.md
| `go_table_fallback` | `bool` | Deprecated (no-op), previously enabled symbol lookup in `.sym` / `.dynsym` sections when `.gopclntab` lookup failed. | `false` | no |
| `hotspot_enabled` | `bool` | A flag to enable ordisable hotspot profiling. | `true` | no |
| `perl_enabled` | `bool` | A flag to enable or disable Perl profiling. | `true` | no |
| `php_enabled` | `bool` | A flag to enable or disable PHP profiling. | `true` | no |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should go_enabled be also documented?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current revision of the go consumes shit ton of memory and cpu and it works much better without it (i.e. using our symbolizer). We may want to add go here after we update our fork to include open-telemetry/opentelemetry-ebpf-profiler#456 which is not straightforward because it removes the rust lib and we rely on it. tldr - yes, but later.

docs/sources/reference/components/pyroscope/pyroscope.ebpf.md Outdated
| `perl_enabled` | `bool` | A flag to enable or disable Perl profiling. | `true` | no |
| `php_enabled` | `bool` | A flag to enable or disable PHP profiling. | `true` | no |
| `pid_cache_size` | `int` | Deprecated (no-op), previously controlled the size of the PID -> proc symbols table LRU cache. | `32` | no |
| `pid_map_size` | `int` | The size of eBPF PID map. | `2048` | no |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer used? Do you keep it for backwards compatibility? I think we should either update it to deprecated or just remove it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. deprecated. We may want to expose tracer.MapScaleFactor in the future.

internal/component/pyroscope/ebpf/ebpf_linux.go Outdated
}
defer ctlr.Shutdown()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to cleanup FileObserver to avoid resource leaking?

  defer func() {
      ctlr.Shutdown()
      if c.cfg.FileObserver != nil {
          c.cfg.FileObserver.Cleanup()
      }
  }()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, nice catch

@korniltsev korniltsev requested a review from marcsanmi June 26, 2025 12:00
marcsanmi
marcsanmi approved these changes Jun 26, 2025
View reviewed changes
Copy link
Contributor

@marcsanmi marcsanmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

@korniltsev korniltsev mentioned this pull request Jun 27, 2025
Closed
@korniltsev korniltsev merged commit bd763f8 into main Jul 14, 2025
45 checks passed
@korniltsev korniltsev deleted the korniltsev/otel-profiler branch July 14, 2025 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@luweglarz luweglarz luweglarz left review comments

@clayton-cornell clayton-cornell clayton-cornell left review comments

@marcsanmi marcsanmi marcsanmi approved these changes

@simonswine simonswine Awaiting requested review from simonswine

Assignees

@korniltsev korniltsev

Labels
area/pyroscope Issues/PRs primarly affecting `pyroscope.` components type/docs Docs Squad label across all Grafana Labs repos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@korniltsev @CLAassistant @luweglarz @simonswine @marcsanmi @clayton-cornell