Skip to content

Conversation

@simonswine
Copy link
Contributor

@simonswine simonswine commented Sep 23, 2025

With #651 it is possible to attach uprobes, but their results are currently merged together. This PR introduces the use of cookies (available from kernel 5.15+) to keep track of the links that lead to the events being generated.

It now uses the symbol part of the uprobe link and propages it into the sample type. That is something that we currently don't do well enough.

So let's look at this example:

--uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:malloc --uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:malloc

There would be resulting two sample types

  • uprobe_malloc_events:count
  • uprobe_open_events:count

With open-telemetry#651 it is possible to attach uprobes, but they currently are
merged together. This PR introduces the use of cookies (available from kernel 5.15+) to keep
track of the links that lead to the events being generated.

It now uses the symbol part of the uprobe link and propages it into the sample type. That is something that we currently don't do well enough.

So let's look at this example:

```
--uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:malloc --uprobe-link /nix/store/r7pnxs3cfl3qxwacj38iakpm5v1ch6lz-glibc-2.40-66/lib/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:open --uprobe-link /proc/4501/root/usr/lib/aarch64-linux-gnu/libc.so.6:malloc
```

There would be resulting two sample types

- `uprobe_malloc_events:count`
- `uprobe_open_events:count`
@simonswine simonswine force-pushed the 20250923_distinguish-between-uprobes branch from aeb3d34 to 49c080c Compare September 23, 2025 15:03
SEC("uprobe/generic")
int uprobe__generic(void *ctx)
{
u64 cookie = bpf_get_attach_cookie(ctx);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The minimal supported version is currently 5.4 - see https://github.com/open-telemetry/opentelemetry-ebpf-profiler?tab=readme-ov-file#supported-linux-kernel-version.
This bpf helper function is not available with this kernel version.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI will fail, once #826 is merged and this change is rebased.

Copy link
Contributor Author

@simonswine simonswine Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not super experienced with the bpf tooling, but is should be possible to either target uprobes only for kernels 5.15+ or have a program each with/without cookie helper?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so far, the complexity is avoided to differentiate such cases. a work around could be to define a global variable, that is loaded at load time of the eBPF program and sets the value to differentiate different attach points.

@simonswine simonswine changed the title feat: Propate the uprobe link into sample type feat: Propagate the uprobe link into sample type Sep 23, 2025
u64 ts = bpf_ktime_get_ns();

return collect_trace(ctx, TRACE_UPROBE, pid, tid, ts, 0);
return collect_trace(ctx, TRACE_UPROBE, pid, tid, ts, cookie);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this goes ahead I would say we need to rename off_cpu_time to off_cpu_time_or_cookie or add an extra parameter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To follow along this thought: What is the expected field in the OTel Profiling signal to communicate this?

Comment on lines +160 to +164
if origin.ProbeLinkName != "" {
st.SetTypeStrindex(stringSet.Add(fmt.Sprintf("uprobe_%s_events", origin.ProbeLinkName)))
} else {
st.SetTypeStrindex(stringSet.Add("uprobe_events"))
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TypeStr across the Origins are not consistent right now, I think we lack to follow the spec here:

https://github.com/open-telemetry/opentelemetry-proto/blob/0d02f212598f3ec1dda35274e87f59351f619058/opentelemetry/proto/profiles/v1development/profiles.proto#L267-L271

This feels closer to the spec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants