Fix BPF token permission issues with 6.10 and later kernels #2042
Labels
area/datacollector
Issues related to Stirling (datacollector)
Comments
1 task
ddelnano
added a commit
that referenced
this issue
Oct 11, 2024
…er kernels (#2041) Summary: Upgrade bcc and libbpf to fix BPF program compilation on 6.10 and later kernels Bcc provides some "[virtual](https://github.com/iovisor/bcc/blob/cb1ba20f4800f556dc940682ba7016c50bd0a3ac/src/cc/exported_files.cc#L28-L48)" includes to BPF programs. The `compat/linux/virtual_bpf.h` file in particular needs to be kept in sync with libbpf and matches the [header guard](https://github.com/iovisor/bcc/blob/cb1ba20f4800f556dc940682ba7016c50bd0a3ac/src/cc/compat/linux/virtual_bpf.h#L9) of the `include/uapi/linux/bpf.h` file. This means that while our linux headers were updated, our older bcc install was inserting an older copy of the `uapi/linux/bpf.h` file -- one that didn't contain the `bpf_wq` declaration. ``` include/linux/bpf.h:348:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq' return sizeof(struct bpf_wq); ^ ~~~~~~~~~~~~~~~ include/linux/bpf.h:348:24: note: forward declaration of 'struct bpf_wq' return sizeof(struct bpf_wq); ^ include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq' return __alignof__(struct bpf_wq); ^ ~~~~~~~~~~~~~~~ include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_wq' return __alignof__(struct bpf_wq); ``` Note: while this fixes the 6.10 compilation issue, our 6.10 qemu build fails without disabling [this logic](https://github.com/pixie-io/pixie/blob/3c41d554215528e688328aef94192e696db617dc/src/stirling/source_connectors/socket_tracer/socket_trace_connector.cc#L464-L472). 6.10 kernels added BPF token support. This changes the BPF permission model slightly and causes the BPF instruction limit to be dependent on the permissions of the BPF syscall caller ([linux source](https://elixir.bootlin.com/linux/v6.11.1/source/kernel/bpf/syscall.c#L2757)). This new BPF token logic coupled with our qemu setup, causes our 6.10 build to fallback to the 4096 instruction limit. I'll be addressing this in #2040 and #2042. Those issues shouldn't block this change since that loop limit code can be bypasses at runtime with our current cli flags. Relevant Issues: Closes #2035 Type of change: /kind bugfix Test Plan: Built 6.10 and 6.11 kernels and the associated linux headers from #2036 and verified that a local qemu build passes - [x] Verify `#ci:bpf-build-all-kernels` build passes Changelog Message: Upgraded bcc and libbpf to support kernels 6.10 and later --------- Signed-off-by: Dom Del Nano <[email protected]>
ddelnano
changed the title
|
This is now confirmed to happen in environments outside of our qemu builds and means that we have issues on kernels 6.10 and later. I've uploaded the logs for the reported instance of this. |
2 tasks
ddelnano
added a commit
that referenced
this issue
Nov 19, 2024
…es (#2047) Summary: Allow newer kernels to opt into explicit loop and chunk limit overrides Overwriting non default values of the bpf chunk and loop limit is frustrating since it undoes configuration a user explicitly requested. This also will allow for working around #2042, which causes kernels 6.10 and later to fail to start the socket tracer. In addition to this, I tweaked the kernel upgrade logic slightly. The previous logic would have upgraded a 5.0.x kernel to use the new loop limit when the 1M instruction limit isn't available until 5.1. Relevant Issues: Helps to work around #2042 Type of change: /kind bugfix Test Plan: Verified the following on a 6.x kernel running `stirling_wrapper` - [x] Not supplying any arguments results in using an increased loop and chunk limit ``` $ sudo ./bazel-bin/src/stirling/binaries/stirling_wrapper I20241118 05:52:37.358364 3104175 socket_trace_connector.cc:474] Kernel version greater than V5.1 detected (6.8.12), raised loop limit to 882 and chunk limit to 84 ``` - [x] Supplying the loop limit or chunk limit flag disables the automatic increase ``` $ sudo ./bazel-bin/src/stirling/binaries/stirling_wrapper --stirling_bpf_loop_limit=41 I20241118 05:53:21.082810 3104197 source_connector.cc:35] Initializing source connector: socket_tracer I20241118 05:53:21.082886 3104197 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.8.0-1015-gcp I20241118 05:53:21.082916 3104197 linux_headers.cc:395] Detected kernel release (uname -r): 6.8.0-1015-gcp I20241118 05:53:21.082964 3104197 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.8.0-1015-gcp/build. I20241118 05:53:21.083058 3104197 bcc_wrapper.cc:166] Initializing BPF program ... ``` Changelog Message: Ensures that the `--stirling_bpf_loop_limit` and `--stirling_bpf_chunk_limit` values are respected if explicitly provided on the command line. For 5.1 and later kernels, cli provided values would have been ignored --------- Signed-off-by: Dom Del Nano <[email protected]>
|
With the introduction of #2047 (available in Vizier v0.14.13), users using impacted kernels can get Pixie running with the following PEM cli flags:
Note: disabling mux and mongodb isn't specifically required. Each enabled protocol increases the size of Pixie's BPF program and so disabling any two protocols suffices as well. |
ddelnano
added a commit
to ddelnano/pixie
that referenced
this issue
Dec 9, 2024
…es (pixie-io#2047) Summary: Allow newer kernels to opt into explicit loop and chunk limit overrides Overwriting non default values of the bpf chunk and loop limit is frustrating since it undoes configuration a user explicitly requested. This also will allow for working around pixie-io#2042, which causes kernels 6.10 and later to fail to start the socket tracer. In addition to this, I tweaked the kernel upgrade logic slightly. The previous logic would have upgraded a 5.0.x kernel to use the new loop limit when the 1M instruction limit isn't available until 5.1. Relevant Issues: Helps to work around pixie-io#2042 Type of change: /kind bug Test Plan: Verified the following on a 6.x kernel running `stirling_wrapper` - [x] Not supplying any arguments results in using an increased loop and chunk limit ``` $ sudo ./bazel-bin/src/stirling/binaries/stirling_wrapper I20241118 05:52:37.358364 3104175 socket_trace_connector.cc:474] Kernel version greater than V5.1 detected (6.8.12), raised loop limit to 882 and chunk limit to 84 ``` - [x] Supplying the loop limit or chunk limit flag disables the automatic increase ``` $ sudo ./bazel-bin/src/stirling/binaries/stirling_wrapper --stirling_bpf_loop_limit=41 I20241118 05:53:21.082810 3104197 source_connector.cc:35] Initializing source connector: socket_tracer I20241118 05:53:21.082886 3104197 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.8.0-1015-gcp I20241118 05:53:21.082916 3104197 linux_headers.cc:395] Detected kernel release (uname -r): 6.8.0-1015-gcp I20241118 05:53:21.082964 3104197 linux_headers.cc:206] Using Linux headers from: /lib/modules/6.8.0-1015-gcp/build. I20241118 05:53:21.083058 3104197 bcc_wrapper.cc:166] Initializing BPF program ... ``` Changelog Message: Ensures that the `--stirling_bpf_loop_limit` and `--stirling_bpf_chunk_limit` values are respected if explicitly provided on the command line. For 5.1 and later kernels, cli provided values would have been ignored --------- Signed-off-by: Dom Del Nano <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is closely related to #2040. Our qemu builds are unable to pass the newer BPF token permission checks, causing it to use the reduced (4096) BPF instruction limit. We should update our qemu VM image building process to ensure that it's able to use the 1M instruction limit.
Logs
The text was updated successfully, but these errors were encountered: