bpf: tracing multi-link support #9249
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![kernel-patches-daemon-bpf[bot]](https://avatars.githubusercontent.com/u/70728002?s=60&v=4){kind=small}
|
Upstream branch: 38d95be |
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
cc01c15 to
901b4c5
Compare
Implement a hash table to store the BPF progs and the function metadata. The key of this hash table is the kernel function address, and following data is stored in the hash value: - The BPF progs, whose type is FENTRY, FEXIT or MODIFY_RETURN. The struct kfunc_md_tramp_prog is introduced to store the BPF prog and the cookie, and makes the BPF progs of the same type a list with the "next" field. - The kernel function address - The kernel function arguments count - If origin call needed The hlist is used, and we will grow the budgets when the entries count greater than 90% of the budget count by making it double. Meanwhile, we will shrink the budget when the entries count less than 30% of the budget length. We don't use rhashtable here, as the compiler is not clever enough and it refused to inline the hash lookup for me, which bring in addition overhead in the following BPF global trampoline. The release of the metadata is controlled by the percpu ref and RCU together, and have similar logic to the release of bpf trampoline image in bpf_tramp_image_put(). The whole function will be used in the next patch. Link: https://lore.kernel.org/bpf/CADxym3anLzM6cAkn_z71GDd_VeKiqqk1ts=xuiP7pr4PO6USPA@mail.gmail.com/ Link: https://lore.kernel.org/bpf/CAADnVQ+G+mQPJ+O1Oc9+UW=J17CGNC5B=usCmUDxBA-ze+gZGw@mail.gmail.com/ Signed-off-by: Menglong Dong <[email protected]>
Implement the bpf global trampoline "bpf_global_caller" for x86_64. Thanks to Alexei's advice, we implement most of the global trampoline with C instead of asm. We implement the entry of the trampoline with a "__naked" function, who will save the regs to an array on the stack and call bpf_global_caller_run(). The entry will pass the address of the array and the address of the rip to bpf_global_caller_run(). In bpf_global_caller_run(), we will find the metadata by the function ip. For origin call case, we call kfunc_md_enter() to protect the metadata, which is similar to __bpf_tramp_enter(). Then we will call all the BPF progs, just like what BPF trampoline do. Without origin call, the bpf_global_caller_run() will return 0, and the entry will restore the regs and return; In origin call case, it will return 1, and the entry will make the RSP skip the rip before return. In the FENTRY case, the performance of global trampoline is ~10% slower than BPF trampoline. The global trampoline is optimized by inline some function call, such as __bpf_prog_enter_recur and __bpf_prog_exit_recur. However, more condition, branch and memory read is used in the bpf_global_caller. In the FEXIT and MODIFY_RETURN cases, the performance of the global trampoline is the same(or even better) than BPF trampoline. It make sense, as we also make the function call to __bpf_tramp_enter and __bpf_tramp_exit inlined in the bpf_global_caller. In fact, we can do more optimization to the bpf_global_caller. For example, we can define more bpf_global_caller_xx_run() function and make the "if (prog->sleepable)" and "if (do_origin_call)" fixed. It can be done in a next series. After such optimization, I believe the performance of FENTRY_MULTI can be closer or the same to FENTRY. And for the FEXIT/MODIFY_RETURN cases, the performance can be better. Signed-off-by: Menglong Dong <[email protected]>
Factor out ftrace_direct_update() from register_ftrace_direct(), which is used to add new entries to the direct_functions. This function will be used in the later patch. Signed-off-by: Menglong Dong <[email protected]>
For now, we can change the address of a direct ftrace_ops with modify_ftrace_direct(). However, we can't change the functions to filter for a direct ftrace_ops. Therefore, we introduce the function reset_ftrace_direct_ips() to do such things, and this function will reset the functions to filter for a direct ftrace_ops. This function do such thing in following steps: 1. filter out the new functions from ips that don't exist in the ops->func_hash->filter_hash and add them to the new hash. 2. add all the functions in the new ftrace_hash to direct_functions by ftrace_direct_update(). 3. reset the functions to filter of the ftrace_ops to the ips with ftrace_set_filter_ips(). 4. remove the functions that in the old ftrace_hash, but not in the new ftrace_hash from direct_functions. Signed-off-by: Menglong Dong <[email protected]>
Introduce the struct bpf_gtramp_link, which is used to attach
a bpf prog to multi functions. Meanwhile, introduce corresponding
function bpf_gtrampoline_{link,unlink}_prog. The lock global_tr_lock is
held during global trampoline link and unlink.
We create different global trampoline for the kernel functions that have
different argument count. If corresponding global_tr->image is NULL, it
means such function argument count is not supported.
Signed-off-by: Menglong Dong <[email protected]>
In this commit, we add the 'accessed_args' field to struct bpf_prog_aux, which is used to record the accessed index of the function args in btf_ctx_access(). Meanwhile, we add the function btf_check_func_part_match() to compare the accessed function args of two function prototype. This function will be used in the following commit. Signed-off-by: Menglong Dong <[email protected]>
Refactor the struct modules_array to more general struct ptr_array, which is used to store the pointers. Meanwhile, introduce the bpf_try_add_ptr(), which checks the existing of the ptr before adding it to the array. Seems it should be moved to another files in "lib", and I'm not sure where to add it now, and let's move it to kernel/bpf/syscall.c for now. Signed-off-by: Menglong Dong <[email protected]>
Add target btf to the function args of bpf_check_attach_target(), then the caller can specify the btf to check. Signed-off-by: Menglong Dong <[email protected]>
Move the checking of btf_id_deny and noreturn_deny from check_attach_btf_id() to bpf_check_attach_target(). Therefore, we can do such checking during attaching for tracing multi-link in the later patches. Signed-off-by: Menglong Dong <[email protected]>
Factor the function arch_bpf_get_regs_nr() to get the regs count that used by the function args. Signed-off-by: Menglong Dong <[email protected]>
In this commit, we add the support to allow attaching a tracing BPF program to multi hooks, which is similar to BPF_TRACE_KPROBE_MULTI. The use case is obvious. For now, we have to create a BPF program for each kernel function, for which we want to trace, even through all the program have the same (or similar logic). This can consume extra memory, and make the program loading slow if we have plenty of kernel function to trace. The KPROBE_MULTI maybe a alternative, but it can't do what TRACING do. For example, the kretprobe can't obtain the function args, but the FEXIT can. For now, we support to create multi-link for fentry/fexit/modify_return with the following new attach types that we introduce: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We introduce the struct bpf_tracing_multi_link for this purpose, which can hold all the kernel modules, target bpf program (for attaching to bpf program) or target btf (for attaching to kernel function) that we referenced. During loading, the first target is used for verification by the verifer. And during attaching, we check the consistency of all the targets with the first target. Signed-off-by: Menglong Dong <[email protected]>
By default, the kernel btf that we load during loading program will be freed after the programs are loaded in bpf_object_load(). However, we still need to use these btf for tracing of multi-link during attaching. Therefore, we don't free the btfs until the bpf object is closed if any bpf programs of the type multi-link tracing exist. Meanwhile, introduce the new api bpf_object__free_btf() to manually free the btfs after attaching. Signed-off-by: Menglong Dong <[email protected]>
Add supporting for the attach types of: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI Signed-off-by: Menglong Dong <[email protected]>
For now, the libbpf find the btf type id by loop all the btf types and compare its name, which is inefficient if we have many functions to lookup. We add the "use_hash" to the function args of find_kernel_btf_id() to indicate if we should lookup the btf type id by hash. The hash table will be initialized if it has not yet. Signed-off-by: Menglong Dong <[email protected]>
We add skip_invalid and attach_tracing for tracing_multi for the selftests. When we try to attach all the functions in available_filter_functions with tracing_multi, we can't tell if the target symbol can be attached successfully, and the attaching will fail. When skip_invalid is set to true, we will check if it can be attached in libbpf, and skip the invalid entries. We will skip the symbols in the following cases: 1. the btf type not exist 2. the btf type is not a function proto 3. the function args count more that 6 4. the return type is struct or union 5. any function args is struct or union The 5th rule can be a manslaughter, but it's ok for the testings. "attach_tracing" is used to convert a TRACING prog to TRACING_MULTI. For example, we can set the attach type to FENTRY_MULTI before we load the skel. And we can attach the prog with bpf_program__attach_trace_multi_opts() with "attach_tracing=1". The libbpf will attach the target btf type of the prog automatically. This is also used to reuse the selftests of tracing. (Oh my goodness! What am I doing?) Signed-off-by: Menglong Dong <[email protected]>
We need to get all the kernel function that can be traced sometimes, so we move the get_syms() and get_addrs() in kprobe_multi_test.c to trace_helpers.c and rename it to bpf_get_ksyms() and bpf_get_addrs(). Signed-off-by: Menglong Dong <[email protected]>
|
Upstream branch: 1f24c0d |
In this commit, we add some testcases for the following attach types: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI BPF_MODIFY_RETURN_MULTI We reuse the testings in fentry_test.c, fexit_test.c and modify_return.c by attach the tracing bpf prog as tracing_multi. We add some functions to skip for tracing progs to bpf_get_ksyms(). The functions that in the "btf_id_deny" should be skipped. What's more, the kernel can't find the right function address according to the btf type id when duplicated function name exist. So we skip such functions that we meet. The list is not whole, so we still can fail during attaching the FENTRY_MULTI to all the kernel functions. This is something that we need to fix in the feature. Signed-off-by: Menglong Dong <[email protected]>
Add bench testcase for fentry_multi, fexit_multi and fmodret_multi in bench_trigger.c. Signed-off-by: Menglong Dong <[email protected]>
kernel-patches-daemon-bpf
bot
force-pushed
the
series/966845=>bpf-next
branch
from
4c93f19 to
d028a2a
Compare
kernel-patches-daemon-bpf
bot
force-pushed
the
bpf-next_base
branch
from
901b4c5 to
fa8c3a1
Compare
|
At least one diff in series https://patchwork.kernel.org/project/netdevbpf/list/?series=978612 expired. Closing PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: bpf: tracing multi-link support
version: 2
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=978612