Add multi kprobe attach

[vincentmli]

Refactor the single kprobe attach to a function to prepare for add multi kprobe attach

Progresses: #87

Signed-off-by: Vincent Li [email protected]

[vincentmli]

tested this draft PR on kernel running 5.19 which should support multi kprobe, but got operation not supported, it appears the attach type is not set properly ? strace:

172287 bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0xc00055e8a0, license="MIT", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 0), prog_flags=0, prog_name="probe_bpf_kprob", prog_ifindex=0, expected_attach_type=0x2a /* BPF_??? */, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 144) = 7
172287 bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0xc00055e8f0, license="MIT", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 0), prog_flags=0, prog_name="probe_kpm_link", prog_ifindex=0, expected_attach_type=0x2a /* BPF_??? */, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 144) = 16
172287 bpf(BPF_LINK_CREATE, {link_create={prog_fd=16, target_fd=0, attach_type=0x2a /* BPF_??? */, flags=0}, ...}, 48) = -1 EOPNOTSUPP (Operation not supported)

[vincentmli]

upgraded one of my ubuntu 22.04 kernel to 5.19.11-051911-generic in case something wrong with my own custom kernel build, now I got different error from strace

2180 bpf(BPF_LINK_CREATE, {link_create={prog_fd=15, target_fd=0, attach_type=0x2a /* BPF_??? */, flags=0}, ...}, 48) = -1 EINVAL (Invalid argument)

2180 write(2, "2022/09/26 15:42:33 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_5)#15' failed: invalid argument (missing kernel symbol or prog's AttachType not AttachTraceKprobeMulti?)\n", 186 <unfinished ...>

[vincentmli]

I hacked the pwu main.go code to be only like below, the multi kprobe works on my ubuntu 22.04 with 5.19 kernel, seems something wrong with this draft PR code

package main

import (
        "fmt"

        "github.com/cilium/ebpf"
        "github.com/cilium/ebpf/asm"
        "github.com/cilium/ebpf/link"
)

func detectKprobeMulti() bool {
        prog, err := ebpf.NewProgram(&ebpf.ProgramSpec{
                Name: "probe_bpf_kprobe_multi_link",
                Type: ebpf.Kprobe,
                Instructions: asm.Instructions{
                        asm.Mov.Imm(asm.R0, 0),
                        asm.Return(),
                },
                AttachType: ebpf.AttachTraceKprobeMulti,
                License:    "MIT",
        })
        if err != nil {
                return false
        }
        defer prog.Close()

        syms := []string{"vprintk"}
        opts := link.KprobeMultiOptions{Symbols: syms}

        _, err = link.KprobeMulti(prog, opts)
        return err == nil
}

func main() {
        if detectKprobeMulti() {
                fmt.Println(" it works\n")
        }

[brb]

Could you run strace -f -y -yy -e bpf ./pwru ...?

[vincentmli]

limited to attach to kfree_skb function test, same error

root@vli-cilium-dev:/home/vincent/go/src/github.com/vincentmli/pwru# strace -f -y -yy -e bpf ./pwru --filter-func=".*kfree_skb.*

strace: Process 13990 attached
strace: Process 13991 attached
strace: Process 13992 attached
strace: Process 13993 attached
strace: Process 13994 attached
strace: Process 13995 attached
strace: Process 13996 attached

[pid 13999] bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0</dev/pts/0<char 136:0>>, map_name="feature_test", map_ifindex=0, btf_fd=0</dev/pts/0<char 136:0>>, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 7<anon_inode:bpf-map>
[pid 13999] bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0</dev/pts/0<char 136:0>>, map_name=".test", map_ifindex=0, btf_fd=0</dev/pts/0<char 136:0>>, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 7<anon_inode:bpf-map>
[pid 13999] bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\34\0\0\0\34\0\0\0\3\0\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=55, btf_log_size=0, btf_log_level=0}, 32) = 7<anon_inode:btf>
[pid 13999] bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\30\0\0\0\30\0\0\0\3\0\0\0\0\0\0\0\0\0\0\r"..., btf_log_buf=NULL, btf_size=51, btf_log_size=0, btf_log_level=0}, 32) = 7<anon_inode:btf>
[pid 13999] bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\330k\0\0\330k\0\0\331\\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=51401, btf_log_size=0, btf_log_level=0}, 32) = 7<anon_inode:btf>
[pid 13999] bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=52, max_entries=1, map_flags=0, inner_map_fd=0</dev/pts/0<char 136:0>>, map_name="cfg_map", map_ifindex=0, btf_fd=7<anon_inode:btf>, btf_key_type_id=20, btf_value_type_id=24, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 8<anon_inode:bpf-map>
[pid 13999] bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_STACK_TRACE, key_size=4, value_size=400, max_entries=256, map_flags=0, inner_map_fd=0</dev/pts/0<char 136:0>>, map_name="print_stack_map", map_ifindex=0, btf_fd=0</dev/pts/0<char 136:0>>, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 9<anon_inode:bpf-map>
[pid 13999] bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=8, map_flags=0, inner_map_fd=0</dev/pts/0<char 136:0>>, map_name="events", map_ifindex=0, btf_fd=0</dev/pts/0<char 136:0>>, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) = 10<anon_inode:bpf-map>

[pid 13999] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=6, insns=0xc001344030, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0</dev/pts/0<char 136:0>>, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 11<anon_inode:bpf-prog>
[pid 13999] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b16000, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_1", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc003c6a1c0, func_info_cnt=1, line_info_rec_size=16, line_info=0xc001b0d000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 11<anon_inode:bpf-prog>
[pid 13999] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=13989, si_uid=0} ---
[pid 13999] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b17980, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_2", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc003c6a4c0, func_info_cnt=1, line_info_rec_size=16, line_info=0xc0034a0000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 12<anon_inode:bpf-prog>
[pid 13999] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b19300, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_3", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc003c6a700, func_info_cnt=1, line_info_rec_size=16, line_info=0xc0034a1000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 13<anon_inode:bpf-prog>
[pid 13999] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=13989, si_uid=0} ---
[pid 13999] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b1ac80, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_4", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc003c6a940, func_info_cnt=1, line_info_rec_size=16, line_info=0xc004e28000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144 <unfinished ...>

[pid 14001] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b16000, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_5", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc0044521c0, func_info_cnt=1, line_info_rec_size=16, line_info=0xc004e29000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 15<anon_inode:bpf-prog>
2022/09/26 17:34:24 Per cpu buffer size: 4096 bytes
[pid 14001] bpf(BPF_MAP_UPDATE_ELEM, {map_fd=8<anon_inode:bpf-map>, key=0xc0044522c0, value=0xc004452340, flags=BPF_ANY}, 32) = 0
2022/09/26 17:34:24 Attaching kprobes...
[pid 14001] bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0xc000cb45f0, license="MIT", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="probe_kpm_link", prog_ifindex=0, expected_attach_type=0x2a /* BPF_??? /, prog_btf_fd=0</dev/pts/0<char 136:0>>, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 7<anon_inode:bpf-prog>
[pid 14001] bpf(BPF_LINK_CREATE, {link_create={prog_fd=7<anon_inode:bpf-prog>, target_fd=0</dev/pts/0<char 136:0>>, attach_type=0x2a / BPF_??? /, flags=0}, ...}, 48) = 16<anon_inode:bpf_link>
[pid 14001] bpf(BPF_LINK_CREATE, {link_create={prog_fd=11<anon_inode:bpf-prog>, target_fd=0</dev/pts/0<char 136:0>>, attach_type=0x2a / BPF_??? */, flags=0}, ...}, 48) = -1 EINVAL (Invalid argument)
2022/09/26 17:34:24 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_1)#11' failed: invalid argument (missing kernel symbol or prog's AttachType not AttachTraceKprobeMulti?)

[brb]

I think the attach type is wrong:

bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=802, insns=0xc001b17980, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 11), prog_flags=0, prog_name="kprobe_skb_2", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=7<anon_inode:btf>, func_info_rec_size=8, func_info=0xc003c6a4c0, func_info_cnt=1, line_info_rec_size=16, line_info=0xc0034a0000, line_info_cnt=229, attach_btf_id=0, attach_prog_fd=0</dev/pts/0<char 136:0>>, fd_array=NULL}, 144) = 12<anon_inode:bpf-prog>

It has to be BPF_TRACE_KPROBE_MULTI. Try the following patch (don't forget to regenerate with go generate .):

diff --git a/bpf/kprobe_pwru.c b/bpf/kprobe_pwru.c
index 73e5463..1639fed 100644
--- a/bpf/kprobe_pwru.c
+++ b/bpf/kprobe_pwru.c
@@ -380,35 +380,35 @@ handle_everything(struct sk_buff *skb, struct pt_regs *ctx) {
 	return 0;
 }
 
-SEC("kprobe/skb-1")
+SEC("kprobe.multi/skb-1")
 int kprobe_skb_1(struct pt_regs *ctx) {
 	struct sk_buff *skb = (struct sk_buff *) PT_REGS_PARM1(ctx);
 
 	return handle_everything(skb, ctx);
 }
 
-SEC("kprobe/skb-2")
+SEC("kprobe.multi/skb-2")
 int kprobe_skb_2(struct pt_regs *ctx) {
 	struct sk_buff *skb = (struct sk_buff *) PT_REGS_PARM2(ctx);
 
 	return handle_everything(skb, ctx);
 }
 
-SEC("kprobe/skb-3")
+SEC("kprobe.multi/skb-3")
 int kprobe_skb_3(struct pt_regs *ctx) {
 	struct sk_buff *skb = (struct sk_buff *) PT_REGS_PARM3(ctx);
 
 	return handle_everything(skb, ctx);
 }
 
-SEC("kprobe/skb-4")
+SEC("kprobe.multi/skb-4")
 int kprobe_skb_4(struct pt_regs *ctx) {
 	struct sk_buff *skb = (struct sk_buff *) PT_REGS_PARM4(ctx);
 
 	return handle_everything(skb, ctx);
 }
 
-SEC("kprobe/skb-5")
+SEC("kprobe.multi/skb-5")
 int kprobe_skb_5(struct pt_regs *ctx) {
 	struct sk_buff *skb = (struct sk_buff *) PT_REGS_PARM5(ctx);

[vincentmli]

ah, that is it!

[vincentmli]

ah, that is it!

hm, the code is still broken,

do filter on ip test

root@vli-cilium-dev:/home/vincent/go/src/github.com/vincentmli/pwru# ./pwru --filter-dst-ip 10.3.254.103
2022/09/26 18:04:38 Per cpu buffer size: 4096 bytes
2022/09/26 18:04:38 Attaching kprobes...
2022/09/26 18:04:38 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_1)#11' failed: couldn't find one or more symbols: file does not exist
2022/09/26 18:04:38 Listening for events..
SKB CPU PROCESS FUNC
0 / 1568 [________________________________________________________________________________________________________________] 0.00% ? p/s^C2022/09/26 18:04:42 Received signal, exiting program..
2022/09/26 18:04:42 Detaching kprobes...

filter on function name seems working

root@vli-cilium-dev:/home/vincent/go/src/github.com/vincentmli/pwru# ./pwru --filter-func=".*kfree_skb.*"

2022/09/26 17:57:00 Per cpu buffer size: 4096 bytes
2022/09/26 17:57:00 Attaching kprobes...
2022/09/26 17:57:00 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_2)#12' failed: couldn't find one or more symbols: file does not exist
2022/09/26 17:57:00 Attaching kprobes...
15 / 15 [-------------------------------------------------------------------------------------------------------------] 100.00% 201 p/s
2022/09/26 17:57:00 Attached (ignored 3)
2022/09/26 17:57:00 Listening for events..
SKB CPU PROCESS FUNC
0xffff940d43c786e8 5 [] __dev_kfree_skb_any
0xffff940d43c786e8 5 [] __dev_kfree_skb_any

[vincentmli]

added

        for prog, syms := range ksyms {
+               log.Printf("Attached (prog %s ) to (funcs %s)\n", prog, syms)
 
                opts := link.KprobeMultiOptions{Symbols: syms}
 

I can see the symbols printed out

root@vli-cilium-dev:/home/vincent/go/src/github.com/vincentmli/pwru# ./pwru --filter-func=".*kfree_skb.*"

2022/09/26 18:21:47 Per cpu buffer size: 4096 bytes
2022/09/26 18:21:47 Attaching multi kprobes...
2022/09/26 18:21:47 Attached (prog Kprobe(kprobe_skb_1)#11 ) to (funcs [__kfree_skb_defer __kfree_skb kfree_skb_list_reason rtnl_kfree_skbs kfree_skb_reason kfree_skbmem __dev_kfree_skb_irq kfree_skb_partial __dev_kfree_skb_any])
2022/09/26 18:21:47 Attached (prog Kprobe(kprobe_skb_2)#12 ) to (funcs [trace_kfree_skb_hit __bpf_trace_kfree_skb trace_event_raw_event_kfree_skb net_dm_packet_trace_kfree_skb_hit __traceiter_kfree_skb perf_trace_kfree_skb])
2022/09/26 18:21:47 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_2)#12' failed: couldn't find one or more symbols: file does not exist
2022/09/26 18:21:47 Listening for events..
SKB CPU PROCESS FUNC
0xffff940d43421ee8 1 [] __dev_kfree_skb_any
0xffff940d43421ee8 1 [] kfree_skbmem

[vincentmli]

I suspect for some reason the kernel multi kprobe could not find all the symbols supplied by pwru.

[vincentmli]

look through kernel code, wondering if the ESRCH is coming from https://elixir.bootlin.com/linux/latest/source/kernel/trace/ftrace.c#L8078

/**
 * ftrace_lookup_symbols - Lookup addresses for array of symbols
 *
 * @sorted_syms: array of symbols pointers symbols to resolve,
 * must be alphabetically sorted
 * @cnt: number of symbols/addresses in @syms/@addrs arrays
 * @addrs: array for storing resulting addresses
 *
 * This function looks up addresses for array of symbols provided in
 * @syms array (must be alphabetically sorted) and stores them in
 * @addrs array, which needs to be big enough to store at least @cnt
 * addresses.   
 *              
 * This function returns 0 if all provided symbols are found,
 * -ESRCH otherwise.
 */             
int ftrace_lookup_symbols(const char **sorted_syms, size_t cnt, unsigned long *addrs)
{
        struct kallsyms_data args;
        int err;       

        memset(addrs, 0, sizeof(*addrs) * cnt);
        args.addrs = addrs;
        args.syms = sorted_syms;
        args.cnt = cnt; 
        args.found = 0;
        err = kallsyms_on_each_symbol(kallsyms_callback, &args);
        if (err < 0)
                return err;
        return args.found == args.cnt ? 0 : -ESRCH;
}

[brb]

@syms array (must be alphabetically sorted)

Could you try sorting them? UPDATE: please ignore, as bpf_kprobe_multi_link_attach() sorts them.

[brb]

Looks like we need to check whether a function is in /sys/kernel/debug/tracing/available_filter_functions as per https://www.kernel.org/doc/Documentation/trace/ftrace.rst.

[vincentmli]

Looks like we need to check whether a function is in /sys/kernel/debug/tracing/available_filter_functions.

indeed functions like __bpf_trace_kfree_skb is missing in /sys/kernel/debug/tracing/available_filter_functions, I guess we have to grep search the functions manually in /sys/kernel/debug/tracing/available_filter_functions in golang?

[vincentmli]

kind of getting a half "working" code, it appears no guarantee all symbols can be attached

root@vli-cilium-dev:/home/vincent/go/src/github.com/vincentmli/pwru# ./pwru --filter-dst-ip="10.3.254.103" --filter-proto="icmp"

2022/09/26 22:40:19 Per cpu buffer size: 4096 bytes
2022/09/26 22:40:19 Attaching multi kprobes...
2022/09/26 22:40:19 Attaching (prog Kprobe(kprobe_skb_4)#14 ) to (funcs [__traceiter_qdisc_enqueue bpf_run_sk_reuseport check_fully_established cookie_tcp_reqsk_alloc ip6_mr_forward ip_mr_forward ipmr_queue_xmit mctp_addr_notify neigh_xmit tcf_chain_dump tcp_collapse tcp_conn_request tcp_synack_options])
2022/09/26 22:40:19 Attaching (prog Kprobe(kprobe_skb_2)#12 ) to (funcs [__cgroup_bpf_run_filter_skb __dev_forward_skb __dev_forward_skb2 __get_xps_queue_idx __ip_do_redirect __ip_queue_xmit __neigh_event_send __netlink_dump_start __netlink_policy_dump_write_attr __netlink_sendskb __netpoll_send_skb __ping_queue_rcv_skb __ppp_xmit_process __sk_backlog_rcv __sk_receive_skb __skb_free_datagram_locked __skb_send_sock __sock_queue_rcv_skb __strp_recv __tcp_retransmit_skb __tcp_transmit_skb __traceiter_consume_skb __traceiter_kfree_skb __traceiter_net_dev_queue __traceiter_net_dev_xmit __traceiter_netif_receive_skb __traceiter_netif_rx __udp4_lib_mcast_deliver __udp6_lib_mcast_deliver __udp_enqueue_schedule_skb __xdp_build_skb_from_frame __xfrm_mode_beet_prep __xfrm_mode_tunnel_prep __zerocopy_sg_from_iter _ip6mr_fill_mroute _ipmr_fill_mroute accept_all apparmor_inet_conn_request apparmor_ip_postroute apparmor_socket_getpeersec_dgram apparmor_socket_sock_rcv_skb arp_error_report arp_solicit bpf_lsm_inet_conn_established bpf_lsm_inet_conn_request bpf_lsm_netlink_send bpf_lsm_sctp_assoc_established bpf_lsm_sctp_assoc_request bpf_lsm_socket_getpeersec_dgram bpf_lsm_socket_sock_rcv_skb bpf_out_neigh_v6 cookie_v4_check cookie_v6_check cpu_map_generic_redirect dcbnl_build_peer_app dev_fill_metadata_dst dev_forward_skb dev_forward_skb_nomtu dev_gro_receive dev_map_generic_redirect dev_map_redirect_multi dev_pick_tx_cpu_id dev_pick_tx_zero devlink_fmsg_prepare_skb devlink_nl_post_doit devlink_nl_pre_doit devlink_trap_report diag_get do_xdp_generic dst_blackhole_neigh_lookup dummy_ipv6_icmp_error espintcp_parse espintcp_push_skb espintcp_queue_out espintcp_rcv eth_gro_receive ethnl_tunnel_info_fill_reply fanout_demux_rollover fib4_rule_configure fib4_rule_fill fib6_rule_configure fib6_rule_fill fib_table_dump fifo_dump fifo_hd_dump genl_family_rcv_msg_doit genl_family_rcv_msg_dumpit genl_notify genlmsg_multicast_allns get_rps_cpu gre_gro_receive gro_cells_receive handle_nonesp icmp_reply icmpv6_route_lookup igmp_heard_query igmpmsg_netlink_event in6_dump_addrs inet6_csk_xmit inet_frag_queue_insert inet_frag_reasm_finish inet_frag_reasm_prepare inet_gro_receive inet_proto_csum_replace16 inet_proto_csum_replace4 inet_proto_csum_replace_by_diff ioam6_do_encap ioam6_do_inline ip4ip6_gro_receive ip6_dst_neigh_lookup ip6_parse_tlv ip6_protocol_deliver_rcu ip6_xmit ip6mr_cache_report ip6mr_fill_mroute ip6mr_get_route ip6mr_rule_configure ip6mr_rule_fill ip_check_defrag ip_defrag ip_encap ip_frag_queue ip_icmp_error ip_protocol_deliver_rcu ip_queue_xmit ip_rt_get_source ip_send_skb ip_send_unicast_reply ipip_gro_receive ipmr_cache_report ipmr_fill_mroute ipmr_fill_table ipmr_get_route ipmr_rt_fib_lookup ipmr_rule_configure ipmr_rule_fill ipv4_neigh_lookup ipv4_pktinfo_prepare ipv6_gro_receive ipv6_icmp_error ipv6_proxy_select_ident mctp_do_fragment_route mctp_frag_queue mctp_route_discard mctp_route_input mctp_route_output mptcp_data_queue_ofo mptcp_established_options mptcp_established_options_add_addr mptcp_incoming_options mptcp_token_join_cookie_init_state mptcp_try_coalesce mq_dump mr_fill_mroute mr_table_dump mroute6_is_socket mrt6msg_netlink_event napi_gro_receive napi_reuse_skb ncsi_aen_handler ndisc_error_report ndisc_solicit neigh_blackhole neigh_connected_output neigh_direct_output neigh_dump_table neigh_resolve_output net_dm_nl_post_doit net_dm_nl_pre_doit net_dm_packet_trace_kfree_skb_hit netdev_core_pick_tx netdev_get_xmit_slave netdev_pick_tx netdev_rx_csum_fault netlink_attachskb netlink_broadcast netlink_detachskb netlink_dump_done netlink_sendskb netlink_unicast netpoll_send_skb nlmsg_notify pfifo_fast_dump ping_queue_rcv_skb pktgen_xfrm_outer_mode_output ppp_decompress_frame ppp_input ppp_mp_explode ppp_receive_frame ppp_receive_mp_frame ppp_receive_nonmp_frame ppp_send_frame ppp_xmit_process raw_rcv raw_rcv_skb rawv6_rcv rt6_fill_node rtm_to_fib_config rtm_to_nh_config rtnetlink_ifinfo_prep sch_fragment security_inet_conn_established security_inet_conn_request security_netlink_send security_sctp_assoc_established security_sctp_assoc_request security_sock_rcv_skb security_socket_getpeersec_dgram selinux_inet_conn_established selinux_inet_conn_request selinux_ip_forward selinux_ip_output selinux_ip_postroute selinux_netlbl_sctp_assoc_request selinux_netlbl_sock_rcv_skb selinux_netlink_send selinux_sctp_assoc_established selinux_sctp_assoc_request selinux_sctp_process_new_assoc selinux_sock_rcv_skb_compat selinux_socket_getpeersec_dgram selinux_socket_sock_rcv_skb selinux_xfrm_postroute_last selinux_xfrm_sock_rcv_skb sit_ip6ip6_gro_receive sk_filter_trim_cap sk_psock_skb_ingress_self sk_psock_skb_redirect sk_psock_strp_parse sk_psock_strp_read sk_psock_tls_strp_read sk_psock_verdict_apply sk_psock_verdict_recv skb_consume_udp skb_free_datagram skb_kill_datagram skb_queue_head skb_queue_tail skb_send_sock skb_send_sock_locked skb_zerocopy_iter_stream smack_inet_conn_request smack_ip_output smack_socket_getpeersec_dgram smack_socket_sock_rcv_skb sock_diag_put_meminfo sock_queue_err_skb sock_queue_rcv_skb_reason strp_process strp_recv subflow_get_info subflow_init_req_cookie_join_save subflow_syn_recv_sock subflow_v4_conn_request subflow_v4_route_req subflow_v6_conn_request subflow_v6_route_req tc_chain_notify tc_dump_qdisc_root tc_dump_tclass_qdisc tc_dump_tclass_root tcf_dump_walker tcf_fill_node tcf_generic_walker tcp4_gro_receive tcp6_gro_receive tcp_ack_tstamp tcp_add_backlog tcp_check_req tcp_collapse_one tcp_connect_queue_skb tcp_data_queue tcp_data_queue_ofo tcp_downgrade_zcopy_pure tcp_established_options tcp_event_data_recv tcp_event_new_data_sent tcp_fastopen_add_skb tcp_fastopen_create_child tcp_filter tcp_finish_connect tcp_get_cookie_sock tcp_gro_receive tcp_mark_push tcp_mark_skb_lost tcp_match_skb_to_sack tcp_queue_rcv tcp_rack_skb_timeout tcp_rate_skb_delivered tcp_rate_skb_sent tcp_rbtree_insert tcp_rcv_established tcp_rcv_fastopen_synack tcp_rcv_state_process tcp_rcv_synsent_state_process tcp_reset tcp_retrans_try_collapse tcp_retransmit_skb tcp_send_syn_data tcp_shift_skb_data tcp_shifted_skb tcp_skb_entail tcp_splice_data_recv tcp_syn_options tcp_timewait_state_process tcp_trim_head tcp_try_coalesce tcp_try_fastopen tcp_try_rmem_schedule tcp_update_skb_after_send tcp_urg tcp_v4_conn_request tcp_v4_do_rcv tcp_v4_reqsk_send_ack tcp_v4_route_req tcp_v4_send_ack tcp_v4_send_check tcp_v4_send_reset tcp_v4_syn_recv_sock tcp_v6_conn_request tcp_v6_do_rcv tcp_v6_reqsk_send_ack tcp_v6_route_req tcp_v6_send_check tcp_v6_send_reset tcp_v6_send_response tcp_v6_syn_recv_sock tcp_validate_incoming tfilter_notify tpacket_fill_skb trace_kfree_skb_hit tso_fragment tun_select_queue udp4_gro_receive udp6_gro_receive udp6_set_csum udp6_unicast_rcv_skb udp_gro_receive udp_gro_receive_segment udp_queue_rcv_one_skb udp_queue_rcv_skb udp_set_csum udp_skb_destructor udp_unicast_rcv_skb udpv6_queue_rcv_one_skb udpv6_queue_rcv_skb unix_attach_fds unix_detach_fds validate_data_csum vlan_gro_receive wwan_port_rx xdp_do_generic_redirect xennet_move_rx_slot xennet_select_queue xfrm4_remove_beet_encap xfrm4_remove_tunnel_encap xfrm4_udp_encap_rcv xfrm6_remove_beet_encap xfrm6_remove_tunnel_encap xfrm6_udp_encap_rcv xfrm_audit_state_icvfail xfrm_audit_state_replay xfrm_audit_state_replay_overflow xfrm_inner_extract_output xfrm_neigh_lookup xfrm_outer_mode_output xfrm_outer_mode_prep xfrm_output xfrm_output_resume xfrm_prepare_input xfrm_replay_check xfrm_replay_check_bmp xfrm_replay_check_esn xfrm_replay_check_legacy xfrm_replay_overflow xfrm_replay_recheck xfrm_trans_queue_net])
2022/09/26 22:40:19 Attaching (prog Kprobe(kprobe_skb_1)#11 ) to (funcs [___pskb_trim __bpf_redirect __bpf_redirect_neigh __bpf_redirect_no_mac __bpf_skc_lookup __build_skb_around __consume_stateless_skb __copy_skb_header _dev_direct_xmit dev_kfree_skb_any dev_kfree_skb_irq dev_queue_xmit fib_validate_source icmp_send ioam6_fill_trace_data iptunnel_pull_header ipv4_sk_update_pmtu kfree_skb kfree_skb_defer mkroute_input mld_query_work ndisc_fill_addr_option netif_receive_skb netif_receive_skb_one_core netif_rx netlink_deliver_tap nf_queue nlmsg_put pskb_copy_fclone pskb_pull_tail pskb_trim_head qdisc_calculate_pkt_len rtnl_newlink skb_checksum_complete skb_checksum_complete_head skb_clone skb_complete_tx_timestamp skb_ext_del skb_ext_set skb_get_hash skb_gro_checksum_complete skb_gso_segment skb_pad skb_to_sgvec skb_tstamp_tx skb_udp_tunnel_segment skb_unclone_keeptruesize skb_vlan_pop sock_wfree tcf_em_tree_match tcp_v4_send_check udp4_lib_err udp4_lib_rcv udp6_lib_err udp6_lib_rcv udp_gso_segment udpv4_gso_segment_csum xfrm_decode_session xfrm_route_forward add_grec add_grec add_grhead add_grhead addrconf_dad_failure alloc_skb_for_msg arp_rcv arp_xmit audit_list_rules_send audit_receive audit_receive_msg bfifo_enqueue blackhole_enqueue blackhole_netdev_xmit bpf_fill_encap_info bpf_input bpf_lsm_xfrm_decode_session bpf_lwt_input_reroute bpf_lwt_push_ip_encap bpf_lwt_xmit_reroute bpf_prog_run_generic_xdp bpf_push_seg6_encap bpf_sk_lookup bpf_skb_net_grow bpf_skb_net_hdr_pop bpf_skb_net_hdr_push bpf_update_srh_state bpf_xmit build_skb_around calipso_skbuff_delattr calipso_skbuff_delattr calipso_skbuff_setattr calipso_skbuff_setattr cgroupstats_user_cmd channels_fill_reply cipso_v4_error cipso_v4_skbuff_delattr cipso_v4_skbuff_setattr cn_rx_skb coalesce_fill_reply coalesce_put_bool coalesce_put_u32 consume_skb convert___skb_to_skb crypto_acomp_report crypto_aead_report crypto_ahash_report crypto_akcipher_report crypto_kpp_report crypto_rng_report crypto_scomp_report crypto_shash_report crypto_skcipher_report ctrl_dumpfamily ctrl_dumppolicy ctrl_dumppolicy_prep ctrl_dumppolicy_put_op ctrl_getfamily dcb_doit dcbnl_cee_fill dcbnl_cee_pg_fill dcbnl_ieee_fill debug_fill_reply decap_and_validate decode_session4 decode_session6 dev_hard_start_xmit dev_qdisc_enqueue dev_queue_xmit_nit devlink_dpipe_action_put devlink_dpipe_entry_put devlink_dpipe_match_put devlink_dpipe_table_put devlink_dpipe_value_put devlink_nl_cmd_dpipe_entries_get devlink_nl_cmd_dpipe_headers_get devlink_nl_cmd_dpipe_table_counters_set devlink_nl_cmd_dpipe_table_get devlink_nl_cmd_eswitch_get_doit devlink_nl_cmd_eswitch_set_doit devlink_nl_cmd_flash_update devlink_nl_cmd_get_doit devlink_nl_cmd_get_dumpit devlink_nl_cmd_health_reporter_diagnose_doit devlink_nl_cmd_health_reporter_dump_clear_doit devlink_nl_cmd_health_reporter_dump_get_dumpit devlink_nl_cmd_health_reporter_get_doit devlink_nl_cmd_health_reporter_get_dumpit devlink_nl_cmd_health_reporter_recover_doit devlink_nl_cmd_health_reporter_set_doit devlink_nl_cmd_health_reporter_test_doit devlink_nl_cmd_info_get_doit devlink_nl_cmd_info_get_dumpit devlink_nl_cmd_linecard_get_doit devlink_nl_cmd_linecard_get_dumpit devlink_nl_cmd_linecard_set_doit devlink_nl_cmd_param_get_doit devlink_nl_cmd_param_get_dumpit devlink_nl_cmd_param_set_doit devlink_nl_cmd_port_del_doit devlink_nl_cmd_port_get_doit devlink_nl_cmd_port_get_dumpit devlink_nl_cmd_port_new_doit devlink_nl_cmd_port_param_get_doit devlink_nl_cmd_port_param_get_dumpit devlink_nl_cmd_port_param_set_doit devlink_nl_cmd_port_set_doit devlink_nl_cmd_port_split_doit devlink_nl_cmd_port_unsplit_doit devlink_nl_cmd_rate_del_doit devlink_nl_cmd_rate_get_doit devlink_nl_cmd_rate_get_dumpit devlink_nl_cmd_rate_new_doit devlink_nl_cmd_rate_set_doit devlink_nl_cmd_region_del devlink_nl_cmd_region_get_doit devlink_nl_cmd_region_get_dumpit devlink_nl_cmd_region_new devlink_nl_cmd_region_read_dumpit devlink_nl_cmd_reload devlink_nl_cmd_resource_dump devlink_nl_cmd_resource_set devlink_nl_cmd_sb_get_doit devlink_nl_cmd_sb_get_dumpit devlink_nl_cmd_sb_occ_max_clear_doit devlink_nl_cmd_sb_occ_snapshot_doit devlink_nl_cmd_sb_pool_get_doit devlink_nl_cmd_sb_pool_get_dumpit devlink_nl_cmd_sb_pool_set_doit devlink_nl_cmd_sb_port_pool_get_doit devlink_nl_cmd_sb_port_pool_get_dumpit devlink_nl_cmd_sb_port_pool_set_doit devlink_nl_cmd_sb_tc_pool_bind_get_doit devlink_nl_cmd_sb_tc_pool_bind_get_dumpit devlink_nl_cmd_sb_tc_pool_bind_set_doit devlink_nl_cmd_trap_get_doit devlink_nl_cmd_trap_get_dumpit devlink_nl_cmd_trap_group_get_doit devlink_nl_cmd_trap_group_get_dumpit devlink_nl_cmd_trap_group_set_doit devlink_nl_cmd_trap_policer_get_doit devlink_nl_cmd_trap_policer_get_dumpit devlink_nl_cmd_trap_policer_set_doit devlink_nl_cmd_trap_set_doit devlink_nl_fill devlink_nl_flash_update_fill devlink_nl_health_reporter_fill devlink_nl_param_fill devlink_nl_param_value_fill_one devlink_nl_port_attrs_put devlink_nl_port_fill devlink_nl_port_function_attrs_put devlink_nl_put_handle devlink_nl_trap_fill devlink_nl_trap_group_fill devlink_nl_trap_policer_fill devlink_reload_stats_put dump_rules eafnosupport_ipv6_route_input eee_fill_reply eeprom_fill_reply end_dt_vrf_core enqueue_to_backlog eth_gro_complete eth_header eth_type_trans ethnl_act_cable_test ethnl_act_cable_test_tdr ethnl_bcastmsg_put ethnl_default_doit ethnl_default_dumpit ethnl_dump_put ethnl_fill_reply_header ethnl_multicast ethnl_put_bitset ethnl_put_bitset32 ethnl_set_channels ethnl_set_coalesce ethnl_set_debug ethnl_set_eee ethnl_set_features ethnl_set_fec ethnl_set_linkinfo ethnl_set_linkmodes ethnl_set_module ethnl_set_pause ethnl_set_privflags ethnl_set_rings ethnl_set_wol ethnl_tunnel_info_doit ethnl_tunnel_info_dumpit fc_header fddi_header fddi_type_trans features_fill_reply fec_fill_reply fib_add_nexthop fib_compute_spec_dst fib_dump_info fib_dump_info_fnhe fib_nexthop_info fib_nl_delrule fib_nl_dumprule fib_nl_fill_rule fib_nl_newrule fib_validate_source generic_xdp_tx genl_lock_dumpit genl_rcv genl_rcv_msg genlmsg_put gnet_stats_start_copy gnet_stats_start_copy_compat gre_gro_complete gre_gso_segment gro_pull_from_frag0 icmp6_send icmp_build_probe icmp_discard icmp_echo icmp_err icmp_ndo_send icmp_rcv icmp_redirect icmp_socket_deliver icmp_timestamp icmp_unreach icmpv6_echo_reply icmpv6_err icmpv6_ndo_send icmpv6_notify icmpv6_param_prob_reason icmpv6_rcv igmp6_event_query igmp6_event_report igmp_rcv inet6_dump_addr inet6_dump_fib inet6_dump_ifacaddr inet6_dump_ifaddr inet6_dump_ifinfo inet6_dump_ifmcaddr inet6_fill_ifacaddr inet6_fill_ifaddr inet6_fill_ifinfo inet6_fill_ifla6_attrs inet6_fill_link_af inet6_netconf_dump_devconf inet6_netconf_fill_devconf inet6_netconf_get_devconf inet6_rtm_deladdr inet6_rtm_delroute inet6_rtm_getaddr inet6_rtm_getroute inet6_rtm_newaddr inet6_rtm_newroute inet6_rtm_valid_getroute_req inet_dump_fib inet_dump_ifaddr inet_fill_ifaddr inet_fill_link_af inet_gro_complete inet_gso_segment inet_netconf_dump_devconf inet_netconf_fill_devconf inet_netconf_get_devconf inet_rtm_deladdr inet_rtm_delroute inet_rtm_getroute inet_rtm_newaddr inet_rtm_newroute inet_rtm_valid_getroute_req input_action_end input_action_end_b6 input_action_end_b6_encap input_action_end_bpf input_action_end_dt4 input_action_end_dt46 input_action_end_dt6 input_action_end_dx2 input_action_end_dx4 input_action_end_dx6 input_action_end_t input_action_end_x ioam6_fill_encap_info ioam6_fill_trace_data ioam6_genl_addns ioam6_genl_addsc ioam6_genl_delns ioam6_genl_delsc ioam6_genl_dumpns ioam6_genl_dumpsc ioam6_genl_ns_set_schema ip4ip6_gro_complete ip4ip6_gso_segment ip6_copy_metadata ip6_dst_lookup_tunnel ip6_err_gen_icmpv6_unreach ip6_find_1stfragopt ip6_forward ip6_frag_init ip6_frag_next ip6_fraglist_init ip6_fraglist_prepare ip6_input ip6_link_failure ip6_mc_input ip6_mr_input ip6_pkt_discard ip6_pkt_drop ip6_pkt_prohibit ip6_rcv_core ip6_redirect ip6_redirect_no_header ip6_route_input ip6_send_skb ip6_sk_redirect ip6_sk_update_pmtu ip6_tlvopt_unknown ip6_tun_fill_encap_info ip6_update_pmtu ip6addrlbl_dump ip6addrlbl_get ip6addrlbl_newdel ip6ip6_gro_complete ip6ip6_gso_segment ip6mr_rtm_dumproute ip_build_and_send_pkt ip_call_ra_chain ip_copy_metadata ip_error ip_forward ip_forward_options ip_frag_init ip_frag_next ip_fraglist_init ip_fraglist_prepare ip_local_deliver ip_mc_check_igmp ip_mc_validate_checksum ip_mc_validate_source ip_mr_input ip_options_build ip_options_fragment ip_options_rcv_srr ip_rcv ip_rcv_core ip_route_input_noref ip_route_input_rcu ip_route_input_slow ip_route_output_tunnel ip_route_use_hint ip_rt_send_redirect ip_tun_fill_encap_info ipip_gro_complete ipip_gso_segment ipmr_rtm_dumplink ipmr_rtm_dumproute ipmr_rtm_getroute ipmr_rtm_route ipmr_rtm_valid_getroute_req iptunnel_handle_offloads iptunnel_pmtud_build_icmp iptunnel_pmtud_build_icmpv6 ipv4_link_failure ipv4_redirect ipv4_send_dest_unreach ipv4_sk_redirect ipv4_sk_update_pmtu ipv4_skb_to_auditdata ipv4_update_pmtu ipv6_dest_hao ipv6_destopt_rcv ipv6_frag_rcv ipv6_gro_complete ipv6_gso_pull_exthdrs ipv6_gso_segment ipv6_mc_check_mld ipv6_mc_check_mld_msg ipv6_parse_hopopts ipv6_push_exthdr ipv6_push_frag_opts ipv6_push_nfrag_opts ipv6_push_rthdr4 ipv6_raw_deliver ipv6_rcv ipv6_route_input ipv6_rpl_srh_rcv ipv6_rthdr_rcv ipv6_skb_to_auditdata ipv6_srh_rcv kauditd_hold_skb kauditd_rehold_skb kauditd_retry_skb kauditd_send_multicast_skb kfree_skb_list_reason kfree_skb_partial kfree_skb_reason kfree_skbmem linkinfo_fill_reply linkmodes_fill_reply linkstate_fill_reply loopback_xmit lwtunnel_fill_encap lwtunnel_input lwtunnel_xmit maybe_add_creds mctp_delroute mctp_dump_addrinfo mctp_dump_rtinfo mctp_fill_addrinfo mctp_fill_link_af mctp_newroute mctp_pkttype_receive mctp_rtm_deladdr mctp_rtm_delneigh mctp_rtm_getneigh mctp_rtm_newaddr mctp_rtm_newneigh mk_reply mld_sendpack module_fill_reply mptcp_event_add_subflow mptcp_event_put_token_and_ssk mptcp_join_entry_hash mptcp_nl_cmd_add_addr mptcp_nl_cmd_announce mptcp_nl_cmd_del_addr mptcp_nl_cmd_dump_addrs mptcp_nl_cmd_flush_addrs mptcp_nl_cmd_get_addr mptcp_nl_cmd_get_limits mptcp_nl_cmd_remove mptcp_nl_cmd_set_flags mptcp_nl_cmd_set_limits mptcp_nl_cmd_sf_create mptcp_nl_cmd_sf_destroy mptcp_nl_fill_addr mptcp_rfree mptcp_set_owner_r mptcp_update_data_checksum mr_rtm_dumproute msg_zerocopy_callback napi_consume_skb napi_skb_free_stolen_head ncsi_clear_interface_nl ncsi_cmd_handler_ae ncsi_cmd_handler_dc ncsi_cmd_handler_default ncsi_cmd_handler_ebf ncsi_cmd_handler_egmf ncsi_cmd_handler_ev ncsi_cmd_handler_oem ncsi_cmd_handler_rc ncsi_cmd_handler_sl ncsi_cmd_handler_sma ncsi_cmd_handler_snfc ncsi_cmd_handler_sp ncsi_cmd_handler_svf ncsi_pkg_info_all_nl ncsi_pkg_info_nl ncsi_rcv_rsp ncsi_send_cmd_nl ncsi_set_channel_mask_nl ncsi_set_interface_nl ncsi_set_package_mask_nl ncsi_write_package_info ndisc_ra_useropt ndisc_rcv ndisc_recv_na ndisc_recv_ns ndisc_recv_rs ndisc_redirect_rcv ndisc_router_discovery ndisc_send_redirect ndisc_send_skb ndo_dflt_bridge_getlink ndo_dflt_fdb_dump neigh_add neigh_delete neigh_dump_info neigh_fill_info neigh_get neightbl_dump_info neightbl_fill_parms neightbl_set net_dm_cmd_config net_dm_cmd_config_get net_dm_cmd_stats_get net_dm_cmd_trace net_dm_hw_packet_report_fill net_dm_hw_summary_report_fill net_dm_packet_report_fill net_dm_packet_report_in_port_put net_test_loopback_validate netif_receive_generic_xdp netif_receive_skb netif_receive_skb_core netif_rx netif_rx_internal netif_skb_features netlbl_calipso_add netlbl_calipso_list netlbl_calipso_listall netlbl_calipso_remove netlbl_cipsov4_add netlbl_cipsov4_list netlbl_cipsov4_listall netlbl_cipsov4_remove netlbl_mgmt_add netlbl_mgmt_adddef netlbl_mgmt_listall netlbl_mgmt_listdef netlbl_mgmt_listentry netlbl_mgmt_protocols netlbl_mgmt_remove netlbl_mgmt_removedef netlbl_mgmt_version netlbl_skbuff_err netlbl_skbuff_setattr netlbl_unlabel_accept netlbl_unlabel_list netlbl_unlabel_staticadd netlbl_unlabel_staticadddef netlbl_unlabel_staticlist netlbl_unlabel_staticlistdef netlbl_unlabel_staticremove netlbl_unlabel_staticremovedef netlink_ack netlink_policy_dump_write netlink_policy_dump_write_attr netlink_rcv_skb netlink_skb_destructor netlink_skb_set_owner_r netlink_strict_get_check netlink_trim netpoll_start_xmit nf_checksum nf_checksum_partial nf_ct_attach nf_hook_direct_egress nf_hook_slow nf_ip6_checksum nf_ip6_reroute nf_ip_checksum nf_queue nf_reroute nh_fill_node nl_fib_input nla_put_ifalias nla_put_nh_group_res nlmsg_populate_fdb noop_enqueue packet_direct_xmit packet_parse_headers packet_rcv packet_rcv_fanout packet_rcv_spkt parp_redo passthru_features_check pause_fill_reply pfifo_enqueue pfifo_fast_enqueue pfifo_tail_enqueue phc_vclocks_fill_reply pim6_rcv pim_rcv pim_rcv_v1 ping_err ping_rcv pndisc_redo ppp_nl_fill_info ppp_start_xmit privflags_fill_reply pskb_carve_inside_header pskb_carve_inside_nonlinear pskb_expand_head pskb_extract pskb_put pskb_trim_rcsum_slow ptp_msg_is_sync ptp_parse_header put_nla_bpf put_nla_counters put_nla_iif put_nla_nh4 put_nla_nh6 put_nla_oif put_nla_srh put_nla_table put_nla_vrftable qdisc_pkt_len_init raw6_icmp_error raw6_local_deliver raw_icmp_error raw_local_deliver raw_v4_input reg_vif_xmit reg_vif_xmit rings_fill_reply rtm_del_nexthop rtm_dump_nexthop rtm_dump_nexthop_bucket rtm_dump_nexthop_bucket_cb rtm_dump_nexthop_bucket_nh rtm_get_nexthop rtm_get_nexthop_bucket rtm_new_nexthop rtm_to_fib6_config rtmsg_ifinfo_send rtnetlink_put_metrics rtnetlink_rcv rtnetlink_rcv_msg rtnetlink_send rtnl_bridge_dellink rtnl_bridge_getlink rtnl_bridge_setlink rtnl_dellink rtnl_dellinkprop rtnl_dump_all rtnl_dump_ifinfo rtnl_fdb_add rtnl_fdb_del rtnl_fdb_dump rtnl_fdb_get rtnl_fill_ifinfo rtnl_fill_stats rtnl_fill_vf rtnl_fill_vfinfo rtnl_getlink rtnl_kfree_skbs rtnl_link_fill rtnl_net_dumpid rtnl_net_fill rtnl_net_getid rtnl_net_newid rtnl_newlink rtnl_newlink_create rtnl_newlinkprop rtnl_notify rtnl_offload_xstats_fill rtnl_put_cacheinfo rtnl_setlink rtnl_stats_dump rtnl_stats_get rtnl_stats_set rtnl_unicast rtnl_xdp_fill sch_direct_xmit sch_frag_prepare_frag sch_frag_xmit_hook scsi_nl_rcv_msg secpath_set security_skb_classify_flow security_xfrm_decode_session seg6_bpf_has_valid_srh seg6_do_srh seg6_do_srh_encap seg6_do_srh_inline seg6_fill_encap_info seg6_genl_dumphmac seg6_genl_get_tunsrc seg6_genl_set_tunsrc seg6_genl_sethmac seg6_get_srh seg6_hmac_validate_skb seg6_icmp_srh seg6_input seg6_local_fill_encap seg6_local_input seg6_lookup_any_nexthop seg6_lookup_nexthop selinux_ip_postroute_compat selinux_netlbl_err selinux_netlbl_skbuff_getsid selinux_netlbl_skbuff_setsid selinux_skb_peerlbl_sid selinux_xfrm_decode_session selinux_xfrm_skb_sid selinux_xfrm_skb_sid_ingress sit_gro_complete sit_gso_segment sk_psock_skb_ingress_enqueue skb_add_rx_frag skb_advance_to_frag skb_append skb_append_pagefrags skb_attempt_defer_free skb_checksum_help skb_checksum_setup skb_checksum_setup_ip skb_checksum_setup_ipv6 skb_checksum_trimmed skb_clone skb_clone_sk skb_clone_tx_timestamp skb_coalesce_rx_frag skb_complete_tx_timestamp skb_complete_wifi_ack skb_condense skb_copy_and_csum_datagram_msg skb_copy_datagram_from_iter skb_copy_header skb_copy_ubufs skb_cow_data skb_crc32c_csum_help skb_csum_hwoffload_help skb_defer_rx_timestamp skb_do_redirect skb_ensure_writable skb_eth_gso_segment skb_eth_pop skb_eth_push skb_expand_head skb_ext_add skb_find_text skb_free_head skb_gro_receive skb_headers_offset_update skb_mac_gso_segment skb_morph skb_mpls_dec_ttl skb_mpls_pop skb_mpls_push skb_mpls_update_lse skb_network_protocol skb_orphan_partial skb_panic skb_partial_csum_set skb_prepare_seq_read skb_pull skb_pull_data skb_pull_rcsum skb_push skb_put skb_realloc_headroom skb_release_data skb_release_head_state skb_reorder_vlan_header skb_scrub_packet skb_segment skb_segment_list skb_set_owner_w skb_shift skb_splice_bits skb_split skb_store_bits skb_to_sgvec skb_to_sgvec_nomark skb_trim skb_try_coalesce skb_tstamp_tx skb_tunnel_check_pmtu skb_tx_error skb_udp_tunnel_segment skb_unlink skb_vlan_pop skb_vlan_push skb_vlan_untag skb_zerocopy skb_zerocopy_clone smk_skb_to_addr_ipv6 sock_diag_rcv sock_diag_rcv_msg sock_edemux sock_efree sock_ofree sock_pfree sock_rfree sock_rmem_free sock_wfree stat_put stats_fill_reply stats_put_ctrl_stats stats_put_mac_stats stats_put_phy_stats stats_put_rmon_hist stats_put_rmon_stats stats_put_stats strset_fill_reply subflow_add_reset_reason taskstats_user_cmd tc_ctl_action tc_ctl_chain tc_ctl_tclass tc_del_tfilter tc_dump_action tc_dump_chain tc_dump_qdisc tc_dump_tclass tc_dump_tfilter tc_fill_qdisc tc_fill_tclass tc_get_qdisc tc_get_tfilter tc_modify_qdisc tc_new_tfilter tcf_action_copy_stats tcf_action_dump tcf_action_dump_1 tcf_action_dump_old tcf_action_dump_terse tcf_action_exec tcf_classify tcf_dev_queue_xmit tcf_em_tree_dump tcf_exts_dump tcf_exts_dump_stats tcf_exts_terse_dump tcf_qevent_dump tcp4_gro_complete tcp4_gso_segment tcp6_gro_complete tcp6_gso_segment tcp_fragment_tstamp tcp_gro_complete tcp_gso_segment tcp_init_tso_segs tcp_metrics_fill_info tcp_metrics_nl_cmd_del tcp_metrics_nl_cmd_get tcp_metrics_nl_dump tcp_sacktag_walk tcp_skb_collapse_tstamp tcp_skb_shift tcp_update_recv_tstamps tcp_v4_early_demux tcp_v4_err tcp_v4_fill_cb tcp_v4_rcv tcp_v4_restore_cb tcp_v6_early_demux tcp_v6_err tcp_v6_fill_cb tcp_v6_rcv tcp_v6_restore_cb tcp_wfree thermal_genl_cmd_doit thermal_genl_cmd_dumpit tpacket_destruct_skb tpacket_get_timestamp tpacket_rcv tsinfo_fill_reply tso_start tun_fill_info tun_net_xmit udp4_gro_complete udp4_hwcsum udp4_ufo_fragment udp6_csum_init udp6_gro_complete udp6_ufo_fragment udp_err udp_gro_complete udp_rcv udp_send_skb udp_v4_early_demux udp_v6_early_demux udp_v6_send_skb udplite_err udplite_rcv udplitev6_err udplitev6_rcv udpv6_rcv unix_destruct_scm unix_stream_read_actor unix_stream_splice_actor validate_xmit_skb validate_xmit_skb_list validate_xmit_xfrm vlan_gro_complete wol_fill_reply wwan_rtnl_fill_info xennet_start_xmit xfrm4_ah_err xfrm4_ah_rcv xfrm4_esp_err xfrm4_esp_rcv xfrm4_ipcomp_err xfrm4_ipcomp_rcv xfrm4_local_error xfrm4_rcv xfrm4_rcv_cb xfrm4_rcv_encap xfrm4_transport_finish xfrm6_ah_err xfrm6_ah_rcv xfrm6_esp_err xfrm6_esp_rcv xfrm6_input_addr xfrm6_ipcomp_err xfrm6_ipcomp_rcv xfrm6_local_error xfrm6_local_rxpmtu xfrm6_rcv xfrm6_rcv_cb xfrm6_rcv_encap xfrm6_rcv_spi xfrm6_rcv_tnl xfrm6_transport_finish xfrm6_tunnel_check_size xfrm_audit_helper_pktinfo xfrm_audit_state_notfound xfrm_audit_state_notfound_simple xfrm_dev_offload_ok xfrm_dev_resume xfrm_input xfrm_input_resume xfrm_link_failure xfrm_local_error xfrm_output_one xfrm_parse_spi xfrm_rcv_cb xfrm_trans_queue xsk_destruct_skb zerocopy_sg_from_iter])
2022/09/26 22:40:19 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_1)#11' failed: couldn't find one or more symbols: file does not exist
2022/09/26 22:40:19 Listening for events..
SKB CPU PROCESS FUNC
2 / 1568 [>] 0.13% ? p/s
0xffff940d4cb37e00 3 [] inet_gro_receive
0xffff940d4cb37e00 3 [] ip_protocol_deliver_rcu
0xffff940d4cb37e00 3 [] icmp_reply

[vincentmli]

2022/09/26 22:40:19 Failed to attach multi kprobes: attaching 'Kprobe(kprobe_skb_1)#11' failed: couldn't find one or more symbols: file does not exist 2022/09/26 22:40:19 Listening for events.. SKB CPU PROCESS FUNC 2 / 1568

to confirm, i added log in kernel, and indeed it is ftrace_lookup_symbols returning ESRCH

diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 68e5cdd24cef..ec3f95e10660 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -2525,6 +2525,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
                       symbols_swap_r, &data);
 
                err = ftrace_lookup_symbols(us.syms, cnt, addrs);
+               pr_info("ftrace_lookup_symbols return %d\n", err);
                free_user_syms(&us);
                if (err)
                        goto error;

[root@centos-dev bpf-next]# tail -f /var/log/kern.log

Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return 0
Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return 0
Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return 0
Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return 0
Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return 0
Sep 26 19:21:09 centos-dev kernel: ftrace_lookup_symbols return -3 <========ESRCH

[vincentmli]

looked further __bpf_redirect_no_mac is not in /sys/kernel/debug/tracing/available_filter_functions but is still in the traced function log above, looks interSection() is not doing the check properly

grep '__bpf_redirect_no_mac' /sys/kernel/debug/tracing/available_filter_functions

[vincentmli]

it seems working reliably now after taking interSection logic code from https://go.dev/play/p/eGGcyIlZD6y :). the logic is outer loop to get each string from first slice, inner loop to compare the string from first slice to each string in second slice, if it is same string and hash map keyed by the string not exist, add the string to the new slice, set the hash map keyed by the string to true to avoid duplicate I guess.

[vincentmli]

now the draft PR seems working fine for both single kprobe and multi kprobe, need some clean up work, maybe move all the functions other than main from main.go to another go file in internal/pwru directory

[vincentmli]

172287 bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0xc00055e8f0, license="MIT", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 19, 0), prog_flags=0, prog_name="probe_kpm_link", prog_ifindex=0, expected_attach_type=0x2a /* BPF_??? /, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 144) = 16
172287 bpf(BPF_LINK_CREATE, {link_create={prog_fd=16, target_fd=0, attach_type=0x2a / BPF_??? */, flags=0}, ...}, 48) = -1 EOPNOTSUPP (Operation not supported)

for the record, enable kernel config CONFIG_FPROBE resolved this operation not supported issue

[brb]

Closing in favor of #99. Thanks!