Skip to content

libpf: panic: runtime error: slice bounds out of range [456:0] #33

New issue
New issue
Open
Open
libpf: panic: runtime error: slice bounds out of range [456:0]#33
@gipi

Description

@gipi
gipi
opened on Jan 16, 2025

In our environment parca-agent is crashing with the following stacktrace

Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: time="2025-01-16T13:49:38Z" level=info msg="eBPF tracer loaded"
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: panic: runtime error: slice bounds out of range [456:0]
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: goroutine 102 [running]:
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/readatbuf.(*Reader).ReadAt(0xc004da1770, {0xc0054f0600, 0x1f0, 0x1f0}, 0xc004eb7220?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/readatbuf/readatbuf.go:119 +0x277
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).ReadAt(0xc00552ebc8, {0xc0054f0600, 0x1f0, 0x1f0}, 0x0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:640 +0x85
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).Data(0xc00552ebc8, 0xc006bce980?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:681 +0xa8
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).DataReader(0x201e720?, 0xc000013008?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:687 +0x17
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.newFile({0x2902840, 0xc004da1770}, {0x2900560, 0xc00007eed8}, 0x0, 0x0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:275 +0x70d
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.Open({0xc005512180?, 0x10?})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:171 +0x71
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/process.(*systemProcess).OpenELF(0xc005596870, {0xc005277a40, 0x6c})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/process/process.go:253 +0x17c
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*File).OpenDebugLink(0x1d?, {0xc004e80500, 0x32}, {0x2902880, 0xc004dab5c0})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:607 +0x17b
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.(*elfExtractor).extractDebugDeltas(0xc004eb7640)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaextraction.go:97 +0x35
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.ExtractELF(0xc004dab5c0, 0xc000012fa8)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaextraction.go:152 +0x247
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.(*ELFStackDeltaProvider).GetIntervalStructuresForFile(0xc00406a010, 0xc003f5fe60?, 0xc004dab5c0, 0x0?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaprovider.go:40 +0x27
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager/execinfomanager.(*ExecutableInfoManager).AddOrIncRef(0xc000714690, 0x4f1b206233f1e03b, 0xc004dab5c0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/execinfomanager/manager.go:188 +0x152
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).handleNewMapping(0xc00406c840, {0x29321c8, 0xc005596870}, 0xc004eb7998, 0xc004dab5c0?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:219 +0x59
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).processNewExecMapping(0xc00406c840, {0x29321c8, 0xc005596870}, 0xc003c23fc8)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:344 +0x456
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).synchronizeMappings(0xc00406c840, {0x29321c8, 0xc005596870}, {0xc003c21008, 0xf6, 0x13f})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:462 +0x579
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).SynchronizeProcess(0xc00406c840, {0x29321c8, 0xc005596870})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:590 +0x2a7
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/tracer.(*Tracer).processPIDEvents(0xc003f4cdd0, {0x2924aa8, 0xc000909a00})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/tracer/events.go:53 +0x1c7
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: created by github.com/elastic/otel-profiling-agent/tracer.(*Tracer).StartPIDEventProcessor in goroutine 1
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/tracer/events.go:42 +0x7a
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: time="2025-01-16T13:49:38Z" level=error msg="======================= unexpected error ======================="

Adding logging I have found out that the problem is that len(data) is equal to zero. Probably is a problem affecting also upstream.

Let me know if you have any hint regarding debugging this, for now I added locally a break if len(data) == 0 to be able to profile.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions