CI: tests frequently hang in weird places

[lmb]

We've been having regular hangs in CI, where one of the vimto based tests (usually 6.1 but not exclusively so) hangs for 10 minutes and then times out with a weird goroutine dump:

panic: test timed out after 35s
	running tests:
		TestGetSetJumpOp (35s)

goroutine 63 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2484 +0x394
created by time.goFunc
	/usr/local/go/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000a61c0, {0x71e894?, 0xc00010db30?}, 0x731870)
	/usr/local/go/src/testing/testing.go:1859 +0x431
testing.runTests.func1(0xc0000a61c0)
	/usr/local/go/src/testing/testing.go:2279 +0x37
testing.tRunner(0xc0000a61c0, 0xc00010dc70)
	/usr/local/go/src/testing/testing.go:1792 +0xf4
testing.runTests(0xc00008e0a8, {0x964680, 0x10, 0x10}, {0x96b100?, 0x7?, 0x969ec0?})
	/usr/local/go/src/testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc0000905a0)
	/usr/local/go/src/testing/testing.go:2142 +0x64a
main.main()
	_testmain.go:83 +0x9b

goroutine 35 [runnable]:
syscall.Syscall(0x1, 0x1, 0xc0000e4440, 0x2b)
	/usr/local/go/src/syscall/syscall_linux.go:73 +0x25
syscall.write(0xc0000a4060?, {0xc0000e4440?, 0x7f4b5b22fa10?, 0x42f6e0?})
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:964 +0x3b
syscall.Write(...)
	/usr/local/go/src/syscall/syscall_unix.go:211
internal/poll.ignoringEINTRIO(...)
	/usr/local/go/src/internal/poll/fd_unix.go:745
internal/poll.(*FD).Write(0xc0000a4060, {0xc0000e4440, 0x2b, 0x40})
	/usr/local/go/src/internal/poll/fd_unix.go:381 +0x394
os.(*File).write(...)
	/usr/local/go/src/os/file_posix.go:46
os.(*File).Write(0xc000084030, {0xc0000e4440?, 0x2b, 0xc00008aea0?})
	/usr/local/go/src/os/file.go:195 +0x4e
fmt.Fprintf({0x788d00, 0xc000084030}, {0xc00008adb0, 0xe}, {0xc00008aea0, 0x1, 0x1})
	/usr/local/go/src/fmt/print.go:225 +0x97
testing.(*chattyPrinter).Updatef(0xc000018120?, {0xc000018120?, 0xc0000100c0?}, {0x71d8c8?, 0x418e9b?}, {0xc00008aea0, 0x1, 0x1})
	/usr/local/go/src/testing/testing.go:600 +0x187
testing.(*T).Run(0xc000140540, {0xc0000100c0?, 0xc00008af40?}, 0xc0000100d0)
	/usr/local/go/src/testing/testing.go:1842 +0x331
github.com/cilium/ebpf/asm.TestGetSetJumpOp.func1(...)
	/home/lorenz/dev/ebpf/asm/opcode_test.go:12
github.com/cilium/ebpf/asm.TestGetSetJumpOp(0xc000140540)
	/home/lorenz/dev/ebpf/asm/opcode_test.go:49 +0x3e9
testing.tRunner(0xc000140540, 0x731870)
	/usr/local/go/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1851 +0x413
FAIL	github.com/cilium/ebpf/asm	38.394s

(on 6.1.129 from ci-kernels)

The weird part is that the runtime seems to be stuck in a write to stdout. I've managed to reproduce this in a VM which mimics GH Actions and on my local machine, running different host kernels and qemu versions.

It does seem like this is an issue with the guest kernel, since I can't trigger the hang on older versions like v5.4. I've started bisecting the problem but it's been very slow going due to compilation of old kernels on new toolchains being pretty broken.

I did get one money shot from one of the crashes of asm:

panic: test timed out after 1m0s
	running tests:
		TestGetSetJumpOp (1m0s)


goroutine 51 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2484 +0x394
created by time.goFunc
	/usr/local/go/src/time/sleep.go:215 +0x2d


goroutine 1 [chan receive]:
testing.(*T).Run(0xc000003a40, {0x71e894?, 0xc0000efb30?}, 0x731870)
	/usr/local/go/src/testing/testing.go:1859 +0x431
testing.runTests.func1(0xc000003a40)
	/usr/local/go/src/testing/testing.go:2279 +0x37
testing.tRunner(0xc000003a40, 0xc0000efc70)
	/usr/local/go/src/testing/testing.go:1792 +0xf4
testing.runTests(0xc00000e0c0, {0x964680, 0x10, 0x10}, {0x96b100?, 0x7?, 0x969ec0?})
	/usr/local/go/src/testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc00008a500)
	/usr/local/go/src/testing/testing.go:2142 +0x64a
main.main()
	_testmain.go:83 +0x9b


goroutine 23 [runnable]:
syscall.Syscall(0x1, 0x1, 0xc00001c540, 0x2b)
	/usr/local/go/src/syscall/syscall_linux.go:73 +0x25
syscall.write(0xc000012180?, {0xc00001c540?, 0xc000086c00?, 0x45ea9e?})
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:964 +0x3b
syscall.Write(...)
	/usr/local/go/src/syscall/syscall_unix.go:211
internal/poll.ignoringEINTRIO(...)
	/usr/local/go/src/internal/poll/fd_unix.go:745
internal/poll.(*FD).Write(0x0, {0x487faa, 0x95fd00, 0x40})
	/usr/local/go/src/internal/poll/fd_unix.go:381 +0x394
os.(*File).write(...)
	/usr/local/go/src/os/file_posix.go:46
os.(*File).Write(0xc000086d48, {0xc000086d48?, 0x419d5b, 0x96b100?})
	/usr/local/go/src/os/file.go:195 +0x4e
runtime: g 23: unexpected return pc for os.(*File).Write called from 0x4
stack: frame={sp:0xc000086cb0, fp:0xc000086d10} stack=[0xc000086000,0xc000087000)
0x000000c000086bb0:  0x0000000000012100  0x0000000000000000 
0x000000c000086bc0:  0x000000000047dbd2 <runtime.systemstack+0x0000000000000032>  0x000000c000086c08 
0x000000c000086bd0:  0x000000000041c34b <runtime.(*mcache).refill+0x00000000000001ab>  0x000000000098b4a0 
0x000000c000086be0:  0x0000000000000000  0x0000000000000190 
0x000000c000086bf0:  0x000000c000086c10  0x00007f920c3cb1e8 
0x000000c000086c00:  0x0000000000540010 <go/ast.(*CommentGroup).Pos+0x0000000000000010>  0x000000c000086c40 
0x000000c000086c10:  0x000000000041887e <runtime.(*mcache).nextFree+0x000000000000009e>  0x00007f920c3cb1e8 
0x000000c000086c20:  0x0000000000000018  0x000000000000dbd2 
0x000000c000086c30:  0x00007f920c3cb1e8  0x0000000000000010 
0x000000c000086c40:  0x000000c000086ca0  0x00000000004192c5 <runtime.mallocgcSmallScanNoHeader+0x00000000000002e5> 
0x000000c000086c50:  0x00007f9252ffc5c0  0x000000000041ca10 <runtime.(*mcentral).cacheSpan+0x0000000000000170> 
0x000000c000086c60:  0x000000c000006ca8  0x00007f920c3cb328 
0x000000c000086c70:  0x080101000000000a  0x00007f920c3cb1e8 
0x000000c000086c80:  0x00007f9252ffc5c0  0x0000000000000060 
0x000000c000086c90:  0x000000000096b100  0x000000c000188000 
0x000000c000086ca0:  0x000000c000086cd0  0x0000000000475c19 <runtime.mallocgc+0x0000000000000099> 
0x000000c000086cb0: <0x0000000000000058  0x000000000070ede0 
0x000000c000086cc0:  0x0000000000000030  0x000000000096b100 
0x000000c000086cd0:  0x000000c000086cf8  0x0000000000419ca5 <runtime.newobject+0x0000000000000025> 
0x000000c000086ce0:  0x0000000000000058  0x000000000070ede0 
0x000000c000086cf0:  0x000000c000003d01  0x000000c000086d60 
0x000000c000086d00:  0x0000000000443e29 <runtime.acquireSudog+0x0000000000000229>  0x000000c000086d30 
0x000000c000086d10: >0x0000000000475159 <sync/atomic.StorePointer+0x0000000000000039>  0x0000000000000030 
0x000000c000086d20:  0x00000000006fc7a0  0x000000c000184000 
0x000000c000086d30:  0x000000c000086e00  0x000000000048a977 <internal/sync.(*HashTrieMap[go.shape.interface {},go.shape.interface {}]).Swap+0x00000000000003d7> 
0x000000c000086d40:  0x000000c000086d60  0x000000000047810e <runtime.gopark+0x00000000000000ce> 
0x000000c000086d50:  0x0000000000419214 <runtime.mallocgcSmallScanNoHeader+0x0000000000000234>  0x000000c000034508 
0x000000c000086d60:  0x000000c000086dd8  0x0000000000411b45 <runtime.chanrecv+0x0000000000000445> 
0x000000c000086d70:  0x0301010000788d00  0x030000000c3cb3c8 
0x000000c000086d80:  0x00007f920c3cb3c8  0x00007f9252ffc5c0 
0x000000c000086d90:  0x0000000000000018  0x000000000096b100 
0x000000c000086da0:  0x0000000000000000  0x000000c000086dd8 
0x000000c000086db0:  0x000000c000188000  0x000000c00012aa70 
0x000000c000086dc0:  0x000000c000003dc0  0x000000c000086e00 
0x000000c000086dd0:  0x000000000044d61f <runtime.newproc+0x000000000000003f>  0x000000c000086e00 
0x000000c000086de0:  0x00000000004116d2 <runtime.chanrecv1+0x0000000000000012>  0x000000c00012aa10 
0x000000c000086df0:  0x000000c000086e57  0x0000000000532501 <testing.(*T).Run+0x0000000000000361> 
0x000000c000086e00:  0x000000c000086ed8  0x00000000005325d1 <testing.(*T).Run+0x0000000000000431> 
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1851 +0x413
FAIL	github.com/cilium/ebpf/asm	64.861s

This tells me we are looking at memory corruption. Since the crash is in the asm package test I'm fairly certain that the bug isn't in our (unsafe) code, since asm doesn't import anything meaningfully unsafe.

Since we also see a lot of crashes where the stack indicates a write to stdout, I'm guessing that the bug is somewhere in the virtio-console code because that is what stdout for the test program is mapped to.

[lmb]

Here is the script I've been using to reproduce:

#!/bin/bash

set -m

pushd "$(dirname "$(realpath "$0")")"

total=0
timeout_seconds=30
overall_timeout=300

# kernel=":4.9.337"
kernel="${KERNEL:-:5.15.178-debug}"
# kernel=":6.1.129-debug"

test() {
  #taskset -c 0-10 vimto -kernel "$kernel" -- go test -p 50 -short -count 1 -timeout 32s -skip "(ParseModuleSplitSpec|TestRoundtripVMlinux|TestKprobeCreatePMU|TestLinkCgroup|TestRawLink|TestIterator|TestLoadWrongPin|RawTracepoint|TestHaveProgramHelper|TestSkLookup)" "$@" ./...
  taskset -c 0-10 vimto -kernel "$kernel" -- go test -p 50 -short -count 1 -timeout 32s -skip "(ModuleSplitSpec|TestKprobeCreatePMU|TestSkLookup)" "$@" ./...
}

echo Warming up
test || exit 125
echo Warm up finished

start_time=$(date +%s)
while true; do
  current_time=$(date +%s)
  elapsed=$((current_time - start_time))
  if [ $elapsed -ge $overall_timeout ]; then
    echo "No hang detected after running for $elapsed seconds. Considering this commit good."
    exit 0  # Mark as good for git bisect
  fi

  test_start_time=$(date +%s)
  if ! test -json > log.txt; then
    echo "Test failed, total run time was $elapsed seconds"
    pgrep -a qemu-system

    pkg="$(tail -n 1 log.txt | jq -r .Package)"
    jq -j --arg pkg "$pkg" 'select(.Package == $pkg and .Action == "output") | .Output' < log.txt | grep -vE '^(===|---)'

    # if [[ "$pkg" != "github.com/cilium/ebpf/asm" && "$pkg" != "github.com/cilium/ebpf/internal/sysenc" ]]; then
    #   echo "Package $pkg is not a target package for bisection. Aborting bisect."
    #   exit 128  # Abort the entire git bisect run
    # fi

    echo "Hang detected in target package: $pkg"
    exit 1
  else
    test_end_time=$(date +%s)
    test_duration=$((test_end_time - test_start_time))
    echo "Test completed successfully in $test_duration seconds. Retrying..."
  fi
done

Combined with a bisection script:

#!/usr/bin/env bash

# git cherry-pick --empty=drop 2d7ca4a84b58 # objtool: Don't fail on missing symbol table
# git cherry-pick --empty=drop 0baced0e0938 # kbuild: Unify options for BTF generation for vmlinux and modules
# git cherry-pick --empty=drop b775fbf532dc # kbuild: Add skip_encoding_btf_enum64 option to pahole

# def2084e5f56 (HEAD) kbuild: Add skip_encoding_btf_enum64 option to pahole
# 6f866c6ca10e kbuild: Unify options for BTF generation for vmlinux and modules
# b5836e1f66b5 kbuild: Quote OBJCOPY var to avoid a pahole call break the build
# b769bebc6f6b kbuild: Skip module BTF generation for out-of-tree external modules
# 9644f91a0790 kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it
# 8202fc958a1d kbuild: Quote OBJCOPY var to avoid a pahole call break the build
# 476bbd47af6e bpf: Generate BTF_KIND_FLOAT when linking vmlinux
# 770f44ca9d68 objtool: Don't fail on missing symbol table
# 2c85ebc57b3e (tag: v5.10) Linux 5.10

export KBUILD_BUILD_TIMESTAMP="Thu  6 Jul 01:00:00 UTC 2023"
export KBUILD_BUILD_HOST="ci-kernels-builder"
export TARGETPLATFORM=linux/amd64
export WERROR=0
export CFLAGS="-Wno-error=deprecated-declarations -Wno-error=redundant-decls"
export HOSTCFLAGS="$CFLAGS"
make clean
rm -rf tools/testing/selftests/bpf/bpf_testmod
../ci-kernels/configure-vmlinux.sh || exit 128
./scripts/config --disable CONFIG_SYSTEM_TRUSTED_KEYRING
./scripts/config --disable CONFIG_SECURITY_SELINUX
../ci-kernels/build-vmlinux.sh || exit 125
KERNEL="$(pwd)/arch/x86/boot/bzImage" exec ../ebpf/hang.sh

Here is what I've managed to narrow down to so far:

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
git bisect good 219d54332a09e8d8741c1e1982f5eae56099de85
# bad: [2c85ebc57b3e1817b6ce1a6b703928e113a90442] Linux 5.10
git bisect bad 2c85ebc57b3e1817b6ce1a6b703928e113a90442
# bad: [bce159d734091fe31340976081577333f52a85e4] Merge tag 'for-5.8/drivers-2020-06-01' of git://git.kernel.dk/linux-block
git bisect bad bce159d734091fe31340976081577333f52a85e4
# skip: [8fdd4019bcb2d824c5ab45c6fc340293cfed843f] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma
git bisect skip 8fdd4019bcb2d824c5ab45c6fc340293cfed843f
# bad: [b3a9e3b9622ae10064826dccb4f7a52bd88c7407] Linux 5.8-rc1
git bisect bad b3a9e3b9622ae10064826dccb4f7a52bd88c7407
# skip: [47acac8cae28b36668bf89400c56b7fdebca3e75] Merge tag 'hwmon-for-v5.7' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging
git bisect skip 47acac8cae28b36668bf89400c56b7fdebca3e75

$ uname -r
6.13.8-200.fc41.x86_64
lorenz:~/dev/linux  ((a9e90d9931f3...)|BISECTING)
$ qemu-system-x86_64 --version
QEMU emulator version 9.1.3 (qemu-9.1.3-2.fc41)
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

[lmb]

I've found that kernels below v5.4 need the following to compile with somewhat recent toolchains:

git cherry-pick 770f44ca9d68 a9a3ed1eff3601b63aea4fb462d8b3b92c7c1e7e c87a37ebd40b889178664c2c09cc187334146292

Plus an older version of pahole.

$ pahole --version
v1.20

Further testing suggests that my previous bisection results were bogus. All of a sudden I got crashes with v5.4 which I thought was good. Back to the drawing board...

[lmb]

panic: test timed out after 10m0s
	running tests:
		TestGetSetJumpOp (10m0s)

goroutine 52 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2484 +0x394
created by time.goFunc
	/opt/hostedtoolcache/go/1.24.2/x64/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000003a40, {0x71e974?, 0xc0000f3b30?}, 0x7319a0)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1859 +0x431
testing.runTests.func1(0xc000003a40)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2279 +0x37
testing.tRunner(0xc000003a40, 0xc0000f3c70)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1792 +0xf4
testing.runTests(0xc00000e0c0, {0x967640, 0x10, 0x10}, {0x96e0c0?, 0x7?, 0x96ce80?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc00007a460)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2142 +0x64a
main.main()
	_testmain.go:83 +0x9b

goroutine 23 [syscall]:
syscall.Syscall(0x1, 0x1, 0xc0000188c0, 0x24)
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/syscall_linux.go:73 +0x25
syscall.write(0xc000012180?, {0xc0000188c0?, 0x66?, 0x2?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/zsyscall_linux_amd64.go:964 +0x3b
syscall.Write(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/syscall_unix.go:211
internal/poll.ignoringEINTRIO(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/internal/poll/fd_unix.go:745
internal/poll.(*FD).Write(0xc000012180, {0xc0000188c0, 0x24, 0x40})
	/opt/hostedtoolcache/go/1.24.2/x64/src/internal/poll/fd_unix.go:381 +0x394
os.(*File).write(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/os/file_posix.go:46
os.(*File).Write(0xc00004a038, {0xc0000188c0?, 0x24, 0xc00008cda0?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/os/file.go:195 +0x4e
fmt.Fprintf({0x788e80, 0xc00004a038}, {0xc00008cc08, 0x11}, {0xc00008cda0, 0x3, 0x3})
	/opt/hostedtoolcache/go/1.24.2/x64/src/fmt/print.go:225 +0x97
testing.(*chattyPrinter).Updatef(0xc0000121a0?, {0x71e974?, 0xc00008cca0?}, {0x71e9a4?, 0xc00008cce8?}, {0xc00008cda0, 0x3, 0x3})
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:600 +0x187
testing.(*common).flushToParent(0xc000125dc0, {0x71e974, 0x10}, {0x71e9a4--- PASS: TestGetSetJumpOp (0.00s)
, 0x10}, {0xc00008cda0, 0x3, 0x3})
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:871 +0x3a5
testing.(*T).report(0xc000125dc0)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2200 +0x1db
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1775 +0x447
testing.tRunner(0xc000125dc0, 0x7319a0)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1798 +0x12d
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1851 +0x413
FAIL	github.com/cilium/ebpf/asm	602.634s

=== FAIL: btf  (0.00s)
panic: test timed out after 10m0s
	running tests:
		TestType (9m59s)
		TestType/*btf.TypeTag (9m59s)

goroutine 247 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2484 +0x394
created by time.goFunc
	/opt/hostedtoolcache/go/1.24.2/x64/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc000003a40, {0x7e6006?, 0xc000109b30?}, 0x8046e8)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1859 +0x431
testing.runTests.func1(0xc000003a40)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2279 +0x37
testing.tRunner(0xc000003a40, 0xc000109c70)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1792 +0xf4
testing.runTests(0xc00000e0d8, {0xab7d80, 0x46, 0x46}, {0xabeb40?, 0x7?, 0xabd900?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2277 +0x4b4
testing.(*M).Run(0xc0000985a0)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:2142 +0x64a
main.main()
	_testmain.go:215 +0x9b

goroutine 202 [chan receive, 9 minutes]:
testing.(*T).Run(0xc00701afc0, {0xc006e12e70?, 0xc006e02eb0?}, 0xc00368e510)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1859 +0x431
github.com/cilium/ebpf/btf.TestType(0xc00701afc0)
	/home/runner/work/ebpf/ebpf/btf/types_test.go:192 +0x188
testing.tRunner(0xc00701afc0, 0x8046e8)
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1851 +0x413

goroutine 246 [syscall, 9 minutes]:
syscall.Syscall(0x1, 0x1, 0xc000032080, 0x31)
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/syscall_linux.go:73 +0x25
syscall.write(0xc000020180?, {0xc000032080?, 0x7fd9dece5108?, 0x60?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/zsyscall_linux_amd64.go:964 +0x3b
syscall.Write(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/syscall/syscall_unix.go:211
internal/poll.ignoringEINTRIO(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/internal/poll/fd_unix.go:745
internal/poll.(*FD).Write(0xc000020180, {    types_test.go:193: TypeTag[type=Void value=]
0xc000032080, 0x31, 0x80})
	/opt/hostedtoolcache/go/1.24.2/x64/src/internal/poll/fd_unix.go:381 +0x394
os.(*File).write(...)
	/opt/hostedtoolcache/go/1.24.2/x64/src/os/file_posix.go:46
os.(*File).Write(0xc006e09b30, {0x43608e?, 0x47e1b2, 0xc006e09b88?})
	/opt/hostedtoolcache/go/1.24.2/x64/src/os/file.go:195 +0x4e
fmt.Fprintf({0x5, 0x7fd997d76fc8}, {0x5, 0x47e1b2}, {0xc006e09c08, 0x76af80, 0x0})
	/opt/hostedtoolcache/go/1.24.2/x64/src/fmt/print.go:226 +0xb0
runtime: g 246: unexpected return pc for fmt.Fprintf called from 0xc006e09b38
stack: frame={sp:0xc006e09b08, fp:0xc006e09b68} stack=[0xc006e09000,0xc006e0a000)
0x000000c006e09a08:  0x000000c00430e010  0x000000c00430e000 
0x000000c006e09a18:  0x000000c00430e018  0x000000c00430e008 
0x000000c006e09a28:  0x0000000000000097  0x0000000000669ee0 <github.com/google/go-cmp/cmp.(*state).compareSlice.func1+0x0000000000000000> 
0x000000c006e09a38:  0x000000c00174c1e0  0x0000000000765840 
0x000000c006e09a48:  0x000000c004104000  0x0000000000000097 
0x000000c006e09a58:  0x0000000000765840  0x000000c004104018 
0x000000c006e09a68:  0x0000000000000097  0x0000000000669e40 <github.com/google/go-cmp/cmp.(*state).compareSlice.func2+0x0000000000000000> 
0x000000c006e09a78:  0x000000c006fbe000  0x000000c006e09a30 
0x000000c006e09a88:  0x000000c00430e000  0x0000000000000001 
0x000000c006e09a98:  0x0000000000000001  0x000000c00430e010 
0x000000c006e09aa8:  0x0000000000000001  0x0000000000000001 
0x000000c006e09ab8:  0x000000c00430e010  0x000000c00430e000 
0x000000c006e09ac8:  0x000000000075d820  0x0000000000765840 
0x000000c006e09ad8:  0x000000c004104000  0x0000000000765840 
0x000000c006e09ae8:  0x000000c004104018  0x000000c006e09cb0 
0x000000c006e09af8:  0x00000000006671bb <github.com/google/go-cmp/cmp.(*state).compareAny+0x0000000000000bbb>  0x000000c006fbe000 
0x000000c006e09b08: <0x0000000000879478  0x0000000000765840 
0x000000c006e09b18:  0x0000000000765840  0x000000c004104000 
0x000000c006e09b28:  0x000000000078a0c0  0x0000000000765840 
0x000000c006e09b38:  0x000000c004104018  0x00007fd9dece55c0 
0x000000c006e09b48:  0x0000000000000040  0x0000000000abeb40 
0x000000c006e09b58:  0x0000000000879478  0x000000c006e09b90 
0x000000c006e09b68: >0x0000000000476139 <runtime.mallocgc+0x0000000000000099>  0x0000000000000097 
0x000000c006e09b78:  0x0000000000000097  0x000000c006e09c00 
0x000000c006e09b88:  0x00000000006662ac <github.com/google/go-cmp/cmp.(*state).processOption+0x000000000000030c>  0x000000c006e09bb8 
0x000000c006e09b98:  0x0000000000419d05 <runtime.newobject+0x0000000000000025>  0x0000000000000040 
0x000000c006e09ba8:  0x00000000007c4d60  0x000000000078a001 
0x000000c006e09bb8:  0x000000c006e09cb0  0x0000000000660000 <github.com/cilium/ebpf/asm.(*BuiltinFunc).String+0x0000000000000040> 
0x000000c006e09bc8:  0x000000c006e09b00  0x0000000000666710 <github.com/google/go-cmp/cmp.(*state).compareAny+0x0000000000000110> 
0x000000c006e09bd8:  0x000000c006e09c90  0x0000000000000000 
0x000000c006e09be8:  0x0000000000000000  0x0000000000abeb40 
0x000000c006e09bf8:  0x0000000000765840  0x0000000000000097 
0x000000c006e09c08:  0x0000000000000097  0x0000000000000097 
0x000000c006e09c18:  0x0000000000000097  0x0000000000765840 
0x000000c006e09c28:  0x000000c004104000  0x0000000000765840 
0x000000c006e09c38:  0x0000000000765840  0x000000c004104018 
0x000000c006e09c48:  0x0000000000765840  0x000000c004104000 
0x000000c006e09c58:  0x0000000000765840  0x0000000000765840 
created by testing.(*T).Run in goroutine 202
	/opt/hostedtoolcache/go/1.24.2/x64/src/testing/testing.go:1851 +0x413

goroutine 257 [runnable]:
github.com/google/go-cmp/cmp.(*state).callTTBFunc.gowrap1()
	/home/runner/go/pkg/mod/github.com/google/go-cmp@v0.6.0/cmp/compare.go:357
runtime.goexit({})
	/opt/hostedtoolcache/go/1.24.2/x64/src/runtime/asm_amd64.s:1700 +0x1
created by github.com/google/go-cmp/cmp.(*state).callTTBFunc in goroutine 246
	/home/runner/go/pkg/mod/github.com/google/go-cmp@v0.6.0/cmp/compare.go:357 +0x175
FAIL	github.com/cilium/ebpf/btf	602.324s