orchestrator segfault handling two consecutive streamplace mp4 segments

[iameli]

Still investigating but I wanted to get the ticket posted, this from me is knocking over Os everywhere, I have a consistent repro case.

Things that work around this:

• The problem seems to be exclusive to MP4s -- switching to .ts segments causes the error not to reproduce.
• The problem seems to be exclusive to GPU transcoding, running without -nvidia seems to work around the problem as well.

Gateway configuration: livepeer --gateway --httpIngest --network offchain --orchAddr 127.0.0.1:9878 --httpAddr 127.0.0.1:9999
OT configuration: livepeer --orchestrator --transcoder --network offchain --serviceAddr 127.0.0.1:9878 --nvidia=0 -v=6

Deadly segment: godzilla.tar.gz

That script has two segments, both MP4 files. It does this:

#!/bin/bash

set -euo pipefail

function post() {
  FILE="$1"
  SEQ="$2"
  curl -v \
    --data-binary "@$FILE" \
    -X POST \
    -H "accept: multipart/mixed" \
    -o multipart \
    -H 'Content-Duration: 1000' \
    -H 'Content-Resolution: 1920x1080' \
    -H 'Livepeer-Transcode-Configuration: {"manifestID":"did:plc:dkh4rwafdcda4ko7lewe43ml-gm9vtiym","profiles":[{"name":"720p","height":720,"bitrate":3000000,"profile":"h264constrainedhigh"},{"name":"360p","height":360,"bitrate":1000000,"fps":60,"fpsDen":2,"profile":"h264constrainedhigh"},{"name":"240p","height":240,"bitrate":500000,"fps":60,"fpsDen":2,"profile":"h264constrainedhigh"},{"name":"160p","height":160,"bitrate":250000,"fps":60,"fpsDen":2,"profile":"h264baseline"}]}' \
    http://127.0.0.1:9999/live/did:plc:dkh4rwafdcda4ko7lewe43ml-gm9vtiym/$SEQ.mp4
}

post 2025-04-08T19-09-28-222Z.mp4 1
post 2025-04-08T19-09-29-042Z.mp4 2

Then the O crashes:

SIGSEGV: segmentation violation
PC=0x2fdae9a m=7 sigcode=1 addr=0x1
signal arrived during cgo execution

goroutine 195 gp=0xc000585500 m=7 mp=0xc000100708 [syscall]:
runtime.cgocall(0x2e45140, 0xc000767800)
	/__t/go/1.23.2/x64/src/runtime/cgocall.go:167 +0x4b fp=0xc0007677d8 sp=0xc0007677a0 pc=0x1b4288b
github.com/livepeer/lpms/ffmpeg._Cfunc_lpms_transcode(0xc0006786e0, 0xc00061ab40, 0xc0002e6400, 0x4, 0xc000aa0320)
	_cgo_gotypes.go:479 +0x4b fp=0xc000767800 sp=0xc0007677d8 pc=0x24c4ecb
github.com/livepeer/lpms/ffmpeg.(*Transcoder).Transcode.func7(0xc0006786e0, 0xc00061ab40, 0xc0002e6400, {0xc00061ab40?, 0x4, 0x4?}, 0xc000aa0320)
	/github/home/go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20250118014304-79e6dcf08057/ffmpeg/ffmpeg.go:1013 +0x90 fp=0xc000767840 sp=0xc000767800 pc=0x24ca4d0
github.com/livepeer/lpms/ffmpeg.(*Transcoder).Transcode(0xc000806660, 0xc000767da0, {0xc000752488, 0x4, 0x4})
	/github/home/go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20250118014304-79e6dcf08057/ffmpeg/ffmpeg.go:1013 +0xea5 fp=0xc000767cd8 sp=0xc000767840 pc=0x24ca085
github.com/livepeer/go-livepeer/core.(*NvidiaTranscoder).Transcode(0xc000a94000, {0xb25318, 0xc000654db0}, 0xc00041c240)
	/__w/go-livepeer/go-livepeer/core/transcoder.go:151 +0x25c fp=0xc000767e98 sp=0xc000767cd8 pc=0x2b9717c
github.com/livepeer/go-livepeer/core.(*transcoderSession).loop(0xc000ab8e00, {0xb25318, 0xc000806630})
	/__w/go-livepeer/go-livepeer/core/lb.go:194 +0x17b fp=0xc000767fb0 sp=0xc000767e98 pc=0x2b8863b
github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).createSession.func2()
	/__w/go-livepeer/go-livepeer/core/lb.go:127 +0x28 fp=0xc000767fe0 sp=0xc000767fb0 pc=0x2b881e8
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000767fe8 sp=0xc000767fe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).createSession in goroutine 194
	/__w/go-livepeer/go-livepeer/core/lb.go:126 +0x725

goroutine 1 gp=0xc0000061c0 m=nil [select, 8 minutes]:
runtime.gopark(0xc00090fda0?, 0x2?, 0x1?, 0xe1?, 0xc00090f994?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0007ed268 sp=0xc0007ed248 pc=0x1b490ae
runtime.selectgo(0xc0007edda0, 0xc00090f990, 0x1?, 0x0, 0x1?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc0007ed390 sp=0xc0007ed268 pc=0x1b248c5
main.main()
	/__w/go-livepeer/go-livepeer/cmd/livepeer/livepeer.go:103 +0xbde fp=0xc0007edf50 sp=0xc0007ed390 pc=0x2e42b3e
runtime.main()
	/__t/go/1.23.2/x64/src/runtime/proc.go:272 +0x28b fp=0xc0007edfe0 sp=0xc0007edf50 pc=0x1b125eb
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0007edfe8 sp=0xc0007edfe0 pc=0x1b51be1

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle), 2 minutes]:
runtime.gopark(0x8e4776f104ca?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a4fa8 sp=0xc0000a4f88 pc=0x1b490ae
runtime.goparkunlock(...)
	/__t/go/1.23.2/x64/src/runtime/proc.go:430
runtime.forcegchelper()
	/__t/go/1.23.2/x64/src/runtime/proc.go:337 +0xb3 fp=0xc0000a4fe0 sp=0xc0000a4fa8 pc=0x1b12933
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x1b51be1
created by runtime.init.7 in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a5780 sp=0xc0000a5760 pc=0x1b490ae
runtime.goparkunlock(...)
	/__t/go/1.23.2/x64/src/runtime/proc.go:430
runtime.bgsweep(0xc0000d2000)
	/__t/go/1.23.2/x64/src/runtime/mgcsweep.go:317 +0xdf fp=0xc0000a57c8 sp=0xc0000a5780 pc=0x1afa2bf
runtime.gcenable.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:203 +0x25 fp=0xc0000a57e0 sp=0xc0000a57c8 pc=0x1aee985
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a57e8 sp=0xc0000a57e0 pc=0x1b51be1
created by runtime.gcenable in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x3ba09ecc?, 0x3b9aca00?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a5f78 sp=0xc0000a5f58 pc=0x1b490ae
runtime.goparkunlock(...)
	/__t/go/1.23.2/x64/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x4a08460)
	/__t/go/1.23.2/x64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc0000a5fa8 sp=0xc0000a5f78 pc=0x1af7c89
runtime.bgscavenge(0xc0000d2000)
	/__t/go/1.23.2/x64/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc0000a5fc8 sp=0xc0000a5fa8 pc=0x1af8219
runtime.gcenable.gowrap2()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:204 +0x25 fp=0xc0000a5fe0 sp=0xc0000a5fc8 pc=0x1aee925
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a5fe8 sp=0xc0000a5fe0 pc=0x1b51be1
created by runtime.gcenable in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait, 6 minutes]:
runtime.gopark(0x0?, 0xc000012480?, 0x20?, 0x60?, 0x1000000010?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000942e20 sp=0xc000942e00 pc=0x1b490ae
runtime.runfinq()
	/__t/go/1.23.2/x64/src/runtime/mfinal.go:193 +0x107 fp=0xc000942fe0 sp=0xc000942e20 pc=0x1aeda07
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000942fe8 sp=0xc000942fe0 pc=0x1b51be1
created by runtime.createfing in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc000007dc0 m=nil [select, 8 minutes]:
runtime.gopark(0xc0000a6760?, 0x2?, 0x0?, 0x0?, 0xc0000a671c?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000b6da8 sp=0xc0000b6d88 pc=0x1b490ae
runtime.selectgo(0xc0000b6f60, 0xc0000a6718, 0x0?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc0000b6ed0 sp=0xc0000b6da8 pc=0x1b248c5
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc0000ec6f0)
	/github/home/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:71 +0x105 fp=0xc0000b6fc8 sp=0xc0000b6ed0 pc=0x29b59e5
github.com/ipfs/go-log/writer.NewMirrorWriter.gowrap1()
	/github/home/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:36 +0x25 fp=0xc0000b6fe0 sp=0xc0000b6fc8 pc=0x29b5825
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000b6fe8 sp=0xc0000b6fe0 pc=0x1b51be1
created by github.com/ipfs/go-log/writer.NewMirrorWriter in goroutine 1
	/github/home/go/pkg/mod/github.com/ipfs/go-log@v1.0.5/writer/writer.go:36 +0xbf

goroutine 7 gp=0xc0001efc00 m=nil [GC worker (idle), 8 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a6f38 sp=0xc0000a6f18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a6fc8 sp=0xc0000a6f38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a6fe0 sp=0xc0000a6fc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a6fe8 sp=0xc0000a6fe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 18 gp=0xc000104700 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515efb3?, 0x0?, 0x0?, 0x0?, 0xc000194000?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a6738 sp=0xc0000a6718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a67c8 sp=0xc0000a6738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a67e0 sp=0xc0000a67c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a67e8 sp=0xc0000a67e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 19 gp=0xc0001048c0 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb516322f?, 0x1?, 0xa4?, 0x1a?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a0738 sp=0xc0000a0718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a07c8 sp=0xc0000a0738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a07e0 sp=0xc0000a07c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a07e8 sp=0xc0000a07e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 34 gp=0xc000504000 m=nil [GC worker (idle)]:
runtime.gopark(0x4b804e0?, 0x1?, 0x43?, 0x3c?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 8 gp=0xc0001efdc0 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515e397?, 0x3?, 0x94?, 0x3e?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a7738 sp=0xc0000a7718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a77c8 sp=0xc0000a7738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a77e0 sp=0xc0000a77c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a77e8 sp=0xc0000a77e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 20 gp=0xc000104a80 m=nil [GC worker (idle), 8 minutes]:
runtime.gopark(0x8dd1202b036d?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a0f38 sp=0xc0000a0f18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a0fc8 sp=0xc0000a0f38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a0fe0 sp=0xc0000a0fc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a0fe8 sp=0xc0000a0fe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 35 gp=0xc0005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb5162a5f?, 0x3?, 0x74?, 0x36?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 9 gp=0xc0004b4000 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515e3bf?, 0x3?, 0x20?, 0x3?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a7f38 sp=0xc0000a7f18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a7fc8 sp=0xc0000a7f38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a7fe0 sp=0xc0000a7fc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 21 gp=0xc000104c40 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515f19d?, 0x1?, 0xbc?, 0xdd?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a1738 sp=0xc0000a1718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a17c8 sp=0xc0000a1738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a17e0 sp=0xc0000a17c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a17e8 sp=0xc0000a17e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 36 gp=0xc000504380 m=nil [GC worker (idle)]:
runtime.gopark(0x4b804e0?, 0x3?, 0x73?, 0x1d?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 37 gp=0xc000504540 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4b94c6c11d?, 0xc00011c080?, 0x1a?, 0xa?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050bf38 sp=0xc00050bf18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050bfc8 sp=0xc00050bf38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc00050bfe0 sp=0xc00050bfc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 22 gp=0xc000104e00 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515e383?, 0x3?, 0xe5?, 0xa3?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a1f38 sp=0xc0000a1f18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a1fc8 sp=0xc0000a1f38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a1fe0 sp=0xc0000a1fc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a1fe8 sp=0xc0000a1fe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 10 gp=0xc0004b41c0 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515ef13?, 0x1?, 0xf9?, 0xae?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 38 gp=0xc000504700 m=nil [GC worker (idle), 8 minutes]:
runtime.gopark(0x8dd1202b10a1?, 0x3?, 0x1?, 0xe8?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050c738 sp=0xc00050c718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050c7c8 sp=0xc00050c738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc00050c7e0 sp=0xc00050c7c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050c7e8 sp=0xc00050c7e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 23 gp=0xc000104fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb516123b?, 0x3?, 0x96?, 0x32?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a2738 sp=0xc0000a2718 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a27c8 sp=0xc0000a2738 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a27e0 sp=0xc0000a27c8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a27e8 sp=0xc0000a27e0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 24 gp=0xc000105180 m=nil [GC worker (idle)]:
runtime.gopark(0x8e4bb515f19d?, 0x1?, 0x49?, 0x2e?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000a2f38 sp=0xc0000a2f18 pc=0x1b490ae
runtime.gcBgMarkWorker(0xc0000e2230)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000a2fc8 sp=0xc0000a2f38 pc=0x1af0ca9
runtime.gcBgMarkStartWorkers.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x25 fp=0xc0000a2fe0 sp=0xc0000a2fc8 pc=0x1af0b85
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a2fe8 sp=0xc0000a2fe0 pc=0x1b51be1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1279 +0x105

goroutine 11 gp=0xc000604540 m=nil [select, 8 minutes, locked to thread]:
runtime.gopark(0xc00050dfa8?, 0x2?, 0x40?, 0x45?, 0xc00050df94?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050de30 sp=0xc00050de10 pc=0x1b490ae
runtime.selectgo(0xc00050dfa8, 0xc00050df90, 0x0?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc00050df58 sp=0xc00050de30 pc=0x1b248c5
runtime.ensureSigM.func1()
	/__t/go/1.23.2/x64/src/runtime/signal_unix.go:1060 +0x19f fp=0xc00050dfe0 sp=0xc00050df58 pc=0x1b4039f
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x1b51be1
created by runtime.ensureSigM in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/signal_unix.go:1043 +0xc8

goroutine 25 gp=0xc000105340 m=12 mp=0xc000580708 [syscall, 8 minutes]:
runtime.notetsleepg(0x4b7ed80, 0xffffffffffffffff)
	/__t/go/1.23.2/x64/src/runtime/lock_futex.go:246 +0x29 fp=0xc0003067a0 sp=0xc000306778 pc=0x1ae2fe9
os/signal.signal_recv()
	/__t/go/1.23.2/x64/src/runtime/sigqueue.go:152 +0x29 fp=0xc0003067c0 sp=0xc0003067a0 pc=0x1b4b3a9
os/signal.loop()
	/__t/go/1.23.2/x64/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0003067e0 sp=0xc0003067c0 pc=0x1c02f53
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003067e8 sp=0xc0003067e0 pc=0x1b51be1
created by os/signal.Notify.func1.1 in goroutine 1
	/__t/go/1.23.2/x64/src/os/signal/signal.go:151 +0x1f

goroutine 12 gp=0xc000504fc0 m=nil [select]:
runtime.gopark(0xc0000b9f80?, 0x2?, 0x60?, 0x0?, 0xc0000b9f74?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000b9e10 sp=0xc0000b9df0 pc=0x1b490ae
runtime.selectgo(0xc0000b9f80, 0xc0000b9f70, 0x0?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc0000b9f38 sp=0xc0000b9e10 pc=0x1b248c5
github.com/golang/glog.(*fileSink).flushDaemon(0x4a082f8)
	/github/home/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:349 +0xb4 fp=0xc0000b9fc8 sp=0xc0000b9f38 pc=0x1c765f4
github.com/golang/glog.init.1.gowrap1()
	/github/home/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:164 +0x25 fp=0xc0000b9fe0 sp=0xc0000b9fc8 pc=0x1c75685
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000b9fe8 sp=0xc0000b9fe0 pc=0x1b51be1
created by github.com/golang/glog.init.1 in goroutine 1
	/github/home/go/pkg/mod/github.com/golang/glog@v1.2.2/glog_file.go:164 +0x126

goroutine 13 gp=0xc000505180 m=nil [select]:
runtime.gopark(0xc000941f78?, 0x3?, 0x9?, 0x3?, 0xc000941f72?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000941e10 sp=0xc000941df0 pc=0x1b490ae
runtime.selectgo(0xc000941f78, 0xc000941f6c, 0xc0006d2900?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc000941f38 sp=0xc000941e10 pc=0x1b248c5
go.opencensus.io/stats/view.(*worker).start(0xc0006d2900)
	/github/home/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000941fc8 sp=0xc000941f38 pc=0x20eb19f
go.opencensus.io/stats/view.init.0.gowrap1()
	/github/home/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000941fe0 sp=0xc000941fc8 pc=0x20ea505
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000941fe8 sp=0xc000941fe0 pc=0x1b51be1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/github/home/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 14 gp=0xc000505500 m=nil [chan receive]:
runtime.gopark(0xc000507760?, 0x1cabfa5?, 0xa0?, 0xe5?, 0xb49c00?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000507718 sp=0xc0005076f8 pc=0x1b490ae
runtime.chanrecv(0xc0000e2b60, 0x0, 0x1)
	/__t/go/1.23.2/x64/src/runtime/chan.go:639 +0x41c fp=0xc000507790 sp=0xc000507718 pc=0x1add5fc
runtime.chanrecv1(0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/chan.go:489 +0x12 fp=0xc0005077b8 sp=0xc000507790 pc=0x1add1b2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1735 +0x2f fp=0xc0005077e0 sp=0xc0005077b8 pc=0x1af198f
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x1b51be1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
	/__t/go/1.23.2/x64/src/runtime/mgc.go:1730 +0x96

goroutine 161 gp=0xc000505c00 m=nil [IO wait]:
runtime.gopark(0xc00074b000?, 0x1000?, 0x0?, 0x10?, 0xb?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0006ed558 sp=0xc0006ed538 pc=0x1b490ae
runtime.netpollblock(0x1b6c298?, 0x1adaa06?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0006ed590 sp=0xc0006ed558 pc=0x1b0af17
internal/poll.runtime_pollWait(0x7d726f0950b8, 0x72)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc0006ed5b0 sp=0xc0006ed590 pc=0x1b483a5
internal/poll.(*pollDesc).wait(0xc000c20480?, 0xc0001ac000?, 0x0)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006ed5d8 sp=0xc0006ed5b0 pc=0x1bcc567
internal/poll.(*pollDesc).waitRead(...)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c20480, {0xc0001ac000, 0x800, 0x800})
	/__t/go/1.23.2/x64/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc0006ed670 sp=0xc0006ed5d8 pc=0x1bcd85a
net.(*netFD).Read(0xc000c20480, {0xc0001ac000?, 0x1ae3e1e?, 0x7d722464dc78?})
	/__t/go/1.23.2/x64/src/net/fd_posix.go:55 +0x25 fp=0xc0006ed6b8 sp=0xc0006ed670 pc=0x1ccdac5
net.(*conn).Read(0xc0007ae158, {0xc0001ac000?, 0x1f75506?, 0xc000a887c0?})
	/__t/go/1.23.2/x64/src/net/net.go:189 +0x45 fp=0xc0006ed700 sp=0xc0006ed6b8 pc=0x1cdf885
net.(*TCPConn).Read(0x10000e000?, {0xc0001ac000?, 0x18?, 0xc000654390?})
	<autogenerated>:1 +0x25 fp=0xc0006ed730 sp=0xc0006ed700 pc=0x1cf51c5
crypto/tls.(*atLeastReader).Read(0xc00084c000, {0xc0001ac000?, 0x0?, 0xc00084c000?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:809 +0x3b fp=0xc0006ed778 sp=0xc0006ed730 pc=0x1e91f1b
bytes.(*Buffer).ReadFrom(0xc000b8b0b8, {0xb0ea00, 0xc00084c000})
	/__t/go/1.23.2/x64/src/bytes/buffer.go:211 +0x98 fp=0xc0006ed7d0 sp=0xc0006ed778 pc=0x1c037b8
crypto/tls.(*Conn).readFromUntil(0xc000b8ae08, {0xb0d3c0, 0xc0007ae158}, 0xc000777868?)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:831 +0xde fp=0xc0006ed808 sp=0xc0006ed7d0 pc=0x1e920fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000b8ae08, 0x0)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:629 +0x3cf fp=0xc0006eda80 sp=0xc0006ed808 pc=0x1e8f20f
crypto/tls.(*Conn).readRecord(...)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc000b8ae08, {0xc00074a000, 0x1000, 0x4a08260?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:1385 +0x150 fp=0xc0006edaf0 sp=0xc0006eda80 pc=0x1e95a70
net/http.(*connReader).Read(0xc0007344b0, {0xc00074a000, 0x1000, 0x1000})
	/__t/go/1.23.2/x64/src/net/http/server.go:798 +0x14b fp=0xc0006edb40 sp=0xc0006edaf0 pc=0x1f76bcb
bufio.(*Reader).fill(0xc000550300)
	/__t/go/1.23.2/x64/src/bufio/bufio.go:110 +0x103 fp=0xc0006edb78 sp=0xc0006edb40 pc=0x1c08963
bufio.(*Reader).Peek(0xc000550300, 0x4)
	/__t/go/1.23.2/x64/src/bufio/bufio.go:148 +0x53 fp=0xc0006edb98 sp=0xc0006edb78 pc=0x1c08a93
net/http.(*conn).serve(0xc000995680, {0xb25318, 0xc000675bc0})
	/__t/go/1.23.2/x64/src/net/http/server.go:2127 +0x738 fp=0xc0006edfb8 sp=0xc0006edb98 pc=0x1f7bf18
net/http.(*Server).Serve.gowrap3()
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x28 fp=0xc0006edfe0 sp=0xc0006edfb8 pc=0x1f81228
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0006edfe8 sp=0xc0006edfe0 pc=0x1b51be1
created by net/http.(*Server).Serve in goroutine 30
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x485

goroutine 16 gp=0xc000505a40 m=nil [select, 4 minutes]:
runtime.gopark(0xc0000b8f90?, 0x2?, 0x10?, 0x9b?, 0xc0000b8f84?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000b8e28 sp=0xc0000b8e08 pc=0x1b490ae
runtime.selectgo(0xc0000b8f90, 0xc0000b8f80, 0x0?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc0000b8f50 sp=0xc0000b8e28 pc=0x1b248c5
github.com/patrickmn/go-cache.(*janitor).Run(0xc000068280, 0xc0007b9b00)
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x7d fp=0xc0000b8fc0 sp=0xc0000b8f50 pc=0x2d8de7d
github.com/patrickmn/go-cache.runJanitor.gowrap1()
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0x25 fp=0xc0000b8fe0 sp=0xc0000b8fc0 pc=0x2d8e025
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000b8fe8 sp=0xc0000b8fe0 pc=0x1b51be1
created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xdb

goroutine 66 gp=0xc000105a40 m=nil [select, 8 minutes]:
runtime.gopark(0xc00006db58?, 0x7?, 0xb8?, 0x85?, 0xc00006cc4a?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00006c780 sp=0xc00006c760 pc=0x1b490ae
runtime.selectgo(0xc00006db58, 0xc00006cc3c, 0xc0009306e0?, 0x0, 0xc00048f448?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc00006c8a8 sp=0xc00006c780 pc=0x1b248c5
github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer({_, _}, {0xc000712810, 0xc000712820, 0xc000712830, 0xc000712840, 0xc000712850, 0xc000712880, 0xc000712860, 0xc000712a50, ...})
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1713 +0x9b6f fp=0xc00006dcb0 sp=0xc00006c8a8 pc=0x2e3a16f
main.main.func1()
	/__w/go-livepeer/go-livepeer/cmd/livepeer/livepeer.go:97 +0x58 fp=0xc00006dfe0 sp=0xc00006dcb0 pc=0x2e42eb8
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x1b51be1
created by main.main in goroutine 1
	/__w/go-livepeer/go-livepeer/cmd/livepeer/livepeer.go:96 +0xb1d

goroutine 82 gp=0xc000584e00 m=nil [select, 8 minutes]:
runtime.gopark(0xc00050cf88?, 0x2?, 0x8?, 0x81?, 0xc00050cf84?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00050ce28 sp=0xc00050ce08 pc=0x1b490ae
runtime.selectgo(0xc00050cf88, 0xc00050cf80, 0xc0007129b0?, 0x0, 0xc0007b5bb8?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc00050cf50 sp=0xc00050ce28 pc=0x1b248c5
database/sql.(*DB).connectionOpener(0xc0002209c0, {0xb25350, 0xc0005a80a0})
	/__t/go/1.23.2/x64/src/database/sql/sql.go:1253 +0x87 fp=0xc00050cfb8 sp=0xc00050cf50 pc=0x21b98e7
database/sql.OpenDB.gowrap1()
	/__t/go/1.23.2/x64/src/database/sql/sql.go:833 +0x28 fp=0xc00050cfe0 sp=0xc00050cfb8 pc=0x21b7c08
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050cfe8 sp=0xc00050cfe0 pc=0x1b51be1
created by database/sql.OpenDB in goroutine 66
	/__t/go/1.23.2/x64/src/database/sql/sql.go:833 +0x130

goroutine 26 gp=0xc000684a80 m=nil [select, 8 minutes]:
runtime.gopark(0xc000304790?, 0x2?, 0x60?, 0x0?, 0xc000304784?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000304628 sp=0xc000304608 pc=0x1b490ae
runtime.selectgo(0xc000304790, 0xc000304780, 0xa600000083?, 0x0, 0x10400000063?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc000304750 sp=0xc000304628 pc=0x1b248c5
github.com/patrickmn/go-cache.(*janitor).Run(0xc0009306a0, 0xc00033b000)
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x7d fp=0xc0003047c0 sp=0xc000304750 pc=0x2d8de7d
github.com/patrickmn/go-cache.runJanitor.gowrap1()
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0x25 fp=0xc0003047e0 sp=0xc0003047c0 pc=0x2d8e025
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003047e8 sp=0xc0003047e0 pc=0x1b51be1
created by github.com/patrickmn/go-cache.runJanitor in goroutine 66
	/github/home/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xdb

goroutine 27 gp=0xc000684c40 m=nil [IO wait, 8 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0000bbb98 sp=0xc0000bbb78 pc=0x1b490ae
runtime.netpollblock(0x10?, 0x1adaa06?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000bbbd0 sp=0xc0000bbb98 pc=0x1b0af17
internal/poll.runtime_pollWait(0x7d726f0956e8, 0x72)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc0000bbbf0 sp=0xc0000bbbd0 pc=0x1b483a5
internal/poll.(*pollDesc).wait(0xc0006ca900?, 0x10?, 0x0)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000bbc18 sp=0xc0000bbbf0 pc=0x1bcc567
internal/poll.(*pollDesc).waitRead(...)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006ca900)
	/__t/go/1.23.2/x64/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0000bbcc0 sp=0xc0000bbc18 pc=0x1bd1935
net.(*netFD).accept(0xc0006ca900)
	/__t/go/1.23.2/x64/src/net/fd_unix.go:172 +0x29 fp=0xc0000bbd78 sp=0xc0000bbcc0 pc=0x1ccfa89
net.(*TCPListener).accept(0xc000a88b40)
	/__t/go/1.23.2/x64/src/net/tcpsock_posix.go:159 +0x1e fp=0xc0000bbdc8 sp=0xc0000bbd78 pc=0x1ce97fe
net.(*TCPListener).Accept(0xc000a88b40)
	/__t/go/1.23.2/x64/src/net/tcpsock.go:372 +0x30 fp=0xc0000bbdf8 sp=0xc0000bbdc8 pc=0x1ce86b0
net/http.(*onceCloseListener).Accept(0xb252e0?)
	<autogenerated>:1 +0x24 fp=0xc0000bbe10 sp=0xc0000bbdf8 pc=0x1fa9d84
net/http.(*Server).Serve(0xc0008e6ff0, {0xb1fde0, 0xc000a88b40})
	/__t/go/1.23.2/x64/src/net/http/server.go:3330 +0x30c fp=0xc0000bbf40 sp=0xc0000bbe10 pc=0x1f80e2c
net/http.(*Server).ListenAndServe(0xc0008e6ff0)
	/__t/go/1.23.2/x64/src/net/http/server.go:3259 +0x71 fp=0xc0000bbf70 sp=0xc0000bbf40 pc=0x1f80af1
github.com/livepeer/go-livepeer/server.(*LivepeerServer).StartCliWebserver(0x71900000446?, 0xc0008e6ff0)
	/__w/go-livepeer/go-livepeer/server/webserver.go:21 +0xdf fp=0xc0000bbfb8 sp=0xc0000bbf70 pc=0x2e15d3f
github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer.func13()
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1636 +0x25 fp=0xc0000bbfe0 sp=0xc0000bbfb8 pc=0x2e3c385
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000bbfe8 sp=0xc0000bbfe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer in goroutine 66
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1635 +0x93bd

goroutine 28 gp=0xc000684e00 m=nil [select, 8 minutes]:
runtime.gopark(0xc000993f30?, 0x2?, 0x80?, 0x1?, 0xc000993ef4?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00093dd78 sp=0xc00093dd58 pc=0x1b490ae
runtime.selectgo(0xc00093df30, 0xc000993ef0, 0xc0009800e0?, 0x0, 0x8332a8?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc00093dea0 sp=0xc00093dd78 pc=0x1b248c5
github.com/livepeer/go-livepeer/server.(*LivepeerServer).StartMediaServer(0xc00080aea0, {0xb25350, 0xc000624a50}, {0xc0008c6ea0, 0x5})
	/__w/go-livepeer/go-livepeer/server/mediaserver.go:247 +0x5ea fp=0xc00093df90 sp=0xc00093dea0 pc=0x2dec36a
github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer.func14()
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1641 +0x37 fp=0xc00093dfe0 sp=0xc00093df90 pc=0x2e3c337
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00093dfe8 sp=0xc00093dfe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer in goroutine 66
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1640 +0x9466

goroutine 30 gp=0xc000685180 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0007657c0 sp=0xc0007657a0 pc=0x1b490ae
runtime.netpollblock(0xc000765810?, 0x1adaa06?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0007657f8 sp=0xc0007657c0 pc=0x1b0af17
internal/poll.runtime_pollWait(0x7d726f0955e0, 0x72)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc000765818 sp=0xc0007657f8 pc=0x1b483a5
internal/poll.(*pollDesc).wait(0xc0008d2b00?, 0x900000036?, 0x0)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000765840 sp=0xc000765818 pc=0x1bcc567
internal/poll.(*pollDesc).waitRead(...)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0008d2b00)
	/__t/go/1.23.2/x64/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0007658e8 sp=0xc000765840 pc=0x1bd1935
net.(*netFD).accept(0xc0008d2b00)
	/__t/go/1.23.2/x64/src/net/fd_unix.go:172 +0x29 fp=0xc0007659a0 sp=0xc0007658e8 pc=0x1ccfa89
net.(*TCPListener).accept(0xc000c1df00)
	/__t/go/1.23.2/x64/src/net/tcpsock_posix.go:159 +0x1e fp=0xc0007659f0 sp=0xc0007659a0 pc=0x1ce97fe
net.(*TCPListener).Accept(0xc000c1df00)
	/__t/go/1.23.2/x64/src/net/tcpsock.go:372 +0x30 fp=0xc000765a20 sp=0xc0007659f0 pc=0x1ce86b0
crypto/tls.(*listener).Accept(0xc0007a4948)
	/__t/go/1.23.2/x64/src/crypto/tls/tls.go:67 +0x27 fp=0xc000765a60 sp=0xc000765a20 pc=0x1ed32a7
net/http.(*onceCloseListener).Accept(0xc00054d560?)
	<autogenerated>:1 +0x24 fp=0xc000765a78 sp=0xc000765a60 pc=0x1fa9d84
net/http.(*Server).Serve(0xc0008e73b0, {0xb20050, 0xc0007a4948})
	/__t/go/1.23.2/x64/src/net/http/server.go:3330 +0x30c fp=0xc000765ba8 sp=0xc000765a78 pc=0x1f80e2c
net/http.(*Server).ServeTLS(0xc0008e73b0, {0xb1fde0, 0xc000c1df00}, {0xc000a97a70, 0x26}, {0xc000a97aa0, 0x25})
	/__t/go/1.23.2/x64/src/net/http/server.go:3401 +0x446 fp=0xc000765d70 sp=0xc000765ba8 pc=0x1f81766
net/http.(*Server).ListenAndServeTLS(0xc0008e73b0, {0xc000a97a70, 0x26}, {0xc000a97aa0, 0x25})
	/__t/go/1.23.2/x64/src/net/http/server.go:3558 +0x115 fp=0xc000765df0 sp=0xc000765d70 pc=0x1f821d5
github.com/livepeer/go-livepeer/server.StartTranscodeServer({0xb46b60, 0xc000624aa0}, {0xc0008c6ea0, 0x5}, 0xc00085e0e0, {0xc00005a440, 0x1d}, 0x0, 0x1, 0xc00048f448)
	/__w/go-livepeer/go-livepeer/server/rpc.go:250 +0x50d fp=0xc000765f70 sp=0xc000765df0 pc=0x2e062cd
github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer.func15.1()
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1653 +0x75 fp=0xc000765fe0 sp=0xc000765f70 pc=0x2e3c255
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000765fe8 sp=0xc000765fe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/cmd/livepeer/starter.StartLivepeer.func15 in goroutine 29
	/__w/go-livepeer/go-livepeer/cmd/livepeer/starter/starter.go:1652 +0x10a

goroutine 67 gp=0xc000685a40 m=nil [select, 2 minutes]:
runtime.gopark(0xc000940fb0?, 0x2?, 0x0?, 0x0?, 0xc000940f94?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000940e38 sp=0xc000940e18 pc=0x1b490ae
runtime.selectgo(0xc000940fb0, 0xc000940f90, 0x0?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc000940f60 sp=0xc000940e38 pc=0x1b248c5
github.com/livepeer/go-livepeer/trickle.(*Server).Start.func2()
	/__w/go-livepeer/go-livepeer/trickle/trickle_server.go:133 +0x7a fp=0xc000940fe0 sp=0xc000940f60 pc=0x2b6cd3a
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000940fe8 sp=0xc000940fe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/trickle.(*Server).Start in goroutine 30
	/__w/go-livepeer/go-livepeer/trickle/trickle_server.go:131 +0xf0

goroutine 159 gp=0xc000604fc0 m=nil [IO wait]:
runtime.gopark(0xc0004d4000?, 0x1000?, 0x0?, 0x10?, 0xb?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0006e9558 sp=0xc0006e9538 pc=0x1b490ae
runtime.netpollblock(0x1b6c298?, 0x1adaa06?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc0006e9590 sp=0xc0006e9558 pc=0x1b0af17
internal/poll.runtime_pollWait(0x7d726f0952c8, 0x72)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc0006e95b0 sp=0xc0006e9590 pc=0x1b483a5
internal/poll.(*pollDesc).wait(0xc000c20280?, 0xc000802000?, 0x0)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006e95d8 sp=0xc0006e95b0 pc=0x1bcc567
internal/poll.(*pollDesc).waitRead(...)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c20280, {0xc000802000, 0x800, 0x800})
	/__t/go/1.23.2/x64/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc0006e9670 sp=0xc0006e95d8 pc=0x1bcd85a
net.(*netFD).Read(0xc000c20280, {0xc000802000?, 0xc000320dc0?, 0xc000320dc0?})
	/__t/go/1.23.2/x64/src/net/fd_posix.go:55 +0x25 fp=0xc0006e96b8 sp=0xc0006e9670 pc=0x1ccdac5
net.(*conn).Read(0xc0007ae010, {0xc000802000?, 0x1f755c6?, 0xc00033a080?})
	/__t/go/1.23.2/x64/src/net/net.go:189 +0x45 fp=0xc0006e9700 sp=0xc0006e96b8 pc=0x1cdf885
net.(*TCPConn).Read(0x10?, {0xc000802000?, 0x18?, 0x1ae8e65?})
	<autogenerated>:1 +0x25 fp=0xc0006e9730 sp=0xc0006e9700 pc=0x1cf51c5
crypto/tls.(*atLeastReader).Read(0xc000af2b40, {0xc000802000?, 0x0?, 0xc000af2b40?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:809 +0x3b fp=0xc0006e9778 sp=0xc0006e9730 pc=0x1e91f1b
bytes.(*Buffer).ReadFrom(0xc000b8a9b8, {0xb0ea00, 0xc000af2b40})
	/__t/go/1.23.2/x64/src/bytes/buffer.go:211 +0x98 fp=0xc0006e97d0 sp=0xc0006e9778 pc=0x1c037b8
crypto/tls.(*Conn).readFromUntil(0xc000b8a708, {0xb0d3c0, 0xc0007ae010}, 0xc0005ef868?)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:831 +0xde fp=0xc0006e9808 sp=0xc0006e97d0 pc=0x1e920fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000b8a708, 0x0)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:629 +0x3cf fp=0xc0006e9a80 sp=0xc0006e9808 pc=0x1e8f20f
crypto/tls.(*Conn).readRecord(...)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc000b8a708, {0xc0005d3000, 0x1000, 0x4a08260?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:1385 +0x150 fp=0xc0006e9af0 sp=0xc0006e9a80 pc=0x1e95a70
net/http.(*connReader).Read(0xc00073c030, {0xc0005d3000, 0x1000, 0x1000})
	/__t/go/1.23.2/x64/src/net/http/server.go:798 +0x14b fp=0xc0006e9b40 sp=0xc0006e9af0 pc=0x1f76bcb
bufio.(*Reader).fill(0xc000b0e360)
	/__t/go/1.23.2/x64/src/bufio/bufio.go:110 +0x103 fp=0xc0006e9b78 sp=0xc0006e9b40 pc=0x1c08963
bufio.(*Reader).Peek(0xc000b0e360, 0x4)
	/__t/go/1.23.2/x64/src/bufio/bufio.go:148 +0x53 fp=0xc0006e9b98 sp=0xc0006e9b78 pc=0x1c08a93
net/http.(*conn).serve(0xc000995560, {0xb25318, 0xc000675bc0})
	/__t/go/1.23.2/x64/src/net/http/server.go:2127 +0x738 fp=0xc0006e9fb8 sp=0xc0006e9b98 pc=0x1f7bf18
net/http.(*Server).Serve.gowrap3()
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x28 fp=0xc0006e9fe0 sp=0xc0006e9fb8 pc=0x1f81228
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0006e9fe8 sp=0xc0006e9fe0 pc=0x1b51be1
created by net/http.(*Server).Serve in goroutine 30
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x485

goroutine 220 gp=0xc000585a40 m=nil [chan receive]:
runtime.gopark(0x4?, 0x0?, 0x5e?, 0x1?, 0x2?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0005f55d8 sp=0xc0005f55b8 pc=0x1b490ae
runtime.chanrecv(0xc0008582a0, 0xc0005f56e0, 0x1)
	/__t/go/1.23.2/x64/src/runtime/chan.go:639 +0x41c fp=0xc0005f5650 sp=0xc0005f55d8 pc=0x1add5fc
runtime.chanrecv1(0xb25318?, 0xc000654db0?)
	/__t/go/1.23.2/x64/src/runtime/chan.go:489 +0x12 fp=0xc0005f5678 sp=0xc0005f5650 pc=0x1add1b2
github.com/livepeer/go-livepeer/core.(*LivepeerNode).sendToTranscodeLoop(0xc00048f448, {0xb25318, 0xc000654db0}, 0xc00041c240, 0xc000ab8400)
	/__w/go-livepeer/go-livepeer/core/orchestrator.go:584 +0x234 fp=0xc0005f5710 sp=0xc0005f5678 pc=0x2b8eb34
github.com/livepeer/go-livepeer/core.(*orchestrator).TranscodeSeg(0x387e7c0?, {0xb25318?, 0xc000654db0?}, 0x759367?, 0x0?)
	/__w/go-livepeer/go-livepeer/core/orchestrator.go:93 +0x25 fp=0xc0005f5748 sp=0xc0005f5710 pc=0x2b8ac05
github.com/livepeer/go-livepeer/server.(*lphttp).ServeSegment(0xc00033b100, {0xb1ff60, 0xc0007ae2a8}, 0xc000327540)
	/__w/go-livepeer/go-livepeer/server/segment_rpc.go:165 +0x101d fp=0xc0005f5bf8 sp=0xc0005f5748 pc=0x2e0b9fd
github.com/livepeer/go-livepeer/server.(*lphttp).ServeSegment-fm({0xb1ff60?, 0xc0007ae2a8?}, 0xc0002a7420?)
	<autogenerated>:1 +0x36 fp=0xc0005f5c28 sp=0xc0005f5bf8 pc=0x2e287d6
net/http.HandlerFunc.ServeHTTP(0xc00085e0e0?, {0xb1ff60?, 0xc0007ae2a8?}, 0x1f0e97f?)
	/__t/go/1.23.2/x64/src/net/http/server.go:2220 +0x29 fp=0xc0005f5c50 sp=0xc0005f5c28 pc=0x1f7d409
net/http.(*ServeMux).ServeHTTP(0xc000654cf0?, {0xb1ff60, 0xc0007ae2a8}, 0xc000327540)
	/__t/go/1.23.2/x64/src/net/http/server.go:2747 +0x1ca fp=0xc0005f5ca0 sp=0xc0005f5c50 pc=0x1f7f30a
github.com/livepeer/go-livepeer/server.(*lphttp).ServeHTTP(0xc00033b100, {0xb1ff60, 0xc0007ae2a8}, 0xc000327540)
	/__w/go-livepeer/go-livepeer/server/rpc.go:198 +0xa5 fp=0xc0005f5cd0 sp=0xc0005f5ca0 pc=0x2e05ca5
net/http.serverHandler.ServeHTTP({0xc0005b1ee0?}, {0xb1ff60?, 0xc0007ae2a8?}, 0x0?)
	/__t/go/1.23.2/x64/src/net/http/server.go:3210 +0x8e fp=0xc0005f5d00 sp=0xc0005f5cd0 pc=0x1f9d9ae
net/http.initALPNRequest.ServeHTTP({{0xb25318?, 0xc00067b980?}, 0xc000378708?, {0xc0008e73b0?}}, {0xb1ff60, 0xc0007ae2a8}, 0xc000327540)
	/__t/go/1.23.2/x64/src/net/http/server.go:3819 +0x231 fp=0xc0005f5eb0 sp=0xc0005f5d00 pc=0x1f83ab1
net/http.(*initALPNRequest).ServeHTTP(0xc00085e380?, {0xb1ff60?, 0xc0007ae2a8?}, 0xc00032a168?)
	<autogenerated>:1 +0x4f fp=0xc0005f5ef8 sp=0xc0005f5eb0 pc=0x1faa00f
net/http.Handler.ServeHTTP-fm({0xb1ff60?, 0xc0007ae2a8?}, 0x1?)
	<autogenerated>:1 +0x3b fp=0xc0005f5f28 sp=0xc0005f5ef8 pc=0x1fab13b
net/http.(*http2serverConn).runHandler(0x1f81228?, 0xc0003077d0?, 0x1f48a05?, 0xc0004b5a40?)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:6249 +0xf5 fp=0xc0005f5fb0 sp=0xc0005f5f28 pc=0x1f4fc15
net/http.(*http2serverConn).scheduleHandler.gowrap1()
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:6183 +0x2c fp=0xc0005f5fe0 sp=0xc0005f5fb0 pc=0x1f4f7cc
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005f5fe8 sp=0xc0005f5fe0 pc=0x1b51be1
created by net/http.(*http2serverConn).scheduleHandler in goroutine 146
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:6183 +0x21d

goroutine 149 gp=0xc0004b4c40 m=nil [IO wait]:
runtime.gopark(0x8e7e44204ff29000?, 0x47e19368da88de0d?, 0x0?, 0x90?, 0xb?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc000943820 sp=0xc000943800 pc=0x1b490ae
runtime.netpollblock(0x1b6c298?, 0x1adaa06?, 0x0?)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc000943858 sp=0xc000943820 pc=0x1b0af17
internal/poll.runtime_pollWait(0x7d726f0953d0, 0x72)
	/__t/go/1.23.2/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc000943878 sp=0xc000943858 pc=0x1b483a5
internal/poll.(*pollDesc).wait(0xc0008d2380?, 0xc0008ee000?, 0x0)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0009438a0 sp=0xc000943878 pc=0x1bcc567
internal/poll.(*pollDesc).waitRead(...)
	/__t/go/1.23.2/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0008d2380, {0xc0008ee000, 0xa000, 0xa000})
	/__t/go/1.23.2/x64/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc000943938 sp=0xc0009438a0 pc=0x1bcd85a
net.(*netFD).Read(0xc0008d2380, {0xc0008ee000?, 0x9ffb?, 0xc0008ee000?})
	/__t/go/1.23.2/x64/src/net/fd_posix.go:55 +0x25 fp=0xc000943980 sp=0xc000943938 pc=0x1ccdac5
net.(*conn).Read(0xc0000a9320, {0xc0008ee000?, 0xc0008effd4?, 0x3894c00?})
	/__t/go/1.23.2/x64/src/net/net.go:189 +0x45 fp=0xc0009439c8 sp=0xc000943980 pc=0x1cdf885
net.(*TCPConn).Read(0x8ee005?, {0xc0008ee000?, 0x18?, 0xc0008ee000?})
	<autogenerated>:1 +0x25 fp=0xc0009439f8 sp=0xc0009439c8 pc=0x1cf51c5
crypto/tls.(*atLeastReader).Read(0xc000ce4078, {0xc0008ee000?, 0xc0004b4c40?, 0xc000ce4078?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:809 +0x3b fp=0xc000943a40 sp=0xc0009439f8 pc=0x1e91f1b
bytes.(*Buffer).ReadFrom(0xc0003789b8, {0xb0ea00, 0xc000ce4078})
	/__t/go/1.23.2/x64/src/bytes/buffer.go:211 +0x98 fp=0xc000943a98 sp=0xc000943a40 pc=0x1c037b8
crypto/tls.(*Conn).readFromUntil(0xc000378708, {0xb0d3c0, 0xc0000a9320}, 0xc000943b30?)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:831 +0xde fp=0xc000943ad0 sp=0xc000943a98 pc=0x1e920fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000378708, 0x0)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:629 +0x3cf fp=0xc000943d48 sp=0xc000943ad0 pc=0x1e8f20f
crypto/tls.(*Conn).readRecord(...)
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc000378708, {0xc00085e200, 0x9, 0xc000943f50?})
	/__t/go/1.23.2/x64/src/crypto/tls/conn.go:1385 +0x150 fp=0xc000943db8 sp=0xc000943d48 pc=0x1e95a70
io.ReadAtLeast({0x7d72244b9548, 0xc000378708}, {0xc00085e200, 0x9, 0x9}, 0x9)
	/__t/go/1.23.2/x64/src/io/io.go:335 +0x90 fp=0xc000943e00 sp=0xc000943db8 pc=0x1bc3070
io.ReadFull(...)
	/__t/go/1.23.2/x64/src/io/io.go:354
net/http.http2readFrameHeader({0xc00085e200, 0x9, 0xc000113180?}, {0x7d72244b9548?, 0xc000378708?})
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:1642 +0x65 fp=0xc000943e50 sp=0xc000943e00 pc=0x1f3bb25
net/http.(*http2Framer).ReadFrame(0xc00085e1c0)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:1909 +0x85 fp=0xc000943ef8 sp=0xc000943e50 pc=0x1f3c265
net/http.(*http2serverConn).readFrames(0xc000604e00)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4691 +0x98 fp=0xc000943fc8 sp=0xc000943ef8 pc=0x1f47398
net/http.(*http2serverConn).serve.gowrap7()
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4804 +0x25 fp=0xc000943fe0 sp=0xc000943fc8 pc=0x1f48a05
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000943fe8 sp=0xc000943fe0 pc=0x1b51be1
created by net/http.(*http2serverConn).serve in goroutine 146
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4804 +0x52a

goroutine 194 gp=0xc0004b4e00 m=nil [select]:
runtime.gopark(0xc000761b70?, 0x2?, 0x60?, 0x82?, 0xc000761b5c?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc0007619e8 sp=0xc0007619c8 pc=0x1b490ae
runtime.selectgo(0xc000761b70, 0xc000761b58, 0x1?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc000761b10 sp=0xc0007619e8 pc=0x1b248c5
github.com/livepeer/go-livepeer/core.(*transcoderSession).Transcode(0xc000ab8e00, {0xb25318, 0xc000654db0}, 0xc00041c240)
	/__w/go-livepeer/go-livepeer/core/lb.go:222 +0x1d9 fp=0xc000761be0 sp=0xc000761b10 pc=0x2b88ab9
github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).Transcode(0xc000624690, {0xb25318, 0xc000654db0}, 0xc00041c240)
	/__w/go-livepeer/go-livepeer/core/lb.go:80 +0x225 fp=0xc000761c78 sp=0xc000761be0 pc=0x2b87785
github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSeg(0xc00048f448, {0xb25318, 0xc000654db0}, {{0xb339b0?, 0xc000ab86c0?}, {0xb339b0?, 0xc000ab86c0?}}, 0xc000ab8400, 0xc00041c240)
	/__w/go-livepeer/go-livepeer/core/orchestrator.go:658 +0x92f fp=0xc000761f10 sp=0xc000761c78 pc=0x2b8f50f
github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSegmentLoop.func1()
	/__w/go-livepeer/go-livepeer/core/orchestrator.go:753 +0x96 fp=0xc000761fe0 sp=0xc000761f10 pc=0x2b90456
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000761fe8 sp=0xc000761fe0 pc=0x1b51be1
created by github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSegmentLoop in goroutine 150
	/__w/go-livepeer/go-livepeer/core/orchestrator.go:738 +0x413

goroutine 146 gp=0xc0004b56c0 m=nil [select]:
runtime.gopark(0xc00040f7b8?, 0x5?, 0x0?, 0x0?, 0xc00040f436?)
	/__t/go/1.23.2/x64/src/runtime/proc.go:424 +0xce fp=0xc00040f2c0 sp=0xc00040f2a0 pc=0x1b490ae
runtime.selectgo(0xc00040f7b8, 0xc00040f42c, 0x4b7d160?, 0x0, 0x0?, 0x1)
	/__t/go/1.23.2/x64/src/runtime/select.go:335 +0x7a5 fp=0xc00040f3e8 sp=0xc00040f2c0 pc=0x1b248c5
net/http.(*http2serverConn).serve(0xc000604e00)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4812 +0x873 fp=0xc00040f848 sp=0xc00040f3e8 pc=0x1f48213
net/http.(*http2Server).serveConn(0xc000809500, {0xb30c98, 0xc000378708}, 0xc0006edb38, 0x0)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4402 +0xbc6 fp=0xc00040fb08 sp=0xc00040f848 pc=0x1f464a6
net/http.(*http2Server).ServeConn(...)
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4266
net/http.http2ConfigureServer.func1(0xc0008e73b0, 0xc000378708, {0xb0dae0, 0xc0007b2dc0})
	/__t/go/1.23.2/x64/src/net/http/h2_bundle.go:4185 +0x128 fp=0xc00040fb98 sp=0xc00040fb08 pc=0x1f45788
net/http.(*conn).serve(0xc0009365a0, {0xb25318, 0xc000675bc0})
	/__t/go/1.23.2/x64/src/net/http/server.go:2000 +0x1213 fp=0xc00040ffb8 sp=0xc00040fb98 pc=0x1f7c9f3
net/http.(*Server).Serve.gowrap3()
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x28 fp=0xc00040ffe0 sp=0xc00040ffb8 pc=0x1f81228
runtime.goexit({})
	/__t/go/1.23.2/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00040ffe8 sp=0xc00040ffe0 pc=0x1b51be1
created by net/http.(*Server).Serve in goroutine 30
	/__t/go/1.23.2/x64/src/net/http/server.go:3360 +0x485

rax    0x7d720c001080
rbx    0x7d71ec000b78
rcx    0x0
rdx    0x7d720c009100
rdi    0x7d720c009168
rsi    0x7d720c009100
rbp    0x7d720c001080
rsp    0x7d7225811650
r8     0x7d720c009100
r9     0x3f
r10    0x5
r11    0x7d720c000988
r12    0x0
r13    0x7d720c002aa0
r14    0x1
r15    0x7d7225811660
rip    0x2fdae9a
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0

[ad-astra-video]

Adding what I found but I am not reliably able to reproduce the seg fault specifically for some reason. @iameli are the segments in order for the stream in your script (...222Z is segment before ..042Z)? Or are the from different streams?

What I have noted in testing this:

• The seg fault comes from running the second segment on a manifest id and seg faults running first segment in twice (with diff segment number) or with the second test file. Mine does not always produce the seg fault though.
• both segments can produce this error below on my dev machine with A4500. It seems like its always after the first segment comes through and the second one using a different manifest Id fails with the error below but no seg fault. If I wait for the streams to close out from the broadcaster and transcoding sessions to end I can re-run either segment and they will work.

[h264 @ 0x751888c43640] decoder->cvdl->cuvidCreateDecoder(&decoder->decoder, params) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory
[h264 @ 0x751888c43640] Failed setup for format cuda: hwaccel initialisation returned error.
Failed to get HW surface format.
[h264 @ 0x751888c43640] decode_slice_header error
[h264 @ 0x751888c43640] no frame!
ERROR: decoder.c:91] Error sending packet to decoder : Invalid data found when processing input
ERROR: transcoder.c:365] Could not decode; stopping : Invalid data found when processing input

[iameli]

@ad-astra-video I had trouble reproducing it too. The segments are in order, yeah, two in a row from my livestream. They're GStreamer output, so perhaps not surprising that they don't play nicely with libav :)

If no one is actively using it it seems totally reasonable to just turn off MP4 support on the O layer.