Gateway crashes with a `panic: runtime error` when its Orchestrator disappears while transcoding a stream

[chrishobcroft]

Describe the bug

When processing a stream being transcoded via an Orchestrator, the Gateway crashes with panic: runtime error: invalid memory address or nil pointer dereference when the Orchestrator process becomes unavailable.

To Reproduce

Steps to reproduce the behaviour:

1. Open 3 terminal windows
2. In first terminal window run an offchain Gateway with:

./livepeer -gateway -orchAddr 127.0.0.1:8935 -transcodingOptions P144p30fps16x9 -v 99

3. In second terminal window run an offchain Orchestrator/Transcoder with:

./livepeer -orchestrator -transcoder -serviceAddr 127.0.0.1:8935 -v 99

4. In third terminal window run an ffmpeg Publisher with:

ffmpeg -re -f lavfi -i testsrc=size=640x360:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 1000k        -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1935/test_source

5. In the second terminal window, press ctrl-C to cancel the Orchestrator process.
6. In the first terminal window, observe that the Gateway process has crashed, reporting the following:

E1018 11:16:37.577703   72365 segment_rpc.go:557] orchSessionID=212063a9 orchestrator=https://127.0.0.1:8935 Unable to submit segment orch=https://127.0.0.1:8935 orch=https://127.0.0.1:8935 uploadDur=400.712µs err="Post \"https://127.0.0.1:8935/segment\": dial tcp 127.0.0.1:8935: connect: connection refused"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2e1ed85]

goroutine 719 [running]:
github.com/livepeer/go-livepeer/server.NewSessionManager.func1({0x0?, 0xc000c379a0?})
	/__w/go-livepeer/go-livepeer/server/broadcast.go:540 +0x25
github.com/livepeer/go-livepeer/server.(*SessionPool).removeSession(0xc001576200, 0xc000a64300)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:453 +0x9e
github.com/livepeer/go-livepeer/server.(*BroadcastSessionsManager).suspendAndRemoveOrch(0xc000a2a5c0, 0xc000a64300)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:569 +0xa5
github.com/livepeer/go-livepeer/server.transcodeSegment({0xb3b100, 0x4bc0e20}, 0xc0003b00f0, 0xc000a9e180, {0xc000c900f0, 0xb}, 0x0, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1180 +0x748
github.com/livepeer/go-livepeer/server.processSegment({0xb3b100, 0x4bc0e20}, 0xc0003b00f0, 0xc000a9e180, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1074 +0xc85
created by github.com/livepeer/go-livepeer/server.(*LivepeerServer).StartMediaServer.gotRTMPStreamHandler.func5.1.1 in goroutine 586
	/__w/go-livepeer/go-livepeer/server/mediaserver.go:437 +0xd2

Expected behavior

In the context of an offchain Gateway, at step 6, I expect the Gateway to continue to ingest and egress the stream, albeit without transcoded content. It ought to handle the disappearance of the Orchestrator gracefully.

In the context of an onchain Gateway, it ought to be able to gracefully discover and employ another Orchestrator from the active set, in order to avoid unnecessary discontinuity of the playback for Consumers viewing the transcoded content.

Screenshots

Environment

Livepeer Node Version: 0.8.8
Golang runtime version: gc go1.23.2
Architecture: amd64
Operating system: linux

[chrishobcroft]

Perhaps also related:

To Reproduce

Steps to reproduce the behaviour:

1. Open 3 terminal windows
2. In first terminal window run an offchain Gateway with:

./livepeer -gateway -orchAddr 127.0.0.1:8935 -transcodingOptions P144p30fps16x9 -v 99

3. In second terminal window run an offchain standalone Orchestrator (i.e. without a Transcoder) with:

./livepeer -orchestrator -orchSecret pineapple -serviceAddr 127.0.0.1:8935 -v 99

4. In third terminal window run an ffmpeg Publisher with:

ffmpeg -re -f lavfi -i testsrc=size=640x360:rate=30,format=yuv420p -f lavfi -i sine -c:v libx264 -b:v 1000k        -x264-params keyint=60 -c:a aac -f flv rtmp://127.0.0.1:1935/test_source

5. In the first terminal window, observe that the Gateway has crashed, reporting:

E1018 11:40:46.437793   80515 segment_rpc.go:618] orchSessionID=eeca637f orchestrator=https://127.0.0.1:8935 Transcode failed for segment orch=https://127.0.0.1:8935 err="no transcoders available"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2e1ed85]

goroutine 33 [running]:
github.com/livepeer/go-livepeer/server.NewSessionManager.func1({0x0?, 0xc0008359a0?})
	/__w/go-livepeer/go-livepeer/server/broadcast.go:540 +0x25
github.com/livepeer/go-livepeer/server.(*SessionPool).removeSession(0xc000a24280, 0xc000942000)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:453 +0x9e
github.com/livepeer/go-livepeer/server.(*BroadcastSessionsManager).suspendAndRemoveOrch(0xc0008c2280, 0xc000942000)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:569 +0xa5
github.com/livepeer/go-livepeer/server.transcodeSegment({0xb3b100, 0x4bc0e20}, 0xc0008b65a0, 0xc0008c22c0, {0xc000894030, 0xb}, 0x0, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1180 +0x748
github.com/livepeer/go-livepeer/server.processSegment({0xb3b100, 0x4bc0e20}, 0xc0008b65a0, 0xc0008c22c0, 0x0)
	/__w/go-livepeer/go-livepeer/server/broadcast.go:1074 +0xc85
created by github.com/livepeer/go-livepeer/server.(*LivepeerServer).StartMediaServer.gotRTMPStreamHandler.func5.1.1 in goroutine 44
	/__w/go-livepeer/go-livepeer/server/mediaserver.go:437 +0xd2

Expected behaviour

At step 5, I expect the Gateway to process the stream without transcoding it, i.e. as if it couldn't find an Orchestrator at all.

Screenshot