[BUG] Resolve fallback behavior with NVIDIA NVENC

[ehfd]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

INFO:gst_app_resize:Display client 'primary' dimensions updated to 2880x1672. Triggering reconfiguration.
INFO:data_websocket:Starting display reconfiguration...
INFO:data_websocket:Stopping all existing capture and backpressure tasks...
INFO:data_websocket:Backpressure logic task for 'primary' cancelled.
INFO:data_websocket:Backpressure logic task for 'primary' finished.
INFO:data_websocket:Backpressure task for 'primary' was stopped. Resetting its frame IDs.
INFO:data_websocket:Resetting frame IDs for display 'primary'.
INFO:data_websocket:Broadcasting primary pipeline reset to all 1 clients: PIPELINE_RESETTING primary
NVENC_INIT_ERROR: nvEncOpenEncodeSessionEx (CUDA Path) FAILED: 2
NVENC Encoder initialization failed. Falling back to x264.
CPU cores available: 80
Stream settings active -> Res: 2880x1674 | FPS: 60.0 | Stripes: 1 | Mode: H264 (CPU) FullFrame | CRF: 25 | PaintOver
X Error of failed request:  BadAccess (attempt to access private resource denied)
  Major opcode of failed request:  130 (MIT-SHM)
  Minor opcode of failed request:  1 (X_ShmAttach)
  Serial number of failed request:  12
  Current serial number in output stream:  14

NVENC Encoder Initialized successfully.
CPU cores available: 80
Stream settings active -> Res: 2880x1674 | FPS: 60.0 | Stripes: 1 | Mode: H264 (NVENC) FullFrame | CRF: 25 | PaintOv
X Error of failed request:  BadAccess (attempt to access private resource denied)
  Major opcode of failed request:  130 (MIT-SHM)
  Minor opcode of failed request:  1 (X_ShmAttach)
  Serial number of failed request:  12
  Current serial number in output stream:  14
[pcmflux] First non-silent audio chunk detected! Encoding...

Two separate examples; one when NVENC works properly (validated through FFmpeg, etc.), and one where it doesn't (due to NVIDIA/k8s-device-plugin#1282).

Expected Behavior

Fallback or NVENC encoding should be done correctly.

Steps To Reproduce

Invoke the container in an NVIDIA GPU environment.

Environment

- OS: Ubuntu 24.04
- How docker service was installed: Docker CE Ubuntu APT package, NVIDIA Container Toolkit

Docker creation

git clone https://github.com/selkies-project/selkies.git
cd selkies
docker run --rm -it --gpus 1 \
  --shm-size=1gb \
  -e DEV_MODE=core \
  -e PUID=1000 \
  -e PGID=1000 \
  -v $(pwd):/config/src \
  -p 3001:3001 linuxserver/webtop:ubuntu-mate-selkies bash

Container logs

Above.

[thelamer]

I don't think this is Nvidia, I added X error catching and retry to this can you test it on your setup ?

taisun/random-images:shm-test

[ehfd]

@thelamer This resolved the problem completely. I agree that it was because of the X error, but either way, it's solved.

[thelamer]

Great this just got baked into the baseimages along with the automated vaapi support on intel/amd, will waterfall to everything this week.