[Bug]: Stream fails to gracefully restart after go2rtc error

[illuzn]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

go2rtc will generate io timeouts from time to time (after the scale of days of uptime). I believe this is where there is some form of packet loss over a long time scale.

go2rtc will correctly regenerate the stream (evidenced by it working in Advanced Camera Card in HA and also directly viewing the stream from go2rtc).

However, frigate generates constant socket errors when this occurs.

Steps to reproduce

1. Trigger an i/o timeout in go2rtc
2. Frigate will generate socket errors constantly

Version

v0.16.0 rc3 (but also rc1 and rc2 - didn't occur in v0.15.x)

In which browser(s) are you experiencing the issue with?

No response

Frigate config file

logger:
  default: info
  logs:
    nginx: warning

tls:
  enabled: false

mqtt:
  enabled: true
  host: homeassistant.lan
  user: frigate
  password: xxx

detectors:
  ov_0:
    type: openvino
    device: CPU
  ov_1:
    type: openvino
    device: CPU

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

record:
  enabled: true
  sync_recordings: false
  retain:
    days: 7
    mode: all
  alerts:
    retain:
      days: 21
    pre_capture: 30
    post_capture: 60
  detections:
    retain:
      days: 21
    pre_capture: 30
    post_capture: 60

detect:
  annotation_offset: -500
  enabled: true

audio:
  enabled: true
  listen:
    - scream
    - speech
    - babbling
    - yell
    - bellow
    - whoop
    - whispering
    - laughter
    - snicker
    - crying
    - sigh
    - singing
    - child_singing
    - humming
    - groan
    - grunt
    - children_playing
  max_not_heard: 200
  min_volume: 500

snapshots:
  enabled: true
  clean_copy: true
  quality: 100
  retain:
    default: 365

go2rtc:
  streams:
    garage:
      - rtsp://xxx:xxx@xxx:554/ch01/0
    front_door:
      - rtsp://xxx:xxx@xxx:554/ch02/0
    front_yard_left:
      - rtsp://xxx:xxx@xxx:554/ch03/0
    front_yard_right:
      - rtsp://xxx:xxx@xxx:554/ch04/0
    courtyard:
      - rtsp://xxx:xxx@xxx:554/ch05/0
    backyard_left:
      - rtsp://xxx:xxx@xxx:554/ch06/0
    side:
      - rtsp://xxx:xxx@xxx:554/ch07/0
    backyard_right:
      - rtsp://xxx:xxx@xxx:554/ch08/0
    baby_monitor:
      - onvif://xxx:xxx@xxx:8000
  webrtc:
    candidates:
      - stun:8555
      - xxx:8555

objects:
  track:
    - person
    - car
    - bus
    - bicycle
    - motorcycle
  filters:
    person:
      min_score: 0.8
      threshold: 0.9
      min_area: 15000
    car:
      min_score: 0.8
      threshold: 0.9
    bicycle:
      min_score: 0.8
      threshold: 0.9
    motorcycle:
      min_score: 0.8
      threshold: 0.9
    bus:
      min_score: 0.8
      threshold: 0.9

review:
  alerts:
    labels:
      - person
      - car
      - bus
      - bicycle
      - motorcycle

ffmpeg:
  hwaccel_args: preset-vaapi
  input_args: preset-rtsp-restream
  output_args:
    record: preset-record-generic

cameras:
  garage:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/garage
          roles:
            - record
            - detect
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      height: 320
      quality: 100
    motion:
      mask: 0.677,0.954,0.677,0.993,0.959,0.998,0.958,0.953
      threshold: 20
      contour_area: 25
      improve_contrast: true
    objects:
      filters:
        car:
          mask: 0.115,0.668,0.207,1,0.818,0.999,0.817,0.655
    genai:
      enabled: false

  front_door:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_door
          roles:
            - record
            - detect
    detect:
      enabled: true
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask: 0.681,0.948,0.682,0.999,0.956,0.998,0.954,0.947
      threshold: 30
      contour_area: 25
      improve_contrast: true
    audio:
      enabled: false
    genai:
      enabled: false
    lpr:
      enabled: false

  front_yard_left:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_yard_left
          roles:
            - record
            - detect
    review:
      alerts:
        required_zones:
          - offstreet_left
      detections:
        required_zones:
          - offstreet_left
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask: 0.69,0.947,0.69,0.998,0.967,0.999,0.966,0.95
      threshold: 30
      contour_area: 20
      improve_contrast: true
    objects:
      filters:
        car:
          mask: 0.063,0.44,0.083,1,1,1,0.714,0.519,0.524,0.333
        person:
          min_area: 3000
    zones:
      offstreet_left:
        coordinates: 
          0,0.367,0.064,0.066,0.071,0.234,0.413,0.187,0.76,0.207,0.775,0.092,0.915,0.206,1,0.304,1,1,0,1
        inertia: 1
        loitering_time: 0
    genai:
      enabled: true
      use_snapshot: true

  front_yard_right:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_yard_right
          roles:
            - record
            - detect
    review:
      alerts:
        required_zones:
          - offstreet_right
      detections:
        required_zones:
          - offstreet_right
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask: 0.689,0.949,0.689,0.998,0.966,0.999,0.966,0.949
      threshold: 30
      contour_area: 20
      improve_contrast: true
    objects:
      filters:
        car:
          mask: 
            0.475,0.159,0.497,0.278,0.457,0.371,0.316,0.514,0.772,0.627,0.76,0.996,0.123,1,0.076,0.859,0.182,0.696,0.148,0.32
        person:
          min_area: 3000
    zones:
      offstreet_right:
        coordinates: 
          0,0,0.132,0.001,0.131,0.104,0.242,0.04,0.248,0.146,0.517,0.162,0.765,0.211,0.777,0.043,0.899,0.5,0.972,1,0,1
        intertia: 1
        loitering_time: 0
    genai:
      enabled: true
      use_snapshot: true

  courtyard:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/courtyard
          roles:
            - record
            - detect
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask: 0.685,0.954,0.687,0.998,0.968,1,0.967,0.953
      threshold: 35
      contour_area: 25
      improve_contrast: true
    objects:
      filters:
        person:
          min_score: 0.9
          threshold: 0.95
    genai:
      enabled: false
    lpr:
      enabled: false

  backyard_left:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/backyard_left
          roles:
            - record
            - detect
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask:
        - 0.021,1,0.102,0.54,0.262,0.074,0.521,0.013,0.764,0.01,0.907,0.063,1,0.127,1,0,0,0,0,1
        - 0.692,0.953,0.691,1,1,1,1,0.953
      threshold: 40
      contour_area: 40
      improve_contrast: true
    objects:
      filters:
        person:
          min_score: 0.9
          threshold: 0.95
        car:
          mask: 0.191,0.309,0.341,0.242,0.409,0.853,0.249,0.929
    genai:
      enabled: false
    lpr:
      enabled: false

  side:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side
          roles:
            - record
            - detect
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask:
        - 0.688,0.953,0.687,0.994,0.964,0.993,0.963,0.953
        - 0.521,0,0.648,0.321,0.768,0.68,0.858,0.998,1,1,1,0.001
      threshold: 35
      contour_area: 20
      improve_contrast: true
    objects:
      filters:
        person:
          min_score: 0.9
          threshold: 0.95
    genai:
      enabled: false
    lpr:
      enabled: false

  backyard_right:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/backyard_right
          roles:
            - record
            - detect
    detect:
      enabled: true
    audio:
      enabled: false
    record:
      enabled: true
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      timestamp: false
      bounding_box: false
      crop: true
      quality: 100
      height: 320
    motion:
      mask:
        - 0.688,0.953,0.69,0.999,0.963,1,0.961,0.95
        - 0,0.16,0.347,0,0.838,0.077,0.906,0.416,0.941,1,1,1,1,0,0,0
      threshold: 40
      contour_area: 40
      improve_contrast: true
    objects:
      filters:
        person:
          min_score: 0.9
          threshold: 0.95
    genai:
      enabled: false
    lpr:
      enabled: false

  baby_monitor:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/baby_monitor
          roles:
            - record
            - detect
            - audio
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      enabled: false
    audio:
      enabled: true
    record:
      enabled: true
      retain:
        days: 1
        mode: all
    snapshots:
      enabled: false
    mqtt:
      enabled: false
    ui:
      order: 999
    genai:
      enabled: false
    lpr:
      enabled: false

version: 0.16-0

semantic_search:
  enabled: true
  reindex: false
  model_size: large

genai:
  enabled: true
  provider: openai
  api_key: abc
  model: google_gemma-3-4b-it-qat-q4_0-gguf_gemma-3-4b-it-q4_0
  object_prompts:
    person: This is an image from a security camera located at {camera}. For each
      person in the image, examine them and describe in detail any characteristics
      of the person (age, gender, clothing, job, what they are doing). Only use bullet
      points. There is no need to describe any static elements, provide an introduction
      or ask for further prompts.
    car: This is an image from a security camera located at {camera}. Describe the
      main car in this image. Provide as much detail as possible about the car in
      bullet point form (type of car, colour, make, model, what the car appears to
      be doing). There is no need to describe anything other than the car.

camera_groups:
  Front:
    order: 2
    icon: LuArrowBigUpDash
    cameras:
      - front_door
      - front_yard_left
      - front_yard_right
      - garage
  Baby:
    order: 1
    icon: LuBaby
    cameras: baby_monitor

face_recognition:
  enabled: true
  unknown_score: 0.8
  detection_threshold: 0.7
  recognition_threshold: 0.95
  min_area: 1000
  min_faces: 1
  save_attempts: 10000
  blur_confidence_filter: true
  model_size: large

lpr:
  enabled: true
  device: CPU
  model_size: small
  detection_threshold: 0.7
  min_area: 500
  recognition_threshold: 0.8
  min_plate_length: 2
  match_distance: 1

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    volumes:
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
      - /drives/nvme/frigate/recordings:/media/frigate
      - /drives/nvme/frigate/config:/config
      - /drives/nvme/frigate/onnx:/onnx
    devices:
      - /dev/dri
      - /dev/kfd
    shm_size: 1.5g
    environment:
      - FRIGATE_RTSP_PASSWORD=pRkocAGj6ZNewIDGWean
      - HSA_OVERRIDE_GFX_VERSION=10.3.0
      - LIBVA_DRIVER_NAME=radeonsi
      - OPENAI_BASE_URL=http://llama.lan:8080/v1
    image: ghcr.io/blakeblackshear/frigate:0.16.0-rc1-rocm
    network_mode: host
    mem_swappiness: 0
    memswap_limit: 0
    deploy:
      resources:
        limits:
          memory: 8G
    logging:
      options:
        max-size: "10m"
        max-file: "3"

Relevant Frigate log output

2025-08-09 10:59:12.724881101  BrokenPipeError: [Errno 32] Broken pipe
2025-08-09 10:59:12.724881923  Call stack:
2025-08-09 10:59:12.724883285    File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
2025-08-09 10:59:12.724884257      self._bootstrap_inner()
2025-08-09 10:59:12.724885660    File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
2025-08-09 10:59:12.724886532      self.run()
2025-08-09 10:59:12.724887774    File "/opt/frigate/frigate/video.py", line 441, in run
2025-08-09 10:59:12.724888696      capture_frames(
2025-08-09 10:59:12.724890059    File "/opt/frigate/frigate/video.py", line 143, in capture_frames
2025-08-09 10:59:12.724891511      logger.error(f"{config.name}: Unable to read frames from ffmpeg process.")
2025-08-09 10:59:12.724892854  Message: 'front_door: Unable to read frames from ffmpeg process.'
2025-08-09 10:59:12.724893686  Arguments: ()
2025-08-09 10:59:12.724894608  --- Logging error ---
2025-08-09 10:59:12.724895640  Traceback (most recent call last):
2025-08-09 10:59:12.724897032    File "/opt/frigate/frigate/video.py", line 137, in capture_frames
2025-08-09 10:59:12.724898315      frame_buffer[:] = ffmpeg_process.stdout.read(frame_size)
2025-08-09 10:59:12.724899587                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-09 10:59:12.724900609  ValueError: read of closed file
2025-08-09 10:59:12.724901311
2025-08-09 10:59:12.724902623  During handling of the above exception, another exception occurred:
2025-08-09 10:59:12.724903325
2025-08-09 10:59:12.724904326  Traceback (most recent call last):
2025-08-09 10:59:12.724905689    File "/usr/lib/python3.11/logging/handlers.py", line 1498, in emit
2025-08-09 10:59:12.724906801      self.enqueue(self.prepare(record))
2025-08-09 10:59:12.724908154    File "/usr/lib/python3.11/logging/handlers.py", line 1456, in enqueue
2025-08-09 10:59:12.724909206      self.queue.put_nowait(record)
2025-08-09 10:59:12.724910298    File "<string>", line 2, in put_nowait
2025-08-09 10:59:12.724911771    File "/usr/lib/python3.11/multiprocessing/managers.py", line 821, in _callmethod
2025-08-09 10:59:12.724912973      conn.send((self._id, methodname, args, kwds))
2025-08-09 10:59:12.724914526    File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
2025-08-09 10:59:12.724915819      self._send_bytes(_ForkingPickler.dumps(obj))
2025-08-09 10:59:12.724917442    File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes
2025-08-09 10:59:12.724918474      self._send(header + buf)
2025-08-09 10:59:12.724919977    File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send
2025-08-09 10:59:12.724921049      n = write(self._handle, buf)
2025-08-09 10:59:12.724922061          ^^^^^^^^^^^^^^^^^^^^^^^^

Relevant go2rtc log output

2025-08-07 05:06:31.757968083  05:06:31.757 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:170 > error="write tcp 192.168.xxx.xxx:39820->192.168.xxx.xxx:554: i/o timeout" url=rtsp://xxx:[email protected]:554/ch03/0

Operating system

Debian

Install method

Docker Compose

Network connection

Wired

Camera make and model

Swann DVR8-5580G (whitelabeled hikvision with custom firmware)

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

Coincidentally, this issue led me to change my max log size (and this should be included in the recommended settings) because my logs became over 100gb because this error repeats itself every 1ms.

Also restarting frigate resolves the issue.

[NickM-27]

We need to see a full copy of the logs, the socket error looks like it is precluded by some type of larger error.

[hawkeye217]

Another thing you could try that may mitigate your original issue is using the ffmpeg module with go2rtc, eg:

go2rtc:
  streams:
    garage:
      - ffmpeg:rtsp://xxx:xxx@xxx:554/ch01/0

[illuzn]

Okay some more information... I've tried passing through ffmpeg first that didn't fix it.

However, I noticed this in my proxmox monitoring:

Thats 8Gigs of disk read in a very short spike which causes the i/o timeout described above. I looked back at my logs and every other crash is caused by a similarly huge disk read:

Now it's not on a recurrent schedule, so I have no idea what would be causing this at all.

[illuzn]

@hawkeye217 Sorry to tag but any ideas. Nothing runs on this lxc except frigate.

[hawkeye217]

Technically Proxmox (especially running in an LXC) is an unsupported installation method. Many of our support requests are due to issues with Proxmox. Frigate runs best in Docker on bare metal.

There's been no other reports of this so I don't know what the issue could be.

The only way we can perhaps help further is to see the start of the errors in the log.

[illuzn]

I might have to switch to running inside a VM instead of an LXC... will get back to you

[hawkeye217]

Your GPU also appears to be overloaded. The full logs will reveal more, as @NickM-27 indicated.

[illuzn]

The GPU usage is reported incorrectly. This seems to be an issue with AMDGPU... I've checked it using radeontop and all is in order.