After a few hours running: Unable to keep up with recording segments in cache

[Ayuus]

Describe the problem you are having

Since Frigate 0.14, detection stops working after a few hours (less than 24 in my case). After starting, everything is running OK, but after a day of continous recording with 3 camera's (2x4MP, 1x8MP) on my Synology D224+, the log is full of messages about "Unable to keep up with recording segments in cache", Coral inference speed is 96ms, Detection fps is 0.
I have not noticed this with 0.13.2.

Steps to reproduce

Start Frigate 0.14 with config as stated below and wait 24 hours.

Version

0.14.0-beta3 and beta4

Frigate config file

mqtt:
  host: 192.168.1.10 #homeassistant.local
  port: 1883
#  port: 8883
#  tls_ca_certs: /etc/ssl/certs/ca-certificates.crt
#  tls_insecure: false
  user: user
  password: password
  
ui:
  time_format: 24hour

#logger:
#  default: info
#  logs:
#    frigate.mqtt: info
#    frigate.ptz.onvif: debug

ffmpeg:
  hwaccel_args: preset-vaapi
  
detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track:
    - person
    - cat
    - dog
#  filters:
#    person:
#      threshold: 0.75
#    cat:
#      threshold: 0.75
#    dog:
#      threshold: 0.75

detect:
  max_disappeared: 5

#motion:
#   lightning_threshold: 0.9

birdseye:
  enabled: false

snapshots:
  enabled: true
  timestamp: true

audio:
  # geen audio events, events zijn wel inclusief audio
  enabled: false
  min_volume: 500
  listen:
    - speech
    - yell
    - scream
  # Optional: Filters to configure detection.
  filters:
    # Label that matches label in listen config.
    speech:
      # Minimum score that triggers an audio event
      threshold: 0.8
    yell:
      threshold: 0.8
    scream:
      threshold: 0.8

record:
  enabled: true
  retain:
    days: 7
    mode: all
  events:
    retain:
      default: 30
      mode: motion

go2rtc:
  streams:
    buitencamera_zijgevel:
      - rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=3 #720p/25fps
    buitencamera_achterzijde:
      - rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2 #720p/25fps
    buitencamera_voordeur:
      - rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=3 #720p/25fps

cameras:
  buitencamera_voordeur:
    ffmpeg:
      output_args:
        #audio format must be AAC, so we set this as format in camera configuration
        record: preset-record-generic-audio-copy
      inputs:
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2   #720p/5fps
          roles:
            - audio
            - detect
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0   #4MP25fps
          roles:
            - record
#    onvif: # necessary for ptz handling via frigate, werkt niet
#      host: 192.168.4.42
#      port: 80
#      user: user
#      password: password

  buitencamera_zijgevel:
    ffmpeg:
      output_args:
        #audio format must be AAC, so we set this as format in camera configuration
        record: preset-record-generic-audio-copy
      inputs:
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2     #720p/5fps
          roles:
            - audio
            - detect
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0     #4MP25fps
          roles:
            - record
    motion:
      mask:
        - 843,261,1100,720,1280,720,1280,0,868,0,484,0,493,267
    zones:
      grind:
        coordinates: 505,266,912,250,1148,720,509,720
    snapshots:
      required_zones:
        - grind
    review:
      alerts:
        required_zones:
          - grind
#    onvif: # necessary for ptz handling via frigate, werkt niet
#      host: 192.168.4.40
#      port: 80
#      user: user
#      password: password

          
  buitencamera_achterzijde:
    ffmpeg:
      output_args:
        #audio format must be AAC, so we set this as format in camera configuration
        record: preset-record-generic-audio-copy
      inputs:
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=1   #720p/5fps
          roles:
            - audio
            - detect
        - path: 
            rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0   #4MP25fps
          roles:
            - record
#    onvif: # necessary for ptz handling via frigate, werkt niet
#      host: 192.168.4.41
#      port: 80
#      user: user
#      password: password

#  ptz_achterzijde: # oude ptz camera
#    ffmpeg:
#      inputs:
#      - path: rtsp://mpfc:[email protected]:554/stream=0
#        roles:
#        - audio
#        - detect
#        - record
#    onvif: # necessary for ptz handling via frigate
#      host: 192.168.1.104
#      port: 8899
#      user: mpfc
#      password: password
#      autotracking:
#        enabled: true
#        zooming: relative
#    motion:
#      mask:
#      - 1738,1017,1213,364   
version: 0.14

Relevant log output

Log is spammed with messages:

Unable to keep up with recording segments in cache for buitencamera_achterzijde. Keeping the 6 most recent segments out of 7 and discarding the rest...

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

EmpireTech/Dahua

Any other information that may be helpful

Continuous recording was no problem with Frigate 0.13.2. Live images still work real-time, Synology CPU is higher than normal (45% vs 28%)

[arhills]

I have also had this issues with version 14, I have tried most of the releases all with the same issue. V13 runs fine with 9 cameras, 6 of them have detection setup. I checked the recording to drive times as suggested in the docs and they are the same in v14 as they are in v13. They are occasionally over 1 second, but v13.2 keeps working fine whereas v14 just crashes. CPU does go to 100% at times but its the same with both versions. I have tried stopping detection and recording on 3 cameras to see if the load is less and whether it stops the crashes or not. Fewer cameras recording and it's the same, just crashes. I can post logs but as they are similar to logs already posted I decided not to. Let me know if they will help.

[NickM-27]

Need a full copy of logs, this is usually triggered by some other crash or error

[hawkeye217]

Have you tuned your motion detection settings?

There are no changes from 0.13.2 to 0.14 in the way the coral is handled. If your machine's load is higher as a result of some of the changes in 0.14 or your coral is being taxed more because of your settings (the USB coral is notoriously power sensitive), that may explain why it would hang.

[Ayuus]

No, I did not tuned motion detection settings. Refer to yaml in my original post, would it help if I limit detection to persons only?

[hawkeye217]

You could try that.

The indications just point to an underpowered system or a lot of activity on your cameras (your coral inference speed is 10x slower than most users). You were probably just right on the edge of having issues with 0.13. Anything you can do to reduce the amount of work Frigate is doing will likely improve your situation.

[Ayuus]

Limiting detections to persons only did not make a difference. Issue occured after midnight, definately not a lot of activiti on my cameras. I thought 9ms inference speed is OK, what numbers am I supposed to see? Later, I will try with an extra USB power supply. For now, my only stable option seems to be reverting to 0.13.2, sadly.
BTW I will be happy to try any suggestion!

[hawkeye217]

Sorry, yes, 9ms is good. Your original post said 96ms.

One suggestion: You could try running atop and grabbing historical data to see if you can pinpoint the issue that loads down your system at the time of the crash.

[apranda]

Check whether you have Python bus errors that occur shortly after the beginning of every hour. This causes the same recording errors to occur on my instance running on a QNAP NAS

[Ayuus]

I run Frigate on a Synology 224+. I did not notice any error there.

[Ayuus]

Indeed, now I see Python Bus errors, but not after every hour...

[apranda]

Interestingly, I removed the only camera from my setup that had the highest CPU/GPU load due to transcoding and resizing and my errors and crashes have completely resolved themselves for the past 12 hours. It appears that if the system is already fairly heavily loaded, the operations that are done at the top of every hour will cause the system to become overloaded, which then shm will start to fill up, eventually causing a Bus error and resulting in a crash.

[Ayuus]

I close this issue, as my original issue "Unable to keep up with recording segments in cache" in log was solved by increasing SHM size from 96 to 256mb (for 3 cameras).
For other issue "crashing ffmpeg" I will start a new discussion.