Memory leak with ffmpeg

[gonzalezcalleja]

Hi!

I have detected some memory leak withe some Foscam cameras (C5M), for example this ffmpeg process:

ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.14.0-da913d8 -rtsp_transport tcp -timeout 5000000 -i rtsp://redacted:[email protected]:8554/cocina?video=copy&audio=aac -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/cocina@%Y%m%d%H%M%S%z.mp4

After many hours working well, this ffmpeg process started to consume a lot of ram (~6Gb and growing) and then swap ... any advice on where to continue in the search for the cause of the problem?*

I'm not using virtualization, is all baremetal:

• AMD Ryzen 7 5700U
• Coral accelerator -PCIe in M2 adapter-
• Ubuntu 24.04
• kernel 6.8.0-40-generic

I'm using this camera through go2rtc:

    cocina:
      - ffmpeg:rtsp://redacted:[email protected]:88/videoMain#video=copy#audio=copy#audio=aac

and this is the camera config:

  cocina:
    ui:
      order: 8
      dashboard: true
    ffmpeg:
      input_args: preset-rtsp-restream
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://redacted:[email protected]:8554/cocina_sub?video=copy&audio=copy
          roles:
            - detect
            - audio
        - path: rtsp://redacted:[email protected]:8554/cocina?video=copy&audio=aac
          roles:
            - record
    objects:
      track:
        - person
        - dog
      filters:
        person:
          min_score: 0.8
    live:
      stream_name: cocina_sub
    audio:
      enabled: true
      min_volume: 600
    motion:
      mask: 0.348,0,0.268,0.032,0,0.49,0,0
    detect:
      width: 1280
      height: 720

Thanks for your support!

[XuZhen86]

Also seeing ffmpeg slow memory leak on my end, I'm using a Jetson Nano 4GB board with Docker image ghcr.io/blakeblackshear/frigate:0.14.0-tensorrt-jp4. The main memory slowly fills up, followed by swap memory (zram and swap file) slowly fills up.

For now I've added a crontab entry to restart Frigate every 24h.

# Restart Frigate at 4:00am everyday.
0 4 * * * curl --request POST http://localhost:5000/api/restart

I've tried simplifying the ffmpeg args to isolate the issue but with no luck. These processes leak memory:

ffmpeg -hide_banner -loglevel warning -c:v h264_nvmpi -rw_timeout 30000000 -i http://127.0.0.1:1984/api/stream.mp4?src=camera_main -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/camera1@%Y%m%d%H%M%S%z.mp4
ffmpeg -hide_banner -loglevel warning -c:v h264_nvmpi -rw_timeout 30000000 -i http://127.0.0.1:1984/api/stream.mp4?src=camera_ext -r 5 -vf fps=5,scale=896:672 -f rawvideo -pix_fmt yuv420p pipe:

These processes do not leak memory:

ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1792x672 -i pipe: -threads 1 -f mpegts -s 1792x672 -codec:v mpeg1video -q 8 -bf 0 pipe:
ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 896x672 -i pipe: -threads 1 -f mpegts -s 896x672 -codec:v mpeg1video -q 8 -bf 0 pipe:

In any case it doesn't seem to be a bug in Frigate and it has been working wonderfully! It would be cool for Frigate to be able to monitor the memory/swap usage and restart ffmpeg processes if needed.

Edit: ZRAM and swap usages

jetson@jetson:~$ zramctl 
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram3 lzo             2G 505.2M 197.6M 207.7M       4 [SWAP]
/dev/zram2 lzo             2G 505.2M 197.5M 207.7M       4 [SWAP]
/dev/zram1 lzo             2G 505.1M 197.5M 207.7M       4 [SWAP]
/dev/zram0 lzo             2G 504.6M 197.3M 207.3M       4 [SWAP]
jetson@jetson:~$ for file in /proc/*/status; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r
frigate.detecto 733524 kB
ffmpeg 272576 kB
ffmpeg 269388 kB
ffmpeg 247060 kB
ffmpeg 240552 kB
python3 232132 kB
frigate.capture 165616 kB
frigate.capture 163528 kB
frigate.process 161744 kB
frigate.process 160576 kB
frigate.output 157908 kB
frigate.review_ 150892 kB
frigate.logger 139016 kB
frigate.recordi 57288 kB

Edit 2:
Instead of restarting Frigate, we can tell ffmpeg to exit after some time using the -t parameter. Frigate will restart ffmpeg after it exists.

ffmpeg:
  input_args:
    - -rw_timeout
    - "30000000" # 30s
    - -t
    - "12:34:56.789" # Exit after running for this duration to release the leaked memory. Frigate then auto restarts ffmpeg.

[gonzalezcalleja]

I would like to detect the reason for ffmpeg memory leak ... I have about 20 cameras from different brand but the memory leak is always with one of my 4 Foscam C5M so I think the problem could be with the stream from this camera.

EDIT: I have frigate with 64GB RAM and recordings over NFS v3 share

[gonzalezcalleja]

@XuZhen86 we should to try ffmpeg 7 update : #13578

[XuZhen86]

Great idea, thanks for pointing that out! I'll give it a try when the next release comes out.

[gonzalezcalleja]

@XuZhen86 Have you found any solution to resolve this issue? Thanks!

[XuZhen86]

I added the -t "12:34:56.789" param to ffmpeg and it has been working well so far. The memory leak is still happening, but ffmpeg gets restarted regularly and the effect is barely noticeable. It's a "good-enough" solution and I didn't look closer.
0.14.2 has not been released but I'm too lazy to build the image myself lol.

[gonzalezcalleja]

Thanks, I will try this timeout.

You can try develop branch without build with this images: ghcr.io/blakeblackshear/frigate. For example last now build is: ghcr.io/blakeblackshear/frigate#eca504c

But I can confirm that the memory leak problem is still present.

[XuZhen86]

I've moved Frigate to an Intel machine with image ghcr.io/blakeblackshear/frigate:0.15.1 and ffmpeg 7.0, but it looks like ffmpeg still has a slow leak. I'm keeping the -t "12:34:56.789" param to mitigate this issue.