[Support]: ffmpeg crashing causing OOM kills #14571
Replies: 5 comments 4 replies
-
Forgot to add that the container usually uses between 1.5-2.5GB of memory during normal running |
Beta Was this translation helpful? Give feedback.
-
You can try updating to a newer version of ffmpeg, some users have said this fixed the issue for them. The next version of frigate will use a newer version by default. |
Beta Was this translation helpful? Give feedback.
-
I have had a very similar issue to this about 6 months ago: #10814 Also using Hikvision cameras, and resizing the stream manually using go2rtc. Unfortunately I've had to keep ffmpeg at 5.0 due to other issues; will have a look through my logs at whether this issue is still happening. I moved some of my Wifi cameras to ethernet which may have been the root cause of the slightly messy camera dropouts that go2rtc/ffmpeg were getting hung up on. |
Beta Was this translation helpful? Give feedback.
-
im kind of a noob but had a similar oom issue that i think i finally solved just last night. it was actually two fixes. 1) i'm running frigate on proxmox in a portainer container and noticed the container reports 4.3gb of memory although i only have 4gb allocated to the lxc. in portainer i changed the resource mapping to be around 3.8gb. 2) my reolink cameras were sending the detect stream in h.264 and the main recording stream in h.265. there is no way to change this on camera due to the higher resolution they say. In the frigate config, I had to enable different hardware accel for each stream (not globally) and specify hevc or h.264 respectively. since doing that i no longer have any ffmpeg errors and no more oom crashes. i'm not using go2rtc at all (at least i dont think so). not sure if this is helpful but I've seen the oom issue in a few posts now. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
Hi there,
I've been struggling with this issue for quite sometime. Originally I had been running Frigate as a container on Unraid, but ran into some strange issues with my server dying overnight. After finally getting the logs persisted I found that the host was being killed by OOM errors and was not able to enforce the memory limit I had set via docker args (--memory=8G). After some additional digging I believe that the cause of this was ffmpeg crashing and requesting a huge amount of memory (much more than th 32GB of the host).
My leading theory is that my cameras have inconsistent streams that constantly force ffmpeg to either restart gracefully or crash each time. Before properly understanding Frigate, I had just bought what I thought to be decent used IP cameras. It was only later that I discovered that most of my cameras don't have high enough sub stream resolutions to be useful for detection. I somehow got around this by downscaling the main stream as a substream to use for detect.
I have now moved Frigate onto kubernetes using the helm template with argocd, all running on proxmox VMs. Now I don't have the issue of the OOM errors killing the entire host, but instead just restarting the container. I had left it running overnight and saw that the container was killed 8 times within the space of about 7 hours. I've tried most of the options I could find to try and make it more stable but don't really have the knowledge to do so with confidence.
Any and all advice would be greatly appreciated along with some mercy when it comes to my config :)
Version
0.14.1-f4f3cfa
What browser(s) are you using?
No response
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output
FFprobe output from your camera
For some reason my ffprobes timeout from within the container and via my local
Frigate stats
Install method
Proxmox via Docker
docker-compose file or Docker CLI command
Object Detector
TensorRT
Network connection
Wired
Camera make and model
Hikvision and TP-Link
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
I know that my inference speed is quite slow with the large model, but as I've just moved the system over, I'm running with my test GPU. The normal GPU I use is a 1660 super with inference times just under 30ms with the largest model
Beta Was this translation helpful? Give feedback.
All reactions