[Support]: Frigate leaking GBs of memory with Proxmox

[stgraber]

Describe the problem you are having

Frigate has been leaking GBs of memory ever since upgrading from 0.12 to 0.13.2.

I used to run on 4GB of RAM which Frigate fully consumed in less than 1h, I then bumped the memory of my instance to 8GB of RAM and Frigate still managed to eat it all overnight.

1524288   493608  2.7  0.9 7393656 4941276 ?     Rsl  05:16  17:28          |   |   \_ python3 -u -m frigate

Above shows a resident memory usage of the main frigate process at over 4.9GB which combined with all the other support processes was taking all 8GB of RAM.

Doing a quick test over a 1h time period:

root@frigate01:~# grep VmRSS /proc/699/status ; sleep 1h ; grep VmRSS /proc/699/status
VmRSS:	  747184 kB
VmRSS:	  860048 kB

So frigate seems to be leaking memory at a rate of just under 2MB/minute here.

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  host: PRIVATE
  user: PRIVATE
  password: PRIVATE

ffmpeg:
  hwaccel_args: preset-vaapi

cameras:
  shf-camera01:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera01
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera02:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera02
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera03:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera03
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera04:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera04
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera05:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera05
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - bird
    record:
      events:
        objects: []

  shf-camera06:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera06
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera07:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera07
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera08:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera08
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - car
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera09:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera09
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera10:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera10
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True

  shf-camera11:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera11
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True
      required_zones:
        - driveway
    record:
      events:
        required_zones:
          - driveway
    zones:
      driveway:
        coordinates: 815,1019,2304,1088,2304,1296,0,1296,0,1168,483,943

  shf-camera12:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera12
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - bird
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera13:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera13
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True

  shf-camera14:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera14
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - bird
        - cat
        - person
    snapshots:
      enabled: True

record:
  enabled: True
  retain:
    days: 7
    mode: all
  events:
    retain:
      default: 10
      mode: active_objects
    post_capture: 0
    pre_capture: 1

detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track: []

snapshots:
  enabled: False
  bounding_box: True
  crop: true
  retain:
    default: 10

birdseye:
  enabled: True
  mode: continuous

live:
  height: 720
  quality: 5

Relevant log output

The process being completely out of memory resulted in no log entries at all for the affected time period (10h):

2024-04-09 05:38:48.358042876  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:36 -0400] "GET /api/shf-camera11/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.667999486  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera06/latest.jpg?h=444 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.668297252  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera07/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 15:52:02.554579245  [INFO] Preparing Frigate...
2024-04-09 15:52:02.566412169  [INFO] Starting NGINX...
2024-04-09 15:52:02.663614853  [INFO] Preparing new go2rtc config...
2024-04-09 15:52:02.697756151  [INFO] Starting Frigate...
2024-04-09 15:52:03.570434492  [INFO] Starting go2rtc...
2024-04-09 15:52:03.682457180  11:52:03.682 INF go2rtc version 1.8.4 linux/amd64
2024-04-09 15:52:03.683346518  11:52:03.683 INF [api] listen addr=:1984
2024-04-09 15:52:03.683645144  11:52:03.683 INF [rtsp] listen addr=:8554
2024-04-09 15:52:03.684545352  11:52:03.684 INF [webrtc] listen addr=:8555

FFprobe output from your camera

Not relevant as not camera specific.

Frigate stats

No response

Operating system

Debian

Install method

Docker CLI

Coral version

USB

Network connection

Wired

Camera make and model

Mix of Reolink and bunch of other vendors (not relevant)

Any other information that may be helpful

Seems similar to #10649 but in my case the system was just preventing memory allocations rather than triggering the kernel out of memory killer.

[NickM-27]

A lot of time has been spent looking in to this. https://github.com/bloomberg/memray is a great tool for understanding where memory allocations are going.

In my usage it appears there is no memory leak, the heap size does not increase at all, python just does not release the memory after it has been garbage collected. I don't think there is any simple / immediate solution. https://www.reddit.com/r/learnpython/comments/17j1aff/rss_memory_inflates_leaks_while_heap_memory_size/?rdt=33549

Perhaps in your case it is something different, a memray output would show what is using memory

[stgraber]

Okay, the fact that there are reports of similar behavior in other python3 applications that similarly are running inside of resource constrained containers makes me want to look at a couple of options here, will report back in a few minutes (or hours if I need to do a full leak test).

[stgraber]

Basically, the other reporter mentioned running this on a Proxmox server, so with Frigate running (with or without Docker, not sure) inside of a resource constrained LXC container.
In my case, I'm running Frigate inside of a Docker container running inside of an Incus container which is similarly resource constrained.

The container I usually give 4GB of RAM out of the 500GB+ of the host system.
By default Docker doesn't use LXCFS which is a way to expose cgroup resource limits to containers.
That means that even though the environment is constrained to 4GB of RAM and that limit is clearly visible in /proc/meminfo and free -m, that same limit isn't visible INSIDE the Docker container that Frigate runs in.

Now if python3 decided to be funny with caching and allow for memory to grow up till some arbitrary value determined from the total memory it think it can use, then that'd explain the problem.

My first attempt at fixing this is simply by manually telling Docker to map LXCFS on /proc/meminfo, /proc/swaps, ... inside of the Frigate container. If python3 is looking at those files to figure out how much memory is available on the system, then that will work fine and it shouldn't get to a point where it runs the whole thing out of RAM anymore.

If that doesn't work, then it's possible that python3 makes use of the sysinfo system call instead to get memory information. If that's the case, then Incus also has a way to handle that through system call interception and emulation, so I can easily turn that one on. That would be bad news for Proxmox users though as I don't believe Proxmox has support for system call interception.

[stgraber]

Still appears to be consuming memory at an alarming rate. I'm also tracing it for good measure and see that something within the python3 process does call the sysinfo system call every minute or so.

I can't tell what the information is used for but it certainly seems likely that this is the culprit and that I'll have to turn on interception to fix this issue.

[stgraber]

Right, initial test shows 8MB/minute when running under a 4GB total limit.
I've not given it enough time to run fully out of memory but it's not showing any sign of slowing down despite total free memory being below 400MB by now.

I'll turn on the sysinfo interception and see if the behavior is different.

[NickM-27]

The every minute sys info call is most likely the stats which are presented via MQTT for the system page / integration

[stgraber]

Ah, that'd make sense, though I guess the same underlying logic within python may be used for setting up some kind of memory limit on process startup.

[stgraber]

And running with sysinfo interception, so far so good, process usage hovers around 485MB and seems to be actually releasing memory back to the OS following spikes. I'm still waiting for the results after keeping this going for an hour or so to make sure things are actually stable.

If they are, then we'll know what python3 is doing now and how to fix it, at least for anyone using Incus, those not running on a container platform capable of system call emulation will be out of luck unfortunately.

[stgraber]

Docker or Podman users will most likely run into the same issue if they specify a memory limit for their Frigate container as the python3 processes running inside of that container will not see the limit and therefore will plan their memory allocation based on the total amount of memory available on the system.

Now I don't think that's necessarily a very common setup and even if it was, as Frigate is likely to get the lion's share of the system's memory anyways, the risk of python3 hitting the issue is pretty small.

That's assuming that most Frigate users run on "normal" systems, Rpi4/Rpi5, Intel NUC and the like.

In my case I'm running into the worst version of this because my server is a 48-core AMD EPYC with 500GB of RAM. So Frigate not seeing the limit of 4GB causes it to assume it's got 500GB to work with!
So even if python3 had a conservative allocation of say, up to 10% of total system memory, that would still allow it to grow to 50GB in my case ;)

[NickM-27]

right, I am asking because I'd be curious to see how it behaves on my system which is an AMD 5700G Unraid NAS with 64GB of RAM but only 8GB allocated to frigate. I have never run out of memory but it does sometimes make its way up to 5GB or so

[stgraber]

@andrewhack can you comment on how much memory your Proxmox host has?

Your report shows you indeed have a 2GB cgroup limit on the container and that python3 consumed all of it causing the OOM killer to trigger, but the kernel trace doesn't show the total system memory you're working with.

I'm kinda hoping that your system has 32GB+ of RAM. If it does, then it's basically a repeat of what I'm seeing, which is going to be annoying for you given you're on Proxmox and can't just turn on sysinfo emulation, but would at least be good news for anyone running Frigate on lower specced systems as they'd hopefully not hit the issue.

[stgraber]

@NickM-27 it really looks to me like python3 is aiming for a sweet spot of around 10% of system memory given I've now limited it to 4GB and it's staying below 500MB of RAM.

So on your system with 64GB of RAM, I'd then expect the process to be in the 5-6GB range which would match up with what you're seeing. Now the issue is that if you were to double your system's RAM to 128GB, then Frigate would try to hit the 10-12GB RAM range and crash your container due to the 8GB limit.

[heisenberg2980]

@NickM-27 this article that @lost-RD shared below is really good for understanding the issue with memory limits inside a container: https://fabiokung.com/2014/03/13/memory-inside-linux-containers/

Now the question is, do we have any way to avoid this? is there any alternative to the python3 process that Frigate can use? or alternatively, does anyone know where we can open an issue so someone can fix the python3 process so it works properly inside a container?

[stgraber]

Right, so a couple hours later and memory usage is stable, so the issue is confirmed to be related to python3 now calling sysinfo which if that returns an amount of total memory higher than what's allowed for the container will cause python3 to start growing the process memory until it crashes.

[heisenberg2980]

@NickM-27 I guess this issue must be happening to anyone running Frigate in a Proxmox LXC with a small percentage of memory assigned to the container in relation to the total memory available in the Proxmox server. My understanding is the only solution available at the moment is going back to Frigate v0.12 (please correct me if I am wrong), are you aware of any idea/plan to solve this issue in the future?

[NickM-27]

frigate version has no effect on this issue. If you are not seeing it in frigate 0.12 but are seeing it in frigate 0.13 then you are seeing a different issue or a combination of issues

[heisenberg2980]

@NickM-27 as @stgraber mentioned in the description of this defect, in my case I also started having this memory issue after upgrading from Frigate v0.12 to v0.13, so I assumed python3 was being used in a different way in v0.12. But regarthless of the version this started, this seems like a really important issue affecting a lot of Frigate users, and we found the reason for the issue in this thread which is great, now the next question is can it be fixed?

[lost-RD]

@stgraber Did you find where in the codebase this is happening?

Might be best to mark this thread as unanswered until we clear up enough of the details to create an issue

[stgraber]

Well, in my case I tracked this down to python3 somehow using sysinfo() to figure out how much memory it has rather than rely on cgroup values or parsing /proc files which could use LXCFS for the correct value.

I'm running on Incus which supports sysinfo emulation specifically to deal with those kind of cases, so I turned it on and the problem went away, effectively confirming the hypothesis and fix.

Now if you're running using a container platform which doesn't support system call interception and emulation of sysinfo, then you're going to have a problem with any piece of software which relies on that system call to figure out system memory.

[LiloBzH]

hello

ok it's a problem of python3 and not frigate but we can't do anything ?

Usage of RAM is really a problem

[heisenberg2980]

@LiloBzH I´m not an expert on this at all, but I guess unless Frigate can replace python3 with something else (which I don´t know if it´s possible) the only other way for this issue to be fixed is when the python3 process or the sysinfo function are fixed so the python3 process is aware of the real memory assigned to the LXC and instead of the total memory available in the machine. Does anyone know which forum we should use to raise this issue? maybe the Proxmox forum?

[LiloBzH]

I reboot the LXC every 3 days to save RAM and SWAP :'(

[heisenberg2980]

@nautilus7 have you executed this command as @LiloBzH mentioned? what is the ouput:

proxmox-boot-tool kernel list

[nautilus7]

The first and only proxmox version I ever installed on my machine is 8.2, so I only got two different versions of kernel 6.8 installed.

[heisenberg2980]

@LiloBzH I can confirm after downgrading proxmox to kernel 6.5.13-5 the memory issues are gone, you are a star!!

[DSew20]

Ditto - I had same issue and downgraded to 6.5.13-5 and runs ok. I guess the issue is now when is it safe to upgrade again!?!

[heisenberg2980]

Unfortunately I was wrong and the issue is still there even with kernel 6.5.13-5. After a bit more investigation I discovered an easy way to check if the issue is there or not without having to wait until the container crashes, and it is as simple as executing the command TOP inside the LXC and then using SHIFT+M to order the processes by memory. If the process python3 is consuming more than 10% of the memory assigned to the LXC that means the issue is present as python3 should consume a maximum of 10% of the memory assigned to the LXC

[LiloBzH]

weird. limit memory does not work for me

docker stats | grep frigate
frigate - 1.041GiB / 4GiB

total : 8 Gib // limited of the container : 4 Gib

stay a 1 Gib / 1.5 Gib but memory usage increases

[LiloBzH]

Soneone test with last kernel 6.8.8 ? or same problem ?

[heisenberg2980]

According to #10921 (comment) the issue is still there in 6.8.8-4

[mazasb]

The latest beta version (v0.14.0-beta3) with 6.5.13-3-pve kernel seems be fine. No memory increase.

[kirsch33]

yup same kernel here and no issues

[heisenberg2980]

Well if the kernel is the same I wouldn´t expect any issues to be honest. The issue happens with the newer kernel (6.8) so we need to wait and see if this is fixed once they release a new kernel

[mazasb]

I had same issues (memory consumption) with 0.13 version and this kernel (6.5.13-3-pve). When i rollback to 0.12, the issue is gone.

[kirsch33]

according to @Redsandro in #5773 (comment), 6.8.8-2-pve seems to be working. i plan to try myself this weekend but was hoping someone else may have tried already?

[kirsch33]

I haven't experienced the memory leak issue at all running 6.8.8-4-pve for the past few weeks and Frigate version 0.14. However, I never had the issue to begin with. Based on the early replies in this thread I was worried that this was a widespread Proxmox kernel issue but that doesnt seem to be the case.

That all said, some users are still experiencing it per replies below so its likely only affecting certain setups. Not sure how it would be a Frigate problem at this point

[heisenberg2980]

@kirsch33 how much memory does your server have in total, and how much have you assigned to the LXC running Frigate as a docker container?

The issue seems to appear when the server has a lot of memory (32GB or more) and only a small portion of that is assigned to the LXC, due to the fact that the process python3 is not aware of the memory assigned to the LXC but it´s only aware of the total memory of the server, and it uses 10% of that which causes the issue. Can you run the command TOP inside the LXC running Frigate and check how much memory is your python3 process using?

[kirsch33]

I have Proxmox on an R720 with 64gb memory total. Frigate is installed via Docker within a Debian bookworm VM. The docker container runs in privileged mode.

The docker VM has 40gb memory assigned to it with ballooning disabled.

[heisenberg2980]

@kirsch33 that explain why you don't experience the issue, as you have most of your memory (40Gb out of 64Gb) assigned to the VM where Frigate is running, so even if python3 process consumes 10% of the 64Gb instead of 10% of the 40% it won´t matter in your case. Also I am not sure if this issue affects VMs in the same way that it affects LXCs, but if you run the TOP command inside the VM you should see if python3 is consuming 4Gb of memory (which would be correct) or 6.4Gb of memory (which means you have the issue).

[Redsandro]

@Redsandro How is your testing going?

Apologies. Due to renovations I have temporarily switched to old hardware running Frigate 0.13 using my old Google Coral Edge TPU.
I am planning to go back to the Proxmox + Intel NPU at some point in the future. If there are any more issues which I hadn't noticed before, I'm secretly hoping they will have been resolved by then.

[LiloBzH]

Frigate version : 0.13.2-6476f8a

I will test this version if you want

root@proxmox:~# lsmod | grep apex
apex                   28672  0
gasket                126976  1 apex
root@proxmox:~# uname -a
Linux proxmox 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux

root@proxmox:~# pveversion
pve-manager/8.2.4/faa83925c9641325 (running kernel: 6.8.8-2-pve)

Test in progress

[kirsch33]

awesome! let us know

[rocket59]

I have no memory issues with Frigate .13.2 on 6.8.4-3 which was the version installed when I upgraded from proxmox 7.4 to 8. Previously on 5.15 I had eventual memory spikes which crashed the entire container.

[lost-RD]

How much memory does the machine have and how much is allocated to the LXC?

[LiloBzH]

not sure if this is fixed.
we will see in 24/48 hours

[LiloBzH]

Sorry.

6.5 instead of 6.8

[SonarClouds]

I switched to kernel 5.10.0 on my 4460 and don't see any evidence of memory leaks after 6 days.
On the 9500T where I did a gpu pass through running a debian vm with kernel 6.10 the machine I can clearly see evidence of a memory leak as the swap is already full with no process using the swap and according to top / meminfo I have 2GB of free memoy.
Really weird behaviour. Anyone got an idea beside that it might be an issue with the igpu driver in more recent kernel versions?

[heisenberg2980]

Since reverting to 6.8.8-4 and implementing a 3Gb memory limit for frigate in docker-compose I have not had any further memory spike issues.

@aegjoyce have you tried the top command to see if your python3 process is using 10% of the memory assigned to the container or more? More details in my other comment #10921 (reply in thread)

[aegjoyce]

Have been monitoring closely and since implementing my 3GB limit (3.5GB total assigned to the container) I haven't had any more crashes.

Just checked and python3 is currently using 15.2% of my total memory.

[Floyd-Schramm]

@aegjoyce Would you mind trying to export a time slot over 30 min as realtime or timelapse_x25?

[heisenberg2980]

@aegjoyce when you say "15.2% of my total memory" do you mean 15.2% of the 3GB? I think even if you haven´t had a crash in a while that could be just luck. If the python3 process is not limited to 10% of the memory assigned to the container it can use a lot more memory anytime and crash the container.

In my case something that increases the memory usage of python3 a lot and makes Frigate to restart is just trying to open the Timeline View in home assistant, maybe you can try that as well:

After clicking the button and playing around with the timeline for some seconds the memory used by python3 process goes through the roof and Frigate restarts automatically due to hitting the memory limit assigned in the docker-compose file.

[Bodengriller]

Unfortunately, I am also affected.
Futro S740 with 16GB RAM
My unpreviled Docker LXC has 6GB memory.
Frigate v0.14 (but I had the same problem from time to time with V0.13)

Since my cameras throw a lot of errors, I first thought it was related to the logging. The following settings brought some improvement. (maybe just a coincidence?)

logger:
  default: info
  logs:
    ffmpeg.Parkplatz.detect: critical
    ffmpeg.Parkplatz.record: critical
    watchdog.Parkplatz: critical
    ffmpeg.Carport.detect: critical
    ffmpeg.Carport.record: critical
    watchdog.Carport: critical
    ffmpeg.Stellplatz.detect: critical
    ffmpeg.Stellplatz.record: critical
    watchdog.Stellplatz: critical

go2rtc:
  log:
    level: fatal

Until recently, my Proxmox was still running with kernel 6.5.13-6-pve
Now it runs with kernel 6.8.12-1-pve.
I have adjusted my docker-compose-file and limited frigate to 3GB memory. Hopefully it will be better

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # see calculation at the end of this file
    deploy:
      resources:
        limits:
          memory: 3G # Limit RAM-Memory for frigate-Container
    devices:
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral
      - /dev/dri/renderD128 # for intel hwaccel
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/frigate/:/config/
      - /media/frigate_storage:/media/frigate
      - /home/frigate/custom-start.sh:/custom-start.sh
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    networks:
      bridge-network:
        ipv4_address: 172.20.0.12
    dns:
      - 127.0.0.1
      - 192.168.178.1
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds
      - "8554:8554" # RTSP feeds
    environment:
      LIBVA_DRIVER_NAME: "i965"

networks:
  bridge-network:
    external: true
    name: my-bridge-network

[EDIT]
3GB-Limit for frigate seems to result in 0.3GB usage for Python3 (=10%)

[heisenberg2980]

@Bodengriller did the memory limit really work for you? I am on the same kernel (6.8.12-1-pve) and limiting frigate to 7G does not limit the memory used by python3 to 10% (0.7GB) at all, as just after restarting the container it´s already on 2.3GB:

It would be interesting to know if there is something else that is affecting this difference in behaviour. This is my config:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
#    mem_limit: 7g
    deploy:
      resources:
        limits:
          memory: 7G # Limit RAM-Memory for frigate-Container
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "500mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./db:/db
      - /mnt/Frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /root/docker-compose/frigate/cache
        tmpfs:
          size: 1g
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
    environment:
      FRIGATE_RTSP_PASSWORD: xxxxxxxxxxx
      LIBVA_DRIVER_NAME: "radeonsi"
      PLUS_API_KEY: xxxxxxxxxxx
    networks:
      default:
        ipv4_address: 172.77.0.13

networks:
  default:
    name: my-network
    external: true

[Bodengriller]

hmm...
Hmm that probably only worked for a short time - my Python process has also grown back to 0.5G

[bib2]

Is this problem only when i use frigate in docker inside a lxc on proxmox?

Is this a problem in a virtual machine also (debian lts vm with docker - inside proxmox)?

What about debian vm inside vmware? Is there the problem also?

I now use proxmox inside lxc on proxmox and my installation is unresponsibel after some time - i can see maximum memory used in proxmox container overview. After container restart it works for some time - until it happens again. Sometimes it happens every hour. So in this way frigate is unuseable at the moment...

[bib2]

By the way... I used the proxmox helper script from tteck to install frigate on my proxmox. But i can´t find anything with docker inside this lxc. Is this installed directly or using docker??? How can i access docker to change the config (mem ressources)?

[NickM-27]

The tteck script does not use docker

[bib2]

So maybe when i install it the recommended way by hand i could have some luck and it will work for me?

[Floyd-Schramm]

You can certainly try. Following this guide (https://www.youtube.com/watch?v=DmbFq5dMsFo&t=574s) I was able to setup frigate very quickly. In normal operation there is no problem with memory usage, I only get the issue when starting a longer export

I've also added the following to the docker-compose file, but I cannot really tell a difference:

deploy:
      resources:
        limits:
          memory: 4G # Limit RAM usage for Frigate

[Floyd-Schramm]

In my case, python3 does not stop at 10% memory usage. My setup: i5-8500T, 16GB RAM total (2x8GB), Docker lxc with only frigate installed, and 8GB of memory assigned. Running frigate, also for multiple days, is no problem, but every time I export anything longer than 10 min the memory leak seems to come into play.


After hitting export, the memory usage slowly rises until the frigate docker container is shut down. I've not been playing around with kernel versions, because I would like to not wreck my proxmox install ;) These kernels are currently in use:

[PilaScat]

I have the same problem with unraid, every day

[heisenberg2980]

Did you check if the python3 process is also the one causing the issue by using too much memory?

[PilaScat]

no is ffmpeg

[lost-RD]

Same problem here, Frigate in Docker in an LXC on Proxmox. Slowly eats up all RAM available, then when RAM is full, the CPU maxes out as it tries to shuffle memory around and then the LXC becomes totally unresponsive.

I can confirm that using psutil within the Frigate container to check memory stats shows the host's available memory, not the LXC's.

My LXC config has these lines to define memory:

...
memory: 8192
lxc.cgroup2.memory.max: 8192M
...

Further info here on that second line: lxc/lxc#4049

This is in the LXC:

root@frigate:~# cat /proc/meminfo 
MemTotal:        8388608 kB
...

Then, attaching to the Frigate container:

root@frigate:~# docker exec -it f2 /bin/bash
root@f2e39a70f6a9:/opt/frigate# cat /proc/meminfo 
MemTotal:       32722532 kB
...

So the issue is that Frigate is running in Docker which runs on the host, not actually within the container.

The simple solution is to add an env var that allows us to specify a hard limit for memory consumption.

[heisenberg2980]

@lost-RD actually TOP also shows the exact number as well, it´s under column RES. For instance, this is a screenshot where python3 was consuming 2.3Gb:

[lost-RD]

@heisenberg2980 Somehow I never looked at those columns. good to know! And you can press E to switch display modes.

[heisenberg2980]

@lost-RD I just realised your command returns a different value (5.4Gb) than the one given by TOP (3.4Gb), not sure why that would be (maybe one is physical memory and the other one includes virtual memory?) and what is the correct value:

Regarding TOP, you can just press SHIFT+M to get the list ordered by memory usage and you will likely get python3 in the first position

[Redsandro]

you can use --memory=8g --kernel-memory=8g to limit the memory, it works on unraid, I don't know if it works on proxmox

Where do you put this?

[NickM-27]

if you are referring to unraid then it is under the extra arguments section, otherwise it is just part of the docker cli command

[photomoose]

Same issue here. I'm running Frigate under Docker inside an LXC. After a few hours, the container memory is maxed out and it freezes. The drops in these charts are a result of me having to restart the LXC container.

As mentioned by @lost-RD, I can also confirm that the Docker container believes it has more memory than what has been allocated to the LXC.

Inside the LXC:

root@frigate:~# cat /proc/meminfo 
MemTotal:        8388608 kB

Inside the Frigate Docker container:

root@790d9940d3b3:/opt/frigate# cat /proc/meminfo 
MemTotal:       24368676 kB

[modem158]

I'm still working through this but it seems that most nights at midnight there is a large dump of memory. Please ignore the other drops as I have setup a cron job at 0300 and 1500 to restart the LXC to dump memory so Frigate stays stable. Anyone. know if the midnight dump is a Frigate thing or a Proxmox thing? If Frigate, could it do this more often?

[NickM-27]

by default frigate checks retention for recordings every hour and for events / snapshots every 5 minutes, this is almost certainly not something frigate itself is doing

[lost-RD]

It must be Frigate. None of my other LXCs on the system have this issue. I've just set up some logging to find out exactly which processes are the culprit, will report back tomorrow.

[NickM-27]

Most containers don't access as many components as Frigate does, Frigate is at a relatively unique crossroads of CPU, GPU, storage, network, and sometimes USB / PCIe as well.

Memory issues in the past have been linked to many different things, typically on the host such as an issue with the kernel, issue with the GPU driver, etc. In the vast majority of cases for users reporting issues like this over the past years it has been a different issue or cause for each of them. More information is helpful for sure, but this is a very complicated and in-depth issue as is explained in the answer selected for this discussion.

https://github.com/bloomberg/memray is a great tool to get specific information about what is using memory and how it is allocated. But as I said in this discussion, even if there are no egregious issues like some of the known-bad proxmox kernels, it usually just comes down to python simply keeping the memory and not releasing it despite the fact that the memory is not actually being used by any frigate processes due to python not having a correct understanding of its total memory available.

[heisenberg2980]

Guys, I´m not sure if I am missing anything here, but we already found what is causing this memory issue in this thread.

The reason for this issue is that when Frigate is running in a docker container inside an LXC, it is not aware of the memory assigned to the container, but rather the total memory of the system, so obviously as Frigate uses a lot of resources, not knowing how much memory is really available in that container causes Frigate to use more memory than it should.

I linked a very interesting article in this comment explaining the issue, and I mentioned in this comment an easy way for everyone to check if they have the issue, which is the python3 process consuming too much memory. So far everyone running Frigate in a docker container inside an LXC seems to have the issue, no matter what kernel version we are running.

This seems to be a huge issue affecting any process running in a container and not only Frigate. We found this issue in Frigate just because Frigate is a resource-hungry application that tries to use all available resources (which is the correct behaviour for an application like Frigate).

EDIT: sorry @NickM-27, I just realised you mentioned the same thing in the last sentence of your comment.

[TheDescender]

I appreciate you emphasizing the core issue. So we have a clear understanding of what is causing this issue, I'm curious what actions people have taken to attempt to solve this so far? Have people posted about this issue extensively on the ProxMox forums? What are your thoughts on how we could best solve this issue or get it solved?

[Morphy99]

Is this exclusive to Proxmox? I'm seeing this on a number of different systems all running Frigate 0.14 on HA OS and different brands of cameras

[TheDescender]

Ok, I agree we should focus on the Python3 issue.

So then what do you suggest? Wanna rename this issue topic to "Python3 memory issue" or something like that?
Or have people create a new topic for it?

As is this topics name is so generic, you're gonna probably get people posting different things without reading the whole thread.

[lost-RD]

I highly doubt this, since I can attest to the opposite. I've increased the core count and still had this issue come up. The problem has also always been that the memory maxes out, which in turn causes the cpu to max out.

Yes, more cores allowed me to delay the issue but it did eventually reach the maxed-memory state in the end.

[deviant77]

Is this exclusive to Proxmox?

I see the same problem on two different Synology servers.

The system is using 12 GB out of 19 GB:

$ free -h --giga
              total        used        free      shared  buff/cache   available
Mem:            19G         12G        246M        725M        7.4G        6.3G

The Frigate (0.15.0-0b71cfa) container has been running for 1 week and is using 10.38 GiB of the 19 GB (54%):

$ docker stats --no-stream frigate
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O   PIDS
a79ac896aef4   frigate   124.83%   10.38GiB / 19.39GiB   53.54%    2.34TB / 189GB   0B / 0B     0

The python3 -u -m frigate process is actively using 6.43 GB (33%) and 8.93 GB total:

$ sortbyfield="rss"; fsep="-o zzz:::zzz%% -o"; ps ax o user:16 $fsep pid $fsep pcpu $fsep pmem $fsep vsz $fsep rss $fsep tty $fsep stat $fsep lstart $fsep time:16 $fsep cmd  --sort -$sortbyfield | awk 'function setprefix(num){{n_suffix=1; while(num > 1000 && n_suffix < suffixes_len) {num /= 1024; n_suffix++;}; num=int(100*num)/100suffixes[n_suffix]}; return num} BEGIN{suffixes_len=split("kB MB GB TB PB EB ZB",suffixes);FS="zzz:::zzz%";} NR>1 {$5=setprefix($5);$6=setprefix($6); }{ printf "%-16s %6s %-5s %-5s %9s %9s %-8s %-8s %-25s %-18s %s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11;}' | head -n30 | cut -c -250
USER                PID %CPU  %MEM        VSZ       RSS TT       STAT                      STARTED              TIME   CMD
root              10621  8.3  33.1     8.93GB    6.43GB ?        Ssl      Tue Oct  8 16:23:43 2024          15:42:28   python3 -u -m frigate
root              11205  6.3   1.6     2.26GB  334.39MB ?        Sl       Tue Oct  8 16:23:51 2024          11:53:25   frigate.process:front1
root              11052  1.8   1.5     1.66GB  297.84MB ?        Sl       Tue Oct  8 16:23:51 2024          03:27:30   frigate.recording_manager
root              11094  4.5   1.2     3.15GB  242.07MB ?        Sl       Tue Oct  8 16:23:51 2024          08:37:56   frigate.output
root              11210  1.6   1.1     2.16GB  224.25MB ?        Sl       Tue Oct  8 16:23:52 2024          03:04:35   frigate.process:front2

There was a sudden increase in system memory usage from 46% to 60% (2.66 GB) between 14:00 and 15:00 on 11th October (3 days after starting Frigate) otherwise it's just a gradual rise:

[lachlan-centrifuge]

In the meantime, I'm managing the issue with Docker Compose deploy resource constraints: https://docs.docker.com/compose/compose-file/deploy/#resources

Set it to only allow Frigate access to 7.5GB with my LXC having 8GB allocated to it. Hasn't crashed yet, a few days in.

[TheDescender]

Yeah I've seen the same behavior with this setting for the past month. So I'd definitely recommend people to do this. Worst case scenario the docker container will crash due to lack of memory and reboot, so the lxc doesn't have the chance to claim exorbitant resources.

[luisrx7]

The same thing is happening on my side. After some random time, the Proxmox server becomes literally a bean.

here is the journalctl dump right from when it started acting up

[luisrx7]

After crashing a few times I added this to the compose file and it appears to reboot the container when it starts eating more RAM than the usual

    deploy:
      resources:
        reservations:
            cpus: '0.25'
            memory: 4G
          limits:
            cpus: '0.50'
            memory: 5G
            pids: 1

Here the stats before and after

[doodlebro]

I just hit this pretty badly on Proxmox, though it seems to happen when running the tteck Frigate LXC script, no docker necessary. I've been working on removing docker due to other memory/crashing issues related to USB, shm, and tmp/cache mounts.

Quick fix was a full system reboot, which brought back stability and stopped the memory leaks, for now. Kernel is 6.8.12-2

[doodlebro]

Eventually landed on the TrueNAS Scale app, very stable compared to proxmox/LXC/docker combos.

[GHJJ123]

unfortunately, I've been having the same issue with Frigate on a Docker LXC in Proxmox.

I've migrated it back to bare metal i3 with less specs for the stability unfortunately. I do like advantage of having it on Proxmox for the ease of backups and migration features, but that is negated if it cant stay stable within the LXC.

I'll wait until this python 3 memory issue is resolved in future kernel updates!

[bobloadmire]

Has there been any updates on this? Still having the issue

[PilaScat]

Probably the number one issue with frigate, every setup I do have this problem

[TheDescender]

I mean. This has worked for me for months now. LXC doesn't crash anymore. #10921 (reply in thread)

The issue is with the Python3 module in proxmox. I don't see how or what Frigate should do to fix this. Does the above mentioned work around not solve your problem?

[PilaScat]

The only fix for now is to set a limit to the container, it works, but its not clean

[nschimme]

I figured out how to reproduce the issue. It seems to be related to when my cameras have a poor WiFi connection and ffmpeg then eats a ton of RAM trying to connect to them as part of go2rtc restreaming. When I improved my camera's connection to the network the problem is avoided.

[SonarClouds]

Probably had the same issue. I removed my two wireless reolink and didn't have the sever out of memory issues as before.
But I am not a 100% sure. In my case my host (debian) even crashed due to out of memory issues (got errors on the terminal "cannot allocate cgroup" before it crashed). Anyway for now the ram usage seems quit stable.
What I don't get but I wouldn't tribute it to frigate but rather the hw decoder for the video is the memory leak.
I suspected in my case that there are some stream errors happening causing the hardware decoder to leak memory over time. I noticed that after a while my swap size grew and grew but I was not able to bring it down even by stopping all services and the system was very very slow and unresponsive.
Right now I have 26 days of uptime (frigate itself was updated and rebooted in between) but the memory of the host is stable now.
I will switch to proxmox with the new Hardware and am still not sure whether I should use LXC or gpu passthrough to a VM. On another system I look after I opted for the VM with passthrough as with LXC I encountered the same leaks crashing the Host as on my debian system. (i5-4460 and i5-9500T so no vaapi but no qsv. Same driver so I guess the same memory leak is happining there)
Did anyone else encounted these leaks?