[Support]: Frigate leaking GBs of memory with Proxmox

[stgraber]

Describe the problem you are having

Frigate has been leaking GBs of memory ever since upgrading from 0.12 to 0.13.2.

I used to run on 4GB of RAM which Frigate fully consumed in less than 1h, I then bumped the memory of my instance to 8GB of RAM and Frigate still managed to eat it all overnight.

1524288   493608  2.7  0.9 7393656 4941276 ?     Rsl  05:16  17:28          |   |   \_ python3 -u -m frigate

Above shows a resident memory usage of the main frigate process at over 4.9GB which combined with all the other support processes was taking all 8GB of RAM.

Doing a quick test over a 1h time period:

root@frigate01:~# grep VmRSS /proc/699/status ; sleep 1h ; grep VmRSS /proc/699/status
VmRSS:	  747184 kB
VmRSS:	  860048 kB

So frigate seems to be leaking memory at a rate of just under 2MB/minute here.

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  host: PRIVATE
  user: PRIVATE
  password: PRIVATE

ffmpeg:
  hwaccel_args: preset-vaapi

cameras:
  shf-camera01:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera01
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera02:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera02
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera03:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera03
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera04:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera04
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera05:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera05
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - bird
    record:
      events:
        objects: []

  shf-camera06:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera06
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera07:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera07
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera08:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera08
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - car
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera09:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera09
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2560
      height: 1920
    objects:
      track:
        - cat
        - person
    record:
      events:
        objects: []

  shf-camera10:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera10
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True

  shf-camera11:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera11
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True
      required_zones:
        - driveway
    record:
      events:
        required_zones:
          - driveway
    zones:
      driveway:
        coordinates: 815,1019,2304,1088,2304,1296,0,1296,0,1168,483,943

  shf-camera12:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera12
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - bird
        - cat
        - person
    snapshots:
      enabled: True

  shf-camera13:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera13
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - cat
        - car
        - person
    snapshots:
      enabled: True

  shf-camera14:
    ffmpeg:
      inputs:
        - path: rtmp://PRIVATE:1935/shf-camera14
          input_args: preset-rtmp-generic
          roles:
            - record
            - detect
    detect:
      width: 2304
      height: 1296
    objects:
      track:
        - bird
        - cat
        - person
    snapshots:
      enabled: True

record:
  enabled: True
  retain:
    days: 7
    mode: all
  events:
    retain:
      default: 10
      mode: active_objects
    post_capture: 0
    pre_capture: 1

detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track: []

snapshots:
  enabled: False
  bounding_box: True
  crop: true
  retain:
    default: 10

birdseye:
  enabled: True
  mode: continuous

live:
  height: 720
  quality: 5

Relevant log output

The process being completely out of memory resulted in no log entries at all for the affected time period (10h):

2024-04-09 05:38:48.358042876  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:36 -0400] "GET /api/shf-camera11/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.667999486  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera06/latest.jpg?h=444 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.668297252  2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera07/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 15:52:02.554579245  [INFO] Preparing Frigate...
2024-04-09 15:52:02.566412169  [INFO] Starting NGINX...
2024-04-09 15:52:02.663614853  [INFO] Preparing new go2rtc config...
2024-04-09 15:52:02.697756151  [INFO] Starting Frigate...
2024-04-09 15:52:03.570434492  [INFO] Starting go2rtc...
2024-04-09 15:52:03.682457180  11:52:03.682 INF go2rtc version 1.8.4 linux/amd64
2024-04-09 15:52:03.683346518  11:52:03.683 INF [api] listen addr=:1984
2024-04-09 15:52:03.683645144  11:52:03.683 INF [rtsp] listen addr=:8554
2024-04-09 15:52:03.684545352  11:52:03.684 INF [webrtc] listen addr=:8555

FFprobe output from your camera

Not relevant as not camera specific.

Frigate stats

No response

Operating system

Debian

Install method

Docker CLI

Coral version

USB

Network connection

Wired

Camera make and model

Mix of Reolink and bunch of other vendors (not relevant)

Any other information that may be helpful

Seems similar to #10649 but in my case the system was just preventing memory allocations rather than triggering the kernel out of memory killer.

[NickM-27]

A lot of time has been spent looking in to this. https://github.com/bloomberg/memray is a great tool for understanding where memory allocations are going.

In my usage it appears there is no memory leak, the heap size does not increase at all, python just does not release the memory after it has been garbage collected. I don't think there is any simple / immediate solution. https://www.reddit.com/r/learnpython/comments/17j1aff/rss_memory_inflates_leaks_while_heap_memory_size/?rdt=33549

Perhaps in your case it is something different, a memray output would show what is using memory

[stgraber]

Okay, the fact that there are reports of similar behavior in other python3 applications that similarly are running inside of resource constrained containers makes me want to look at a couple of options here, will report back in a few minutes (or hours if I need to do a full leak test).

[stgraber]

Basically, the other reporter mentioned running this on a Proxmox server, so with Frigate running (with or without Docker, not sure) inside of a resource constrained LXC container.
In my case, I'm running Frigate inside of a Docker container running inside of an Incus container which is similarly resource constrained.

The container I usually give 4GB of RAM out of the 500GB+ of the host system.
By default Docker doesn't use LXCFS which is a way to expose cgroup resource limits to containers.
That means that even though the environment is constrained to 4GB of RAM and that limit is clearly visible in /proc/meminfo and free -m, that same limit isn't visible INSIDE the Docker container that Frigate runs in.

Now if python3 decided to be funny with caching and allow for memory to grow up till some arbitrary value determined from the total memory it think it can use, then that'd explain the problem.

My first attempt at fixing this is simply by manually telling Docker to map LXCFS on /proc/meminfo, /proc/swaps, ... inside of the Frigate container. If python3 is looking at those files to figure out how much memory is available on the system, then that will work fine and it shouldn't get to a point where it runs the whole thing out of RAM anymore.

If that doesn't work, then it's possible that python3 makes use of the sysinfo system call instead to get memory information. If that's the case, then Incus also has a way to handle that through system call interception and emulation, so I can easily turn that one on. That would be bad news for Proxmox users though as I don't believe Proxmox has support for system call interception.

[stgraber]

Still appears to be consuming memory at an alarming rate. I'm also tracing it for good measure and see that something within the python3 process does call the sysinfo system call every minute or so.

I can't tell what the information is used for but it certainly seems likely that this is the culprit and that I'll have to turn on interception to fix this issue.

[stgraber]

Right, initial test shows 8MB/minute when running under a 4GB total limit.
I've not given it enough time to run fully out of memory but it's not showing any sign of slowing down despite total free memory being below 400MB by now.

I'll turn on the sysinfo interception and see if the behavior is different.

[NickM-27]

The every minute sys info call is most likely the stats which are presented via MQTT for the system page / integration

[stgraber]

Ah, that'd make sense, though I guess the same underlying logic within python may be used for setting up some kind of memory limit on process startup.

[stgraber]

And running with sysinfo interception, so far so good, process usage hovers around 485MB and seems to be actually releasing memory back to the OS following spikes. I'm still waiting for the results after keeping this going for an hour or so to make sure things are actually stable.

If they are, then we'll know what python3 is doing now and how to fix it, at least for anyone using Incus, those not running on a container platform capable of system call emulation will be out of luck unfortunately.

[stgraber]

Docker or Podman users will most likely run into the same issue if they specify a memory limit for their Frigate container as the python3 processes running inside of that container will not see the limit and therefore will plan their memory allocation based on the total amount of memory available on the system.

Now I don't think that's necessarily a very common setup and even if it was, as Frigate is likely to get the lion's share of the system's memory anyways, the risk of python3 hitting the issue is pretty small.

That's assuming that most Frigate users run on "normal" systems, Rpi4/Rpi5, Intel NUC and the like.

In my case I'm running into the worst version of this because my server is a 48-core AMD EPYC with 500GB of RAM. So Frigate not seeing the limit of 4GB causes it to assume it's got 500GB to work with!
So even if python3 had a conservative allocation of say, up to 10% of total system memory, that would still allow it to grow to 50GB in my case ;)

[NickM-27]

right, I am asking because I'd be curious to see how it behaves on my system which is an AMD 5700G Unraid NAS with 64GB of RAM but only 8GB allocated to frigate. I have never run out of memory but it does sometimes make its way up to 5GB or so

[stgraber]

@andrewhack can you comment on how much memory your Proxmox host has?

Your report shows you indeed have a 2GB cgroup limit on the container and that python3 consumed all of it causing the OOM killer to trigger, but the kernel trace doesn't show the total system memory you're working with.

I'm kinda hoping that your system has 32GB+ of RAM. If it does, then it's basically a repeat of what I'm seeing, which is going to be annoying for you given you're on Proxmox and can't just turn on sysinfo emulation, but would at least be good news for anyone running Frigate on lower specced systems as they'd hopefully not hit the issue.

[stgraber]

@NickM-27 it really looks to me like python3 is aiming for a sweet spot of around 10% of system memory given I've now limited it to 4GB and it's staying below 500MB of RAM.

So on your system with 64GB of RAM, I'd then expect the process to be in the 5-6GB range which would match up with what you're seeing. Now the issue is that if you were to double your system's RAM to 128GB, then Frigate would try to hit the 10-12GB RAM range and crash your container due to the 8GB limit.

[heisenberg2980]

@NickM-27 this article that @lost-RD shared below is really good for understanding the issue with memory limits inside a container: https://fabiokung.com/2014/03/13/memory-inside-linux-containers/

Now the question is, do we have any way to avoid this? is there any alternative to the python3 process that Frigate can use? or alternatively, does anyone know where we can open an issue so someone can fix the python3 process so it works properly inside a container?

[stgraber]

Right, so a couple hours later and memory usage is stable, so the issue is confirmed to be related to python3 now calling sysinfo which if that returns an amount of total memory higher than what's allowed for the container will cause python3 to start growing the process memory until it crashes.

[heisenberg2980]

@NickM-27 I guess this issue must be happening to anyone running Frigate in a Proxmox LXC with a small percentage of memory assigned to the container in relation to the total memory available in the Proxmox server. My understanding is the only solution available at the moment is going back to Frigate v0.12 (please correct me if I am wrong), are you aware of any idea/plan to solve this issue in the future?

[NickM-27]

frigate version has no effect on this issue. If you are not seeing it in frigate 0.12 but are seeing it in frigate 0.13 then you are seeing a different issue or a combination of issues

[heisenberg2980]

@NickM-27 as @stgraber mentioned in the description of this defect, in my case I also started having this memory issue after upgrading from Frigate v0.12 to v0.13, so I assumed python3 was being used in a different way in v0.12. But regarthless of the version this started, this seems like a really important issue affecting a lot of Frigate users, and we found the reason for the issue in this thread which is great, now the next question is can it be fixed?

[lost-RD]

@stgraber Did you find where in the codebase this is happening?

Might be best to mark this thread as unanswered until we clear up enough of the details to create an issue

[stgraber]

Well, in my case I tracked this down to python3 somehow using sysinfo() to figure out how much memory it has rather than rely on cgroup values or parsing /proc files which could use LXCFS for the correct value.

I'm running on Incus which supports sysinfo emulation specifically to deal with those kind of cases, so I turned it on and the problem went away, effectively confirming the hypothesis and fix.

Now if you're running using a container platform which doesn't support system call interception and emulation of sysinfo, then you're going to have a problem with any piece of software which relies on that system call to figure out system memory.

[LiloBzH]

hello

ok it's a problem of python3 and not frigate but we can't do anything ?

Usage of RAM is really a problem

[heisenberg2980]

@LiloBzH I´m not an expert on this at all, but I guess unless Frigate can replace python3 with something else (which I don´t know if it´s possible) the only other way for this issue to be fixed is when the python3 process or the sysinfo function are fixed so the python3 process is aware of the real memory assigned to the LXC and instead of the total memory available in the machine. Does anyone know which forum we should use to raise this issue? maybe the Proxmox forum?

[LiloBzH]

I reboot the LXC every 3 days to save RAM and SWAP :'(

[heisenberg2980]

@nautilus7 have you executed this command as @LiloBzH mentioned? what is the ouput:

proxmox-boot-tool kernel list

[nautilus7]

The first and only proxmox version I ever installed on my machine is 8.2, so I only got two different versions of kernel 6.8 installed.

[heisenberg2980]

@LiloBzH I can confirm after downgrading proxmox to kernel 6.5.13-5 the memory issues are gone, you are a star!!

[DSew20]

Ditto - I had same issue and downgraded to 6.5.13-5 and runs ok. I guess the issue is now when is it safe to upgrade again!?!

[heisenberg2980]

Unfortunately I was wrong and the issue is still there even with kernel 6.5.13-5. After a bit more investigation I discovered an easy way to check if the issue is there or not without having to wait until the container crashes, and it is as simple as executing the command TOP inside the LXC and then using SHIFT+M to order the processes by memory. If the process python3 is consuming more than 10% of the memory assigned to the LXC that means the issue is present as python3 should consume a maximum of 10% of the memory assigned to the LXC

[LiloBzH]

weird. limit memory does not work for me

docker stats | grep frigate
frigate - 1.041GiB / 4GiB

total : 8 Gib // limited of the container : 4 Gib

stay a 1 Gib / 1.5 Gib but memory usage increases

[LiloBzH]

Soneone test with last kernel 6.8.8 ? or same problem ?

[heisenberg2980]

According to #10921 (comment) the issue is still there in 6.8.8-4

[mazasb]

The latest beta version (v0.14.0-beta3) with 6.5.13-3-pve kernel seems be fine. No memory increase.

[kirsch33]

yup same kernel here and no issues

[heisenberg2980]

Well if the kernel is the same I wouldn´t expect any issues to be honest. The issue happens with the newer kernel (6.8) so we need to wait and see if this is fixed once they release a new kernel

[mazasb]

I had same issues (memory consumption) with 0.13 version and this kernel (6.5.13-3-pve). When i rollback to 0.12, the issue is gone.

[kirsch33]

according to @Redsandro in #5773 (comment), 6.8.8-2-pve seems to be working. i plan to try myself this weekend but was hoping someone else may have tried already?

[kirsch33]

I haven't experienced the memory leak issue at all running 6.8.8-4-pve for the past few weeks and Frigate version 0.14. However, I never had the issue to begin with. Based on the early replies in this thread I was worried that this was a widespread Proxmox kernel issue but that doesnt seem to be the case.

That all said, some users are still experiencing it per replies below so its likely only affecting certain setups. Not sure how it would be a Frigate problem at this point

[heisenberg2980]

@kirsch33 how much memory does your server have in total, and how much have you assigned to the LXC running Frigate as a docker container?

The issue seems to appear when the server has a lot of memory (32GB or more) and only a small portion of that is assigned to the LXC, due to the fact that the process python3 is not aware of the memory assigned to the LXC but it´s only aware of the total memory of the server, and it uses 10% of that which causes the issue. Can you run the command TOP inside the LXC running Frigate and check how much memory is your python3 process using?

[kirsch33]

I have Proxmox on an R720 with 64gb memory total. Frigate is installed via Docker within a Debian bookworm VM. The docker container runs in privileged mode.

The docker VM has 40gb memory assigned to it with ballooning disabled.

[heisenberg2980]

@kirsch33 that explain why you don't experience the issue, as you have most of your memory (40Gb out of 64Gb) assigned to the VM where Frigate is running, so even if python3 process consumes 10% of the 64Gb instead of 10% of the 40% it won´t matter in your case. Also I am not sure if this issue affects VMs in the same way that it affects LXCs, but if you run the TOP command inside the VM you should see if python3 is consuming 4Gb of memory (which would be correct) or 6.4Gb of memory (which means you have the issue).

[Redsandro]

@Redsandro How is your testing going?

Apologies. Due to renovations I have temporarily switched to old hardware running Frigate 0.13 using my old Google Coral Edge TPU.
I am planning to go back to the Proxmox + Intel NPU at some point in the future. If there are any more issues which I hadn't noticed before, I'm secretly hoping they will have been resolved by then.

[LiloBzH]

Frigate version : 0.13.2-6476f8a

I will test this version if you want

root@proxmox:~# lsmod | grep apex
apex                   28672  0
gasket                126976  1 apex
root@proxmox:~# uname -a
Linux proxmox 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux

root@proxmox:~# pveversion
pve-manager/8.2.4/faa83925c9641325 (running kernel: 6.8.8-2-pve)

Test in progress

[kirsch33]

awesome! let us know

[rocket59]

I have no memory issues with Frigate .13.2 on 6.8.4-3 which was the version installed when I upgraded from proxmox 7.4 to 8. Previously on 5.15 I had eventual memory spikes which crashed the entire container.

[lost-RD]

How much memory does the machine have and how much is allocated to the LXC?

[LiloBzH]

not sure if this is fixed.
we will see in 24/48 hours

[Morphy99]

Is this exclusive to Proxmox? I'm seeing this on a number of different systems all running Frigate 0.14 on HA OS and different brands of cameras

[lost-RD]

I highly doubt this, since I can attest to the opposite. I've increased the core count and still had this issue come up. The problem has also always been that the memory maxes out, which in turn causes the cpu to max out.

Yes, more cores allowed me to delay the issue but it did eventually reach the maxed-memory state in the end.

[deviant77]

Is this exclusive to Proxmox?

I see the same problem on two different Synology servers.

The system is using 12 GB out of 19 GB:

$ free -h --giga
              total        used        free      shared  buff/cache   available
Mem:            19G         12G        246M        725M        7.4G        6.3G

The Frigate (0.15.0-0b71cfa) container has been running for 1 week and is using 10.38 GiB of the 19 GB (54%):

$ docker stats --no-stream frigate
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O   PIDS
a79ac896aef4   frigate   124.83%   10.38GiB / 19.39GiB   53.54%    2.34TB / 189GB   0B / 0B     0

The python3 -u -m frigate process is actively using 6.43 GB (33%) and 8.93 GB total:

$ sortbyfield="rss"; fsep="-o zzz:::zzz%% -o"; ps ax o user:16 $fsep pid $fsep pcpu $fsep pmem $fsep vsz $fsep rss $fsep tty $fsep stat $fsep lstart $fsep time:16 $fsep cmd  --sort -$sortbyfield | awk 'function setprefix(num){{n_suffix=1; while(num > 1000 && n_suffix < suffixes_len) {num /= 1024; n_suffix++;}; num=int(100*num)/100suffixes[n_suffix]}; return num} BEGIN{suffixes_len=split("kB MB GB TB PB EB ZB",suffixes);FS="zzz:::zzz%";} NR>1 {$5=setprefix($5);$6=setprefix($6); }{ printf "%-16s %6s %-5s %-5s %9s %9s %-8s %-8s %-25s %-18s %s\n", $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11;}' | head -n30 | cut -c -250
USER                PID %CPU  %MEM        VSZ       RSS TT       STAT                      STARTED              TIME   CMD
root              10621  8.3  33.1     8.93GB    6.43GB ?        Ssl      Tue Oct  8 16:23:43 2024          15:42:28   python3 -u -m frigate
root              11205  6.3   1.6     2.26GB  334.39MB ?        Sl       Tue Oct  8 16:23:51 2024          11:53:25   frigate.process:front1
root              11052  1.8   1.5     1.66GB  297.84MB ?        Sl       Tue Oct  8 16:23:51 2024          03:27:30   frigate.recording_manager
root              11094  4.5   1.2     3.15GB  242.07MB ?        Sl       Tue Oct  8 16:23:51 2024          08:37:56   frigate.output
root              11210  1.6   1.1     2.16GB  224.25MB ?        Sl       Tue Oct  8 16:23:52 2024          03:04:35   frigate.process:front2

There was a sudden increase in system memory usage from 46% to 60% (2.66 GB) between 14:00 and 15:00 on 11th October (3 days after starting Frigate) otherwise it's just a gradual rise:

[lachlan-centrifuge]

In the meantime, I'm managing the issue with Docker Compose deploy resource constraints: https://docs.docker.com/compose/compose-file/deploy/#resources

Set it to only allow Frigate access to 7.5GB with my LXC having 8GB allocated to it. Hasn't crashed yet, a few days in.

[TheDescender]

Yeah I've seen the same behavior with this setting for the past month. So I'd definitely recommend people to do this. Worst case scenario the docker container will crash due to lack of memory and reboot, so the lxc doesn't have the chance to claim exorbitant resources.

[mateuszdrab]

I think this issue would affect any containerized instances on Docker since they all use cgroups for resource limiting.
I'm running 2 instances of Frigate on an RPI 5 with 8GB RAM and they regularly crash - I'll try the docker compose limits but I presume this will just offload the OOM restarting to Docker and will not actually resolve the python3 syscall issue.

[luisrx7]

The same thing is happening on my side. After some random time, the Proxmox server becomes literally a bean.

here is the journalctl dump right from when it started acting up

[luisrx7]

After crashing a few times I added this to the compose file and it appears to reboot the container when it starts eating more RAM than the usual

    deploy:
      resources:
        reservations:
            cpus: '0.25'
            memory: 4G
          limits:
            cpus: '0.50'
            memory: 5G
            pids: 1

Here the stats before and after

[doodlebro]

I just hit this pretty badly on Proxmox, though it seems to happen when running the tteck Frigate LXC script, no docker necessary. I've been working on removing docker due to other memory/crashing issues related to USB, shm, and tmp/cache mounts.

Quick fix was a full system reboot, which brought back stability and stopped the memory leaks, for now. Kernel is 6.8.12-2

[doodlebro]

Eventually landed on the TrueNAS Scale app, very stable compared to proxmox/LXC/docker combos.

[GHJJ123]

unfortunately, I've been having the same issue with Frigate on a Docker LXC in Proxmox.

I've migrated it back to bare metal i3 with less specs for the stability unfortunately. I do like advantage of having it on Proxmox for the ease of backups and migration features, but that is negated if it cant stay stable within the LXC.

I'll wait until this python 3 memory issue is resolved in future kernel updates!

[bobloadmire]

Has there been any updates on this? Still having the issue

[PilaScat]

Probably the number one issue with frigate, every setup I do have this problem

[TheDescender]

I mean. This has worked for me for months now. LXC doesn't crash anymore. #10921 (reply in thread)

The issue is with the Python3 module in proxmox. I don't see how or what Frigate should do to fix this. Does the above mentioned work around not solve your problem?

[PilaScat]

The only fix for now is to set a limit to the container, it works, but its not clean

[bobloadmire]

I mean. This has worked for me for months now. LXC doesn't crash anymore. #10921 (reply in thread)

The issue is with the Python3 module in proxmox. I don't see how or what Frigate should do to fix this. Does the above mentioned work around not solve your problem?

is there a way to update the pyhton3 module in prox to fix this correctly?

[nschimme]

I figured out how to reproduce the issue. It seems to be related to when my cameras have a poor WiFi connection and ffmpeg then eats a ton of RAM trying to connect to them as part of go2rtc restreaming. When I improved my camera's connection to the network the problem is avoided.

[SonarClouds]

Probably had the same issue. I removed my two wireless reolink and didn't have the sever out of memory issues as before.
But I am not a 100% sure. In my case my host (debian) even crashed due to out of memory issues (got errors on the terminal "cannot allocate cgroup" before it crashed). Anyway for now the ram usage seems quit stable.
What I don't get but I wouldn't tribute it to frigate but rather the hw decoder for the video is the memory leak.
I suspected in my case that there are some stream errors happening causing the hardware decoder to leak memory over time. I noticed that after a while my swap size grew and grew but I was not able to bring it down even by stopping all services and the system was very very slow and unresponsive.
Right now I have 26 days of uptime (frigate itself was updated and rebooted in between) but the memory of the host is stable now.
I will switch to proxmox with the new Hardware and am still not sure whether I should use LXC or gpu passthrough to a VM. On another system I look after I opted for the VM with passthrough as with LXC I encountered the same leaks crashing the Host as on my debian system. (i5-4460 and i5-9500T so no vaapi but no qsv. Same driver so I guess the same memory leak is happining there)
Did anyone else encounted these leaks?

[jaaacck]

Since adding Reolink cameras without the most stable WiFi I have been experiencing memory leaks as well. Previously with hardwired Hikvision Frigate was 100% stable with low memory usage. 1 week with Reolink in on spotty wifi and we're up to ~16G RAM usage on the container.

Running in a container on Unraid 6.12.13. Memory leaks are happening on 0.14 and also on 0.15 beta. I can see no disceranable memory differences between stable and beta.

[sredfern]

Just something for those of you waiting until this gets fixed. I've found putting this health check on the docker compose has been rock solid since. Just update cameras.front_porch.camera_fps

    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:5000/api/stats | jq -e '.cameras.front_porch.camera_fps > 0' || exit 1"]
      interval: 60s
      timeout: 10s
      retries: 3
      start_period: 90s

[RALFISBORN]

@sredfern Where exactly should I replace this code?
Use the Frigate addon under HA
Thanks

[RALFISBORN]

Just for understanding, since the discussion is considered answered:

I have the same problem with Proxmox V8.3.3

Is this now a Proxmox problem or one of Frigate?

Is a solution still being worked on here?

Thank you very much :)

[andrewhack]

The issue exist in 0.15 as well (proxmox 8.3 kernel: 6.8.12-8-pve )
LXC container with 3Gb RAM. 4x HD, 2x 4K cameras - 720p detector resolution.

In most cases the container is crashing during steaming from Home Assistant but not every time.
Normally the utilized memory is far from the limit. Even increasing the limit doesn't help.

My observation from past months - higher detector resolution, higher probability to hit the issue.

[346350.292306] Tasks state (memory values in pages): [346350.292737] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [346350.293160] [4185666] 0 4185666 41406 672 608 64 0 98304 0 0 systemd [346350.293637] [4185873] 0 4185873 1842 96 32 64 0 49152 0 0 agetty [346350.294087] [ 10351] 0 10351 1842 96 32 64 0 53248 0 0 agetty [346350.294538] [4185770] 0 4185770 25426 256 192 64 0 237568 0 -250 systemd-journal [346350.294996] [4185788] 104 4185788 4012 224 192 32 0 81920 0 0 systemd-network [346350.295454] [4185809] 0 4185809 24970 343 311 32 0 81920 0 0 dhclient [346350.295917] [4185818] 107 4185818 1975 128 96 32 0 57344 0 0 rpcbind [346350.296386] [4185824] 105 4185824 6025 983 983 0 0 90112 0 0 systemd-resolve [346350.296862] [4185844] 0 4185844 2152 64 32 32 0 57344 0 0 cron [346350.297329] [4185845] 106 4185845 2023 128 128 0 0 61440 0 -900 dbus-daemon [346350.297836] [4185967] 0 4185967 43591 11207 11207 0 0 327680 0 0 puppet [346350.298315] [4185848] 0 4185848 55199 320 288 32 0 98304 0 0 rsyslogd [346350.298798] [4185849] 0 4185849 3350 224 160 64 0 73728 0 0 systemd-logind [346350.299292] [4185866] 0 4185866 468753 4136 4072 64 0 331776 0 -999 containerd [346350.299791] [ 491412] 0 491412 309563 1272 1240 32 0 126976 0 -998 containerd-shim [346350.300302] [1120406] 0 1120406 456733 672 672 0 0 217088 0 -998 runc [346350.300813] [4185863] 0 4185863 123877 4864 4800 64 0 282624 0 0 ruby [346350.301329] [4185865] 112 4185865 478703 2211 2147 64 0 278528 0 0 zabbix_agent2 [346350.301899] [4185870] 0 4185870 1842 64 32 32 0 57344 0 0 agetty [346350.302440] [4185907] 0 4185907 613976 6733 6733 0 0 536576 0 -500 dockerd [346350.302967] [ 491314] 0 491314 4148 3216 3184 32 0 77824 0 -500 fuse-overlayfs [346350.303506] [ 491319] 0 491319 417809 384 352 32 0 176128 0 -500 docker-proxy [346350.304049] [ 491324] 0 491324 436242 256 224 32 0 180224 0 -500 docker-proxy [346350.304610] [ 491332] 0 491332 436242 288 256 32 0 184320 0 -500 docker-proxy [346350.305151] [ 491339] 0 491339 417809 256 224 32 0 172032 0 -500 docker-proxy [346350.305701] [ 491349] 0 491349 417809 224 224 0 0 172032 0 -500 docker-proxy [346350.306276] [ 491354] 0 491354 417809 256 224 32 0 167936 0 -500 docker-proxy [346350.306828] [ 491361] 0 491361 417809 288 224 64 0 180224 0 -500 docker-proxy [346350.307381] [ 491370] 0 491370 417809 256 224 32 0 172032 0 -500 docker-proxy [346350.307939] [4186205] 110 4186205 4652 246 246 0 0 73728 0 0 exim4 [346350.308506] [4188775] 0 4188775 3338 256 224 32 0 69632 0 -1000 sshd [346350.309075] [ 491431] 0 491431 52 0 0 0 0 28672 0 0 s6-svscan [346350.309649] [ 491473] 0 491473 53 0 0 0 0 28672 0 0 s6-supervise [346350.310260] [ 491476] 0 491476 50 0 0 0 0 24576 0 0 s6-linux-init-s [346350.310844] [ 491482] 0 491482 53 0 0 0 0 28672 0 0 s6-supervise [346350.311429] [ 491483] 0 491483 53 0 0 0 0 28672 0 0 s6-supervise [346350.312031] [ 491484] 0 491484 53 0 0 0 0 28672 0 0 s6-supervise [346350.312624] [ 491485] 0 491485 53 0 0 0 0 28672 0 0 s6-supervise [346350.313226] [ 491486] 0 491486 53 0 0 0 0 28672 0 0 s6-supervise [346350.313794] [ 491487] 0 491487 53 0 0 0 0 28672 0 0 s6-supervise [346350.314388] [ 491488] 0 491488 53 0 0 0 0 28672 0 0 s6-supervise [346350.314966] [ 491489] 0 491489 53 0 0 0 0 28672 0 0 s6-supervise [346350.315543] [ 491490] 0 491490 53 0 0 0 0 28672 0 0 s6-supervise [346350.316120] [ 491491] 0 491491 53 0 0 0 0 28672 0 0 s6-supervise [346350.316720] [ 491492] 0 491492 53 0 0 0 0 28672 0 0 s6-supervise [346350.317321] [ 491503] 0 491503 131 32 32 0 0 28672 0 0 s6-fdholderd [346350.317900] [ 491504] 0 491504 47 0 0 0 0 24576 0 0 s6-ipcserverd [346350.318496] [ 491544] 65534 491544 70 32 0 32 0 40960 0 0 s6-log [346350.319077] [ 491546] 65534 491546 70 64 0 64 0 40960 0 0 s6-log [346350.319659] [ 491547] 65534 491547 76 32 0 32 0 36864 0 0 s6-log [346350.320279] [ 491548] 65534 491548 69 0 0 0 0 40960 0 0 s6-log [346350.320864] [ 491556] 0 491556 326861 17551 17551 0 0 339968 0 0 go2rtc [346350.321456] [ 491564] 0 491564 936290 223511 223388 32 91 3026944 0 0 python3 [346350.322056] [ 491565] 0 491565 973 128 64 64 0 45056 0 0 bash [346350.322659] [ 491587] 0 491587 146224 1413 581 64 768 102400 0 0 nginx [346350.323286] [ 491617] 0 491617 212045 857 825 32 0 348160 0 0 nginx [346350.323896] [ 491618] 0 491618 212109 953 921 32 0 352256 0 0 nginx [346350.324512] [ 491651] 0 491651 212045 889 857 32 0 348160 0 0 nginx [346350.325141] [ 491652] 0 491652 211983 857 825 32 0 348160 0 0 nginx [346350.325774] [ 491686] 0 491686 146271 633 633 0 0 81920 0 0 nginx [346350.326404] [ 491764] 0 491764 973 160 64 96 0 49152 0 0 bash [346350.327031] [ 491907] 0 491907 317130 48664 48654 10 0 1011712 0 0 frigate.recordi [346350.327707] [ 491909] 0 491909 256701 45218 45218 0 0 778240 0 0 frigate.review_ [346350.328376] [ 491924] 0 491924 235971 46032 45924 10 98 757760 0 0 frigate.detecto [346350.329026] [ 491951] 0 491951 768327 48860 48814 42 4 1265664 0 0 frigate.output [346350.329690] [ 491956] 0 491956 39387 1088 1024 64 0 188416 0 0 ffmpeg [346350.330381] [ 491965] 0 491965 39387 1109 1045 64 0 176128 0 0 ffmpeg [346350.331010] [ 491975] 0 491975 39387 1110 1046 64 0 176128 0 0 ffmpeg [346350.331646] [ 491984] 0 491984 39387 1110 1046 64 0 184320 0 0 ffmpeg [346350.332281] [ 491996] 0 491996 39387 1085 1021 64 0 184320 0 0 ffmpeg [346350.332906] [ 492006] 0 492006 39387 1088 1024 64 0 184320 0 0 ffmpeg [346350.333544] [ 492013] 0 492013 39387 1120 1024 96 0 184320 0 0 ffmpeg [346350.334143] [ 492088] 0 492088 543416 62012 61651 42 319 1134592 0 0 frigate.process [346350.334743] [ 492089] 0 492089 542252 60871 60764 10 97 1126400 0 0 frigate.process [346350.335365] [ 492095] 0 492095 543194 61737 61312 42 383 1134592 0 0 frigate.process [346350.335968] [ 492101] 0 492101 544670 63355 62944 10 401 1146880 0 0 frigate.process [346350.336586] [ 492118] 0 492118 537848 56519 56348 74 97 1093632 0 0 frigate.process [346350.337190] [ 492123] 0 492123 543446 62161 61777 1 383 1134592 0 0 frigate.process [346350.337786] [ 492132] 0 492132 510102 61218 54778 42 6398 978944 0 0 frigate.capture [346350.338397] [ 492140] 0 492140 516862 60949 54796 74 6079 974848 0 0 frigate.capture [346350.339003] [ 492141] 0 492141 522587 61637 54823 42 6772 974848 0 0 frigate.capture [346350.339588] [ 492146] 0 492146 530023 61886 54791 10 7085 974848 0 0 frigate.capture [346350.340191] [ 492151] 0 492151 538156 60912 54769 74 6069 974848 0 0 frigate.capture [346350.340798] [ 492156] 0 492156 545930 61606 54796 42 6768 983040 0 0 frigate.capture [346350.341415] [ 492172] 0 492172 135822 1558 1494 64 0 274432 0 0 ffmpeg [346350.342005] [ 492174] 0 492174 135822 1493 1461 32 0 253952 0 0 ffmpeg [346350.342599] [ 492725] 0 492725 292577 13095 12263 32 800 540672 0 0 ffmpeg [346350.343172] [ 492727] 0 492727 293050 14012 13154 32 826 524288 0 0 ffmpeg [346350.343762] [ 790511] 0 790511 137046 3637 3573 64 0 286720 0 0 ffmpeg [346350.344392] [ 790522] 0 790522 292794 16653 15797 32 824 548864 0 0 ffmpeg [346350.345067] [ 846777] 0 846777 136596 2941 2877 64 0 274432 0 0 ffmpeg [346350.345650] [ 846779] 0 846779 292216 15108 14708 32 368 536576 0 0 ffmpeg [346350.346253] [ 847395] 0 847395 233720 11893 11039 32 822 450560 0 0 ffmpeg [346350.346856] [ 847397] 0 847397 233807 10370 9392 32 946 425984 0 0 ffmpeg [346350.347453] [1110468] 0 1110468 622 64 0 64 0 45056 0 0 sleep [346350.348044] [1120069] 0 1120069 622 0 0 0 0 49152 0 0 sleep [346350.348650] [1120115] 0 1120115 155816 8944 8912 32 0 417792 0 0 ffmpeg [346350.349258] [1120116] 0 1120116 157735 12254 12222 32 0 458752 0 0 ffmpeg [346350.349841] [1120167] 0 1120167 4354 128 96 32 0 69632 0 0 curl [346350.350450] [1120391] 0 1120391 38321 0 0 0 0 40960 0 0 ffmpeg [346350.351039] [1120392] 0 1120392 38321 0 0 0 0 36864 0 0 ffmpeg [346350.351643] [1120417] 0 1120417 38321 0 0 0 0 36864 0 0 ffmpeg [346350.352293] [1120451] 0 1120451 326861 17551 17551 0 0 339968 0 0 go2rtc [346350.352900] [1120455] 0 1120455 345943 388 388 0 0 131072 0 0 runc:[2:INIT] [346350.353534] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=mcollective.service,mems_allowed=0,oom_memcg=/lxc/2203,task_memcg=/lxc/2203/ns/system.slice/docker-bfe530eb8cfb071f116a1b4a483ca58b83970a1505af20cf5594f448afacfdd8.scope,task=python3,pid=491564,uid=0 [346350.354930] Memory cgroup out of memory: Killed process 491564 (python3) total-vm:3745160kB, anon-rss:893552kB, file-rss:128kB, shmem-rss:364kB, UID:0 pgtables:2956kB oom_score_adj:0

[kingp0dd]

I opened a similar discussion (#17188 (reply in thread) ) before reading this thread. Has anyone figured this one out yet?

In my case, I am not using LXC but a VM. And python is correctly seeing the VM's total RAM and not the Proxmox host RAM.

Findings:

I tried to dig deeper, please let me know if some of my assumptions are not accurate. I read #10921 and it seems that the consensus is:

• If Frigate is running inside an LXC, python will check the total RAM of the Proxmox host instead of the LXC
• Python will consume 10% of the total RAM (in this case, it gets total RAM of Proxmox host instead of the LXC)
• Resulting to python consuming more than the LXC assigned RAM

In my case:

• Frigate is running inside a VM instead of an LXC
• meminfo inside Frigate's bash shows the correct VM assigned RAM:

inside Frigate container:
$ docker exec -it frigate cat /proc/meminfo | grep Total
MemTotal:       21230128 kB
SwapTotal:        999420 kB
VmallocTotal:   34359738367 kB
HugePages_Total:       0

inside VM:
$ cat /proc/meminfo  | grep Total
MemTotal:       21230128 kB
SwapTotal:        999420 kB
VmallocTotal:   34359738367 kB
HugePages_Total:       0

• VM RAM is set at 20GB, 10% of which is just 2GB
• Frigate on boot consumes 4GB already, and is steadily rising up to 6.5GB after 8 days

[jbcortes]

I'm running into the same issue. I'm also using a VM (based on Ubuntu 22.04). Still haven't found a solution.

[lost-RD]

Have you tried using deploy limits?

    deploy:
      resources:
        limits:
          memory: 1000M

[kingp0dd]

Have you tried using deploy limits?

    deploy:
      resources:
        limits:
          memory: 1000M

i tried that, but if it reaches near that limit, docker will kill the container through OOM and then Frigate will restart automatically. this will go unnoticed to the user since frigate is restarting by itself. the issue is, during those restarts, frigate will miss a few minutes of motion detection/recording. i only found out when i looked at grafana graphs wherein the RAM usage goes up and down, and i confirmed through journalctl that the container is indeed being killed and frigate is restarting

[Charlielx]

It took me ages to figure out that this was the problem since I had had a somewhat similar issue with the VS Code Add-on recently and just immediately assumed it was that. I am running into this exact same issue though. Hoping there's an actual solution because it really sucks to lose any recording time at all. Not really sure what even happened, was working fine up until the last week or 2, haven't changed any of my configs in the past 6+ months. Very strange

[mdkeil]

i tried that, but if it reaches near that limit, docker will kill the container through OOM and then Frigate will restart automatically. this will go unnoticed to the user since frigate is restarting by itself. the issue is, during those restarts, frigate will miss a few minutes of motion detection/recording. i only found out when i looked at grafana graphs wherein the RAM usage goes up and down, and i confirmed through journalctl that the container is indeed being killed and frigate is restarting

Same on my configuration.. I deployed the container via docker compose:

deploy:
      resources:
        limits:
          memory: 6144M
        reservations:
          memory: 5632M

When the container reached the limits, mostly docker kills/restarts the container..

meminfo from inside the docker container shows the informations from the host machine. In my opinion, it is not only a proxmos problem

docker exec -it ae26 cat /proc/meminfo | grep Total
MemTotal:       32784024 kB
SwapTotal:      19530748 kB

[el-bakkali]

wow, i was going crazy thinking my hardware is overheating or hardware fault
lol, adjusting my pwm fan
checking my syslog messages and couldnt find anything specific
Glad i found this thread, im having same problem, im using lxc container with pve community script and runnning pve 8.4.1

[lucaspdude]

I'm running on a proxmox VM and experiencing the same issue?

Did anyone figure out a way to solve this?

[kingp0dd]

Also proxmox and vm here

#17188 (reply in thread)

[tsaG1337]

I it seems its also leaking here, but I am using Truenas

[wallacebrf]

I it seems its also leaking here, but I am using Truenas

dang, 38.57GB of ram used. mine is at 4.24 GB after 19 hours since last restart. currently have docker set to limit memory to 8GB.

i am thinking of making a script that will restart frigate every 24 hours around like 1 or 2 AM when not much is going on

edit: i am also on TrueNAS with 125.5GB of total RAM so i am also thinking of limiting the container to 14GB (10% of 125.5 = 12.5) which should be more then enough for the container to run and not crash.

[doodlebro]

Daily reboots keep things pretty stable. I occasionally have a crash at 21 hours though. 12GB RAM for the container.

[grahamgraham]

I'd like to share my experience. I am using 14.1 on Proxmox 8.2.7.

I was also having this experience, running two Reolink hardwired cameras and one Wyze wireless camera (which will get replaced... eventually).
I am using the Proxmox Community script for Frigate, so I am not running in Docker.

This thread mentioned this could be due to python3 taking up too much memory, then it freaks out. They hypothesized this could be due to the amount of memory allotted, or it could be related to the streaming method used. Originally, I had 6GB of RAM allotted, then 8, and still had issues. Finally, I increased my RAM to 10GB. Python3 is sitting at 9% memory usage now.

Additionally, I believe another thread mentioned that changing preset-rtsp-restream to preset-rtsp-generic fixed it for them, so I did that same thing.

I only have 3 days uptime at this moment, but I do not see any huge spikes in memory usage, and I was getting daily freezes before.

Another poster said that v15 still had issues. So this might be more than just "what version are you on".

Here's my config, just in case.

mqtt:
  enabled: false

go2rtc:
  streams:
    kidCamera:
      - rtsp://admin:[email protected]:554/h264Preview_01_main
    kidCamera_sub:
      - rtsp://admin:[email protected]:554/h264Preview_01_sub
    drivewayCamera:
      - rtsp://admin:[email protected]:554/h264Preview_01_main
    drivewayCamera_sub:
      - rtsp://admin:[email protected]:554/h264Preview_01_sub

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128

cameras:
  drivewayCamera:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://192.168.68.202:8554/drivewayCamera
          input_args: preset-rtsp-generic
          roles:
            - record
        - path: rtsp://192.168.68.202:8554/drivewayCamera_sub
          input_args: preset-rtsp-generic
          roles:
            - detect
    detect:
      width: 1536
      height: 576
      fps: 20
    motion:
      mask:
        - 0.562,0.266,0.604,0.175,0.721,0.208,0.729,0.325
        - 0.742,0.847,0.692,0.314,0.321,0.262,0.193,0.861
        - 0.769,0.299,0.805,0.326,0.803,0.273,0.778,0.252
        - 0.812,0.387,0.908,0.529,0.936,0.46,0.846,0.354
        - 0.033,0.543,0.099,0.464,0.077,0.363,0,0.431
      threshold: 41
      contour_area: 10
      improve_contrast: 'true'
    objects:
      filters:
        car:
          mask: []
  mancaveCamera:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://admin:[email protected]/live
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      width: 1280
      height: 720
      fps: 5

    motion:
      mask:
        - 980,601,1096,720,1280,553,1184,430
  fosterCamera:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
        - path: rtsp://admin:[email protected]/live
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      width: 1280
      height: 720
      fps: 5
  kidCamera:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp://192.168.68.202:8554/kidCamera
          input_args: preset-rtsp-generic
          roles:
            - record
        - path: rtsp://192.168.68.202:8554/kidCamera_sub
          input_args: preset-rtsp-generic
          roles:
            - detect
    detect:
      width: 1280
      height: 720
      fps: 5

detectors:
  coral1:
    type: edgetpu
    device: pci
model:
  path: plus://4a26b3e105180e8419b69ce8a7804ac2

birdseye:
  enabled: true
  mode: continuous

record:
  enabled: true
  retain:
    days: 12
    mode: all
  events:
    retain:
      default: 30
      mode: active_objects
snapshots:
  enabled: true
  timestamp: true

objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - car
    - bicycle
    - motorcycle
    - bus
    - bird
    - cat
    - dog
    - horse
    - bear
  filters:
    dog:
      min_score: .7
      threshold: .9
    cat:
      min_score: .65
      threshold: .8
    face:
      min_score: .7
    package:
      min_score: .65
      threshold: .9
    license_plate:
      min_score: .6
    amazon:
      min_score: .75
    ups:
      min_score: .75
    fedex:
      min_score: .75
    person:
      min_score: .65
      threshold: .85
    car:
      min_score: .65
      threshold: .85
      mask:
        - 329,452,501,120,1101,151,1235,512
motion:
  mask:
    - 864,153,919,115,1104,135,1119,187
    - 1140,488,836,164,623,165,297,496
    - 1181,172,1236,188,1233,157,1195,145

version: 0.14

[grahamgraham]

I’ve definitely considered daily reboots, but that feels more like a bandaid than a cure!On Jun 27, 2025, at 11:12 AM, Jess Boerner ***@***.***> wrote: Daily reboots keep things pretty stable. I occasionally have a crash at 21 hours though. 12GB RAM for the container. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[bobloadmire]

A bandaid is better than bleeding out every 48hrs

[wallacebrf]

I wrote a script that monitors the docker memory usage of the container. I have mine set to limit the container to 8GB.

The script will restart frigate if the memory usage is above 6GB and the time is after 2:00 AM.

That way I can have it restart and control it so it.onoy restarts when needed and restarts at a time of day when little is happening