[Support]: Seems to not delete files from shm #10019
Replies: 3 comments 8 replies
-
|
This is covered in the docs https://docs.frigate.video/troubleshooting/faqs#fatal-python-error-bus-error There's no reason the detect output args would change the shm handling as it is handled by frigate either way. |
Beta Was this translation helpful? Give feedback.
-
|
what is your docker compose file |
Beta Was this translation helpful? Give feedback.
-
|
I'm running it in kubernetes. Here's the deployment spec: The only part out of the ordinary from any other container I run is the dev/shm mount, but as mentioned it works perfectly except for these specific two cameras. I have a hunch it's a filename length problem? I'm going to rename the cameras to something shorter and test. |
Beta Was this translation helpful? Give feedback.
-
|
It's not the filename length nor the detect_args. I have another camera with a longer name and similar detect_args. Here's the [working] dahua with an even longer name: and here's the full config for the hikvision [that leaves files behind]: Also, the hikvision are the final two cameras defined in the config. I'll try to move them to a higher position and see if that changes anything. |
Beta Was this translation helpful? Give feedback.
-
|
Changing the order in the config didn't matter. It's still the same two cameras that fill the shm folder. The only other difference I see is that I'm resizing that video to 1920x1080 instead of 1280x960. Here's a partial directory listing to show what's happening: |
Beta Was this translation helpful? Give feedback.
-
|
Ok, my bad that I didn't see that the logs were filling up, as well. I'm getting a lot of this for that camera: I'm guessing there's something wrong in my setup and frigate discards the file for future processing, including not cleaning it up. What's odd is that everything is working despite those errors. The cameras look good, they capture events, etc. I'll go take a close look at the setup and see if I can figure out what's wrong with them. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I've found the problem @NickM-27 It appears that after a certain threshold of resolution * num_cameras, ffmpeg starts throwing errors. When ffmpeg throws an error during processing, Frigate doesn't clean up the temp files in /dev/shm, and that fills up the shm folder, which eventually crashes Frigate. One step at a time: The camera is 4000x3000 pixels (fisheye). The detect_args takes the 3000x4000 image and selects a small portion of it, flattens it out, and resizes it to 1280x960. If I have one instance of that camera, everything is fine. I've also found that two streams of 4000x3000 are fine alongside 5 streams at 1920x1080. If I add a 6th 1920x1080 stream, I get the same crashes/errors. It seems to be a function of the total pixels processed in detect_args. Normally, I would never connect Frigate to a 4000x3000 stream, but the goal here is to zoom way in on a 1280*960 section of the fisheye and extract a frame for the detection process. I can't start with a smaller 720x720 substream, zoom way in to a 96x96 box, and resize to 1280x960 with good results. This is where I'm stuck. I don't know enough about ffmpeg to understand why it can't process the additional streams even though memory and CPU are sufficient. I suspect it's something to do with the number of threads simultaneously accessing the VAAPI interface, but I'm way out of my depth now. The strangest part is that everything works great even though ffmpeg is throwing all those errors. The cameras look great, the events are detected, etc. The only artifact of the errors is that Frigate doesn't clean them up and they overrun the shm folder. |
Beta Was this translation helpful? Give feedback.
-
|
you could try running a stream or two without the preset-vaapi (by overriding hwaccel_args at the camera level) and see if it improves. Just looks like vaapi doesn't like the data it is getting. |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to close this out because my own problem is resolved. Here's the takeaway from this whole thread:
Nick - thanks for the support |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the problem you are having
My first indicator that something was wrong was the well known error in the log file:
Looking at my /dev/shm folder, I see MANY files from the cameras where I'm using a custom ffmpeg command. For example, here's the count of files using a normal camera without custom ffmpeg:
Here's the count of files for one of the cameras with a custom ffmpeg command:
Looks like the temp files for cameras with custom ffmpeg commands aren't getting cleaned up, causing the /dev/shm folder to run out of space:
Version
0.13.0-C35C7DA
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
{"cameras":{"camera_balcony":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.38,"capture_pid":558,"detection_enabled":1,"detection_fps":12.3,"ffmpeg_pid":564,"pid":518,"process_fps":0.33,"skipped_fps":0.0},"camera_casita":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.57,"capture_pid":568,"detection_enabled":1,"detection_fps":11.09,"ffmpeg_pid":574,"pid":522,"process_fps":0.51,"skipped_fps":0.0},"camera_driveway":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.19,"capture_pid":538,"detection_enabled":1,"detection_fps":21.64,"ffmpeg_pid":544,"pid":510,"process_fps":0.17,"skipped_fps":0.0},"camera_familyroom":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":4.1,"capture_pid":583,"detection_enabled":1,"detection_fps":5.86,"ffmpeg_pid":589,"pid":528,"process_fps":3.77,"skipped_fps":0.0},"camera_familyroom_pool":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":3.95,"capture_pid":578,"detection_enabled":1,"detection_fps":3.73,"ffmpeg_pid":585,"pid":526,"process_fps":3.51,"skipped_fps":0.0},"camera_front_door":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.0,"capture_pid":573,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":580,"pid":524,"process_fps":0.0,"skipped_fps":0.0},"camera_front_gate_driveway":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.0,"capture_pid":543,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":550,"pid":512,"process_fps":0.0,"skipped_fps":0.0},"camera_front_gate_walkway":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.38,"capture_pid":548,"detection_enabled":1,"detection_fps":16.8,"ffmpeg_pid":554,"pid":514,"process_fps":0.33,"skipped_fps":0.0},"camera_front_yard":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":1.14,"capture_pid":563,"detection_enabled":1,"detection_fps":11.81,"ffmpeg_pid":569,"pid":520,"process_fps":0.52,"skipped_fps":0.57},"camera_garage":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.76,"capture_pid":553,"detection_enabled":1,"detection_fps":12.27,"ffmpeg_pid":559,"pid":516,"process_fps":0.5,"skipped_fps":0.19},"camera_pool":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.94,"capture_pid":530,"detection_enabled":1,"detection_fps":21.27,"ffmpeg_pid":535,"pid":507,"process_fps":0.93,"skipped_fps":0.0},"camera_rear_patio":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":0.76,"capture_pid":534,"detection_enabled":1,"detection_fps":14.85,"ffmpeg_pid":539,"pid":508,"process_fps":0.53,"skipped_fps":0.19}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"14":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"18":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"23":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"40":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"41":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"79":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"80":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"88":{"cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"0.0","cpu_average":"0","mem":"0.2"},"96":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"97":{"cmdline":"python3 -u -m frigate","cpu":"1.9","cpu_average":"2","mem":"23.7"},"103":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.2"},"141":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"142":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"143":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"144":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"147":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"164":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"195":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"232":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"264":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.1"},"440":{"cmdline":"frigate.logger ","cpu":"0.0","cpu_average":"0","mem":"1.4"},"442":{"cmdline":"frigate.recording_manager","cpu":"0.8","cpu_average":"0","mem":"3.5"},"451":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(80)","cpu":"0.0","cpu_average":"0","mem":"0.1"},"452":{"cmdline":"frigate.detector.coral","cpu":"0.1","cpu_average":"0","mem":"2.0"},"454":{"cmdline":"frigate.output ","cpu":"0.3","cpu_average":"0","mem":"1.9"},"460":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"461":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"462":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"463":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x960 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"464":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x960 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"465":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"466":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"467":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"468":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"469":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"470":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"471":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"472":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1920x1080 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"507":{"cmdline":"frigate.process:camera_pool","cpu":"0.1","cpu_average":"0","mem":"2.5"},"508":{"cmdline":"frigate.process:camera_rear_patio","cpu":"0.1","cpu_average":"0","mem":"2.5"},"510":{"cmdline":"frigate.process:camera_driveway","cpu":"0.0","cpu_average":"0","mem":"2.5"},"512":{"cmdline":"frigate.process:camera_front_gate_driveway","cpu":"0.0","cpu_average":"0","mem":"2.4"},"514":{"cmdline":"frigate.process:camera_front_gate_walkway","cpu":"0.1","cpu_average":"0","mem":"2.5"},"516":{"cmdline":"frigate.process:camera_garage","cpu":"0.1","cpu_average":"0","mem":"2.5"},"518":{"cmdline":"frigate.process:camera_balcony","cpu":"0.1","cpu_average":"0","mem":"2.5"},"520":{"cmdline":"frigate.process:camera_front_yard","cpu":"0.0","cpu_average":"0","mem":"2.5"},"522":{"cmdline":"frigate.process:camera_casita","cpu":"0.1","cpu_average":"0","mem":"2.5"},"524":{"cmdline":"frigate.process:camera_front_door","cpu":"0.1","cpu_average":"0","mem":"2.4"},"526":{"cmdline":"frigate.process:camera_familyroom_pool","cpu":"0.0","cpu_average":"0","mem":"2.6"},"528":{"cmdline":"frigate.process:camera_familyroom","cpu":"0.0","cpu_average":"0","mem":"2.5"},"15037":{"cmdline":"/bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c \"/bin/bash\" /dev/null || exec /bin/bash) || exec /bin/sh","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15043":{"cmdline":"/bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c \"/bin/bash\" /dev/null || exec /bin/bash) || exec /bin/sh","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15044":{"cmdline":"/usr/bin/script -q -c /bin/bash /dev/null","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15045":{"cmdline":"sh -c /bin/bash","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15046":{"cmdline":"/bin/bash","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15085":{"cmdline":"/bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c \"/bin/bash\" /dev/null || exec /bin/bash) || exec /bin/sh","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15091":{"cmdline":"/bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c \"/bin/bash\" /dev/null || exec /bin/bash) || exec /bin/sh","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15092":{"cmdline":"/usr/bin/script -q -c /bin/bash /dev/null","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15093":{"cmdline":"sh -c /bin/bash","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15094":{"cmdline":"/bin/bash","cpu":"0.0","cpu_average":"0","mem":"1.0"},"15440":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":131.61,"detectors":{"coral":{"detection_start":0.0,"inference_speed":7.14,"pid":452}},"gpu_usages":{"intel-vaapi":{"gpu":"0.0%","mem":"-%"}},"processes":{"go2rtc":{"pid":88},"logger":{"pid":440},"recording":{"pid":442}},"service":{"last_updated":1708801691,"latest_version":"0.13.2","storage":{"/dev/shm":{"free":33.7,"mount_type":"tmpfs","total":488.3,"used":454.6},"/media/frigate/clips":{"free":21062813.9,"mount_type":"cifs","total":55465798.3,"used":34402984.4},"/media/frigate/recordings":{"free":21062813.9,"mount_type":"cifs","total":55465798.3,"used":34402984.4},"/tmp/cache":{"free":384903.4,"mount_type":"overlay","total":468766.1,"used":64492.2}},"temperatures":{},"uptime":30175,"version":"0.13.0-c35c7da"}}Operating system
Other
Install method
Docker Compose
Coral version
USB
Network connection
Wired
Camera make and model
dahua and hikvision
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions