0.15.0 Beta 4

[blakeblackshear]

Beta Documentation: https://deploy-preview-13787--frigate-docs.netlify.app/

Images

• ghcr.io/blakeblackshear/frigate:0.15.0-beta4
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-standard-arm64
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-tensorrt
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-tensorrt-jp4
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-tensorrt-jp5
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-rk
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-rocm
• ghcr.io/blakeblackshear/frigate:0.15.0-beta4-h8l

Changes since Beta 3

• Fix a bug with SHM frame names that prevented cameras from starting in some cases
• Ensure pre_capture and post_capture work correctly with refactored alerts/detections record config
• Fix a UI bug that caused an infinite loop when navigating to History from Explore
• Add the ability to specify tls_insecure for onvif cameras
• Catch onvif error when running an operation the camera doesn't support
• Catch OS error when checking storage utilization
• Fix a bug that caused identical thumbnails to be sent to a GenAI provider under certain circumstances
• Improve handling of error conditions with Ollama and snapshot regeneration
• Don't crop by region for GenAI snapshot for manual events
• Ensure go2rtc credentials are cleaned before returning the config
• Optimize HTTP requests with connection pooling for DeepStack detector
• Cap storage bandwidth at a logical maximum for storage cleanup
• Standardize handling of config files
• Return 401 for login failures
• Other minor UI fixes and tweaks
• Documentation updates

Major Changes for 0.15.0

Breaking Changes

There are several breaking changes in this release, Frigate will attempt to update the configuration automatically. In some cases manual changes may be required. It is always recommended to back up your current config and database before upgrading:

1. Simply copy your current config file to a new location
2. Stop Frigate and make a copy of the frigate.db file

• Rewritten SHM system
  • The SHM system has been rewritten to be more stable. This means that a large % of shared memory (SHM) will be used by default but the total usage will not fluctuate, and Frigate will not fail from SHM overflowing. However, this also means that having the correct SHM size for your cameras, detector, and hardware is more important to having a smooth experience with Frigate. A new calculation is done on startup, so if your shm_size is too low then a warning will be printed in the log stating that it needs to be increased.
• Recording configuration changed
  • The record config has been refactored to allow for direct control of how long alerts and detections are retained. These values will be automatically populated from your current config, but you may want to adjust the values that are set after updating. See the updated docs here and ensure your updated config retains the footage you want it to.
• ffmpeg updates
  • ffmpeg has been updated to version 7. Some Intel users may need to change what hwaccel preset they are using (preset-vaapi may need to now be preset-intel-qsv-h264 or preset-intel-qsv-h265) if camera feeds are not functioning correctly after upgrading. This may need to be adjusted on a per-camera basis. If a qsv preset is not working properly, you may still need to use a preset-vaapi or revert to the previous ffmpeg version as described below.
  • The new version provides many improvements, but the old version can still be run if needed by specifying path: "5.0" in your ffmpeg: config entry. For example:

    ffmpeg:
      path: "5.0"
    

  • ffmpeg presets have been updated. It is always recommended to use ffmpeg presets rather than manual arguments.
  • ffmpeg is no longer part of $PATH. In most cases this is handled automatically.
    • go2rtc exec streams will need to add the full path for ffmpeg which in most cases will be /usr/lib/ffmpeg/7.0/bin/ffmpeg.
    • If the go2rtc config has bin: ffmpeg defined, it needs to be removed.
• Submitting to Frigate+
  • The Frigate+ view in the UI has been replaced by the new Explore view. See the notes below.

Explore

The new Explore pane in Frigate 0.15 makes it easy to explore every object tracked by Frigate. It offers a variety of filters and supports keyword and phrase-based text search, searching for similar images, and searching through descriptive text generated by AI models.

The default Explore pane shows a summary of your most recent tracked objects organized by label. Clicking the small arrow icon at the end of the list will bring you to an infinitely scrolling grid view. The grid view can also be set as the default by changing the view type from the Settings button in the top right corner of the pane.

The Explore pane also serves as the new way to submit images to Frigate+. Filters can be applied to only display tracked objects with snapshots that have not been submitted to Frigate+. The left/right arrow keys on the keyboard allow quick navigation between tracked object snapshots when looking at the Tracked Object Details pane from the grid view.

AI/ML Search

Frigate 0.15 introduces two powerful search features: Semantic Search and GenAI Search. Semantic Search can be enabled on its own, while GenAI Search works in addition to Semantic Search.

Semantic Search

Semantic Search uses a CLIP model to generate embeddings (numerical representations of images) for the thumbnails of your tracked objects, enabling searches based on text descriptions or visual similarity. This is all done locally.

For instance, if Frigate detects and tracks a car, you can use similarity search to see other instances where Frigate detected and tracked that same car. You can also quickly search your tracked objects using an "image caption" approach. Searching for "red car driving on a residential street" or "person in a blue shirt walking on the sidewalk at dawn" or even "a person wearing a black t-shirt with the word 'SPORT' on it" will produce some stunning results.

Semantic Search works by running an AI model locally on your system. Small or underpowered systems like a Raspberry Pi will not run Semantic Search reliably or at all. A dedicated GPU and 16GB of RAM is recommended for best performance.

See the Semantic Search docs for system requirements, setup instructions, and usage tips.

Generative AI

GenAI Search employs generative AI models to create descriptive text for the thumbnails of your tracked objects, which are stored in the Frigate database to enhance future searches. Supported providers include Google Gemini, Ollama, and OpenAI, so you can choose whether you want to send data to the cloud or use a locally hosted provider.

See the GenAI docs for setup instructions and use case suggestions.

Improved Tools for Debugging

Review Item Details Pane

A new Review Item Details pane can be viewed by clicking / tapping on the gray chip on a review item in the Review pane. This shows more information about the review item as well as thumbnails or snapshots for individual objects (if enabled). The pane also provides links to share the review item, download it, submit images to Frigate+, view object lifecycles, and more.

Object Lifecycle Pane

The Recordings Timeline from Frigate 0.13 has been improved upon and returns to 0.15 as the Object Lifecycle, viewable in the Review Details pane as well as the new Explore page. The new pane shows the significant moments during the object's lifecycle: when it was first seen, when it entered a zone, became stationary, etc. It also provides information about the object's area and size ratio to assist in configuring Frigate to tune out false positives.

Native Notifications

Frigate now supports notifications using the WebPush protocol. This allows Frigate to deliver notifications to devices that have registered to receive notifications in the Frigate settings, delivering them in a timely and secure manner. Currently, notifications will be delivered for all review items marked as alerts. More options for native notifications will be supported in the future.

See the notifications docs.

New Object Detectors

ONNX

ONNX is an open model standard which allows for a single model format that can run on different types of GPUs. The default, tensorrt, and rocm Frigate build variants include GPU support for efficient object detection via ONNX models, simplifying configuration and support more models. There are no default included ONNX models for object detection.

AMD MiGraphX

Support has been added for AMD GPUs via ROCm and MiGraphX. Currently there is no default included model for this detector.

Hailo-8

Support has been added for the Hailo8 and Hailo-8L hardware for object detection on both arm64 and amd64 platforms.

Other UI Changes

• Rework the Storage metrics pane to be more clear how much storage each camera is using
• Camera stream info button has been added to the camera metrics page allowing easy access to see camera codec information
• Add button for downloading the full set of logs
• The timeline handlebar will now automatically scroll while viewing recordings
• Review items and exports can now be shared via your browser's sharing pane or via direct link url. A secure context is required for the browser/OS sharing pane to function
• Add the ability to rename camera groups
• Add middle mouse button support in the Review pane to open full recordings view in a new tab
• Add Nvidia GPU encode and decode stats to system metrics
• New desktop keyboard shortcuts. In addition to the numerous keyboard shortcuts available with 0.14, 0.15 adds several more. In Live view, the numeric keys (1-9) will move PTZ cameras to presets. In Review, Page Up and Page Down will scroll through review items
• Accessibility updates for users with screen readers

Other Backend Changes

• FFMPEG updated to 7.0
• OpenVINO updated to 2024.3
• Use FastAPI instead of Flask for the Frigate API
• Add service manager infrastructure
• Install onevpl tools and update drivers for Intel
• If record is disabled for a camera then that camera will not save frames to the preview during that time
• Fix stalled connections to certain onvif cameras
• Fix autotracking absolute zoom interpolation logic
• TensorRT models are no longer generated by default, the ENV var must be set

---

This discussion was created from the release 0.15.0 Beta 4.

[PablaV]

Hmm updated and all my cameras stopped working that used the rtsp URL rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0

Reverted back to Beta 3 and all back to normal. NVR was logs were showing incorrect authentication

[tubalainen]

Hmm updated and all my cameras stopped working that used the rtsp URL rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0

Reverted back to Beta 3 and all back to normal. NVR was logs were showing incorrect authentication

Huh! Strange. Did you "let it settle" for 10 minutes? Sometimes it takes minutes for ffmpeg to sync up. Especially if you use WIFI cams (that can be glitchy) ive noticed.
Can you provide the ffmpeg log?

[tubalainen]

Thanks for beta4!

After upgrading I sorted through the false positives and noticed that it did not go trough to the frigate+ site.
Checked the logs and noticed that my API key isn't valid anymore. Is this a coincidence or a breaking change of beta4?

2024-12-23 08:57:45.618352051  192.168.xxx.139 - - [23/Dec/2024:08:57:45 +0100] "GET /api/events/1734800977.255047-n037aa/snapshot.jpg HTTP/1.1" 200 236745 "http://myhost:port/explore" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-"
2024-12-23 08:57:47.796960606  192.168.xxx.139 - - [23/Dec/2024:08:57:47 +0100] "PUT /api/events/1734800977.255047-n037aa/false_positive HTTP/1.1" 400 51 "http://myhost:port/explore" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-"
2024-12-23 08:57:47.797751513  [2024-12-23 08:57:47] frigate.api.event              ERROR   : Unable to refresh API token
2024-12-23 08:57:47.797758760  Traceback (most recent call last):
2024-12-23 08:57:47.797761777    File "/opt/frigate/frigate/api/event.py", line 755, in send_to_plus
2024-12-23 08:57:47.797764475      plus_id = request.app.frigate_config.plus_api.upload_image(image, event.camera)
2024-12-23 08:57:47.797766953    File "/opt/frigate/frigate/plus.py", line 105, in upload_image
2024-12-23 08:57:47.797769098      r = self._get("image/signed_urls")
2024-12-23 08:57:47.797771115    File "/opt/frigate/frigate/plus.py", line 84, in _get
2024-12-23 08:57:47.797773232      f"{self.host}/v1/{path}", headers=self._get_authorization_header()
2024-12-23 08:57:47.797775442    File "/opt/frigate/frigate/plus.py", line 79, in _get_authorization_header
2024-12-23 08:57:47.797777279      self._refresh_token_if_needed()
2024-12-23 08:57:47.797779517    File "/opt/frigate/frigate/plus.py", line 75, in _refresh_token_if_needed
2024-12-23 08:57:47.797781449      raise Exception("Unable to refresh API token")
2024-12-23 08:57:47.797783237  Exception: Unable to refresh API token

Renewed the API key and it all worked fine.

Merry xmas!

[Popoff-fr]

Everything is ok for me !

TY for the job guys !

Merry Xmas !!!

[easez88]

mqtt no longer works for me. If I go back to stable it starts working again. I'm getting this in the logs.

2024-12-23 12:59:11.324730966  [2024-12-23 12:59:11] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-12-23 12:59:11.894240437  [2024-12-23 12:59:11] frigate.comms.mqtt             ERROR   : MQTT disconnected
2024-12-23 12:59:12.567834659  [2024-12-23 12:59:12] frigate.comms.mqtt             DEBUG   : Unable to publish to model_state: client is not connected
2024-12-23 12:59:12.568634722  [2024-12-23 12:59:12] frigate.comms.mqtt             DEBUG   : Unable to publish to model_state: client is not connected
2024-12-23 12:59:12.903348978  [2024-12-23 12:59:12] frigate.comms.mqtt             DEBUG   : MQTT connected
2024-12-23 12:59:13.901394316  [2024-12-23 12:59:13] frigate.comms.mqtt             ERROR   : MQTT disconnected
2024-12-23 12:59:14.904174521  [2024-12-23 12:59:14] frigate.comms.mqtt             DEBUG   : MQTT connected
2024-12-23 12:59:15.907771078  [2024-12-23 12:59:15] frigate.comms.mqtt             ERROR   : MQTT disconnected
2024-12-23 12:59:16.911208434  [2024-12-23 12:59:16] frigate.comms.mqtt             DEBUG   : MQTT connected
2024-12-23 12:59:17.915157130  [2024-12-23 12:59:17] frigate.comms.mqtt             ERROR   : MQTT disconnected

[NickM-27]

are you running multiple instances of frigate at the same time? That is usually the isue

[ElectronicBattle]

all OK here (via docker compose):

[thelovemyth]

Nice :) what is your hardware ?

[ElectronicBattle]

It's a thirteen year old PC (dual Xeon with a lot of RAM, outperformed by a current i3 or i5 in most instances!) with a modern Radeon RX580 graphics card and I have an m.2 to PCI adaptor which mounts a Coral accelerator. I run this manually day by day to play with and learn from.

(I have another "24/7" Frigate instance which is permanently set to 0.13.2 because the NAS it runs on doesn't seem to work with 0.14 or later due to high CPU demand - still trying to find out why, exactly).

[thelovemyth]

thank you so much
after 4 betas, we'll have stable version soon.

[thelovemyth]

green icon in review ? another is gray ? so what is it ?

[NickM-27]

Green means it has been reviewed

[btrudgill]

Is anyone else getting more errors relating to genAI? Mine seemed to be fine using beta 3, but now I get dozens of these errors/warnings a day?

Gemini returned an error: 503 GOAWAY received; Error code: 0; Debug Text: session_timed_out

I'd say its 50:50 whether a event gets a genAI description or not. If I click regenerate, it usually generates one correctly.

[hawkeye217]

Frigate is just passing your requests to the Gemini API. This error indicates Gemini is just unavailable or unwilling to process your request. Frigate can't do anything about this.

[btrudgill]

Frigate is just passing your requests to the Gemini API. This error indicates Gemini is just unavailable or unwilling to process your request. Frigate can't do anything about this.

Ah ok, strange that I'm getting it more frequently. is it possible to be anything to do with any DNS adblocker (pihole) or any firewall rules that make things time out?
Is there any config in frigate that defines the timeout time or is it all at the mercy of gemini?

Not sure I want to run a model locally, on a gpu I don't even have, and currently gemini isn't overly helpful.

[hawkeye217]

Adblockers and firewalls typically block connections completely, so it seems unlikely that it's your pihole.

If you have not seen this tip regarding Semantic Search and Generative AI, check it out. The other main devs and I no longer use GenAI as we have not really found much value from it. While some users may still find benefit from GenAI, Semantic Search (all local) has outperformed our expectations.

[btrudgill]

Ah ok good point.

Yeh I think semantic search is great when wanting to find certain clips, but my use for genAI is to provide a description for notification in home assistant on a doorbell push, or an intruder in the garden for example.

I do hope that genAI isn't overlooked or dropped support for if the devs don't use it anymore.

[hawkeye217]

There's no plans to drop support for it, we just introduced it in 0.15.