[Question]: Birdbox motion detection events, low threshold object detection, and question about order of operations

[SpangleLabs]

What is your question:

Hey there,
This might end up rather muddled, but the long and short of my question is that I have a bird box with a camera inside (It's a Green Feathers camera which I believe is a rebranded Tuya), and I'm trying to get notifications even when a bird just looks inside, without getting notifications when cobwebs blow around.

The problem is that while the object detection is pretty good at cars and people, it's not very good at birds, especially at partial views of birds.

(The Green Feathers app, btw, detects only raw motion, and gives 5-15 notifications a day, especially at sunset when the camera switches to night vision.
But the main frustration with using that is that it's all inside their app, the notifications only display the first frame of the video which is almost always empty, there's no way to go from the notification to the video clip, and there's no way to simply download the entire video clip.
Frigate's UI is frankly wonderful. Even when not compared to this other app, but especially when compared to that app!)

Setup

The bird box camera is 1920 x 1080, and detect resolution is set to 1280x720.
Frigate is running in docker on Ubuntu on an old desktop with a decent graphics card, (GeForce 970) with GPU acceleration working, but CPU detection, no Coral/TPU.
There are 5 other cameras being managed by Frigate (4 cctv, 1 doorbell). We did not install the cctv system in this house, so I mostly like using the garden cctv cameras for bird watching, and the front-facing cameras for foxes and general use.
Frigate is publishing to MQTT on another home server running Home Assistant, which has automations sending start and end events to Telegram. (Filtered on minimum size to avoid cars driving past from triggering cameras)
I have an exporter running exporting Frigate stats to prometheus, sending those to a grafana dashboard, to monitor its health. All cameras are running detection at 5fps with negligible skipped frames and about 45ms inference speed.

Relevant parts of config:

go2rtc:
  streams:
    bird_box:
      - rtsp://192.168.1.242:8554/main
      
ffmpeg:
  hwaccel_args: preset-nvidia-h264
  
detect:
  enabled: true
  width: 1280
  height: 720

record:
  enabled: true
  
objects:
  track:
    - person
    - car
    - bird
    - cat
    - dog
    - horse
    - sheep
    - cow
    - elephant
    - bear
    - zebra
    - giraffe
    - banana # Using this to try and eagerly detect birds in bird box
  filters:
    banana:
      min_score: 0.5  # Set these high, because I don't care about bananas elsewhere
      threshold: 0.7
      
cameras:
  bird_box:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/bird_box
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
          # 1920 x 1080
      output_args:
        record: preset-record-generic-audio-copy
    objects:
      filters:
        person:
          min_score: 0.2
          threshold: 0.65
        bird:
          min_score: 0.1
          threshold: 0.6
        banana:  # Using this to try and just create events on any motion detection
          min_score: 0.0
          threshold: 0.0
    zones:
      bird_box_entrance:
        coordinates: 794,0,795,191,498,188,505,0
        filters:
          banana:  # Using this to try and just create events on any motion detection
            min_score: 0.0
            threshold: 0.0
    motion:
      mask: 397,68,394,24,78,26,84,66  # Ignore the timestamp changing

Examples

Some examples from my testing:

Example 1:

A bird (blue tit) peeks inside, climbs fully inside the box, turns around, leaves the box
https://github.com/blakeblackshear/frigate/assets/2192223/4e02a8b0-5d15-4283-b71c-b9803fe02410
Frigate correctly identifies this as a bird, reporting 63.1% top score at the end of the event

Example 2:

There is no bird, there is nothing, some slight lighting change happens, a cobweb fragment blows in the wind, video compression maybe does some tweaks
https://github.com/blakeblackshear/frigate/assets/2192223/8e9b8ddf-a49e-41e5-bcb6-848345d6e826
Frigate identifies this as a bird, reporting 74.0% top score at the end of the event

Example 3:

A bird (great tit?) lands, peeks inside the box, and flies away
https://github.com/blakeblackshear/frigate/assets/2192223/646bd6f4-19b8-46df-a31c-3681e1f991af
Frigate detects no event at all. :(

My attempts at tuning thus far

I've tried a few things so far, but these 3 examples are very recent, after most of my tuning efforts, and the confidence levels between the two detected events, and the missing 3rd one, has left me a bit lost, and with a few questions.

I tried setting up a zone for the bird box entrance, with the hope of getting events on any motion there, but I've not had much luck with the motion events

I've not yet tried tuning the motion settings added in 0.13, that seems like it might help avoid Example 2 being detected, but wouldn't help with example 3?

So, I can be reasonably certain that anything detected inside the box is a bird (well, there is a spider in there lately, and the Green Feathers app once notified me of a fly that flew inside the box for a bit), so I've been trying out stuff where I'll set the object detection thresholds incredibly low in the hope of getting a raw motion event feed, but that leads to the questions:

Questions

Q1) I understand how the thresholds and min scores work across time for a given label, but how does it work across the label-set? I assumed the image region being analyzed would be passed to the model, and the model gives a list of weights for each of the labels, and then filters get applied, so I tried setting filters: banana: {min_score: 0, threshold: 0} with the thought that that would mean any motion that get sent to the model comes back with maybe higher scores for other labels, but at least a 0.0001 score for banana, and hence will be counted as an event. But it seems like this is not the case? Is it instead that only the highest rated label is the one passed to the filtering, perhaps? Or should I continue to pursue that as a way to trigger events on motion? Even with banana threshold set to 0 on the entrance, example 3 did not trigger an event.

Q2) Any idea which ideas for motion tuning might be most successful? I probably need to play around with increasing motion.threshold, and modifying the motion.contour_area at least
But would it be good to lower the detect resolution for this camera, so that things like cobwebs are lost in the resize?
Or is the default motion.frame_height of 100 already resizing it to something like 177x100?
It's tempting to mask out the bottom of the box because that's where the vast majority of the false events are, but I would be worried about missing a bird going fully into the box that way

Q3) Somewhat less related, but in my testing, I see a few notifications via home assistant for the starts of events, without a corresponding end event getting posted. Just wanted to check: Is this a thing that Frigate can do? Like, cancel or revoke an event after it starts? My other theory is that it may be related to the minimum size limit for notifications. Is there a way to get the maximum size of an object in the MQTT event, rather than the current size? In the same way that we get top_score, perhaps a top_area?

Thanks again for this wonderful software by the way! I was very glad to get rid of the NVR that was fitted in this house, and I am enjoying exploring Frigate's options, even if it can be frustrating sometimes. Even the incorrect classifications are making me and friends chuckle (especially when Frigate informs me of a giraffe visiting the garden)
And again, gotta shout out how pleasant the user interface is, and config management. I wish Home Assistant's UI was half as nice.

[NickM-27]

1. The score of some label won't affect the scores of other labels. And no there is no limit to detections that are sent from the model, just a maximum of 20 for each region.

2. Every event will have an end. This would be due to filtering you are doing.

[SpangleLabs]

Well that scuppers basically all my plans there!

Maybe I can work around it with a local patch to label everything as banana or something, and have all other cameras ignore that.. I dunno.

Different models or detectors per camera seems a handy feature though, so I'll file a ticket on that
Here's the ticket: #9917

[NickM-27]

there is already a feature request for it

[SpangleLabs]

Ahh, whoops, sorry!

I'm not sure whether that completely blocks my hopes here, tbh

[NickM-27]

I mean you can already run multiple instances of frigate

[SpangleLabs]

I suppose so... Certainly not ideal, but yeah.

I guess I could run one frigate instance which takes in just the bird box, uses go2rtc to restream it to the other/main/current instance so that it can display all 6 cameras in birdseye view..
Then keep recordings in one or the other, and may as well do event recordings in both as the stock model seems sufficient for some detection