chaining an object classification model off frigate

[tve]

I'd like to "chain" an object classification model off of frigate object detection in order to be able to classify detected objects more finely. I'm most interested in bird species, but it could also be persons or specific pets. More background in #1426

I've been going back and forth about how to try this out and my conclusion is that I'd like to start by building something external to frigate that can consume events published by frigate via MQTT. If the whole thing is successful, and others are interested, and it fits into frigate's mission, and the stars align, then maybe it can be integrated later. But initially building something independent seems a lot better.

What I'm missing is two things: access to better camera frames for further processing and more intermediate events. The first issue is that when an event is published the second stage really needs the exact same camera frame so it can apply the object bbox and do image classification. Right now this does not seem possible: it's a race to fetch /api/{camera_name} and hope things haven't moved too much, which is very unsatisfactory. Maybe I'm missing something.

The second issue is that frigate only publishes an event update if the object tracker thinks it has a better view of the object, but that doesn't necessarily match with what a downstream classifier might conclude. At least that's my experience so far. So I'd like to add a setting that causes frigate to output an event update every N seconds even if the current view is no better than the previous view.

The concrete proposal I have is:

• in CameraState.update delay the deletion of a frame in the FrameManager so it sticks around for 10-20 seconds (prob configurable, this could be conditioned on there being an active event)
• in the API add a /api/camera/at/.jpg endpoint to fetch a camera frame from the FrameManager given the timestamp, so this would work to fetch the exact frame corresponding to an event update for 10-20 seconds
• in the detect configuration section add a force_update section to specify an interval in seconds at which frigate issues an event even if the view is no better

Thoughts? Concerns? Alternatives? I'd love to get some feedback before I start hacking and produce a PR...

[blakeblackshear]

This would significantly increase the required space at /dev/shm. I would be open to making the rate limit of the events topic configurable, but I'm not convinced this approach is even practical. There is a reason I am managing all of the frames in an uncompressed shared memory store. I don't think you will be able to implement this successfully.

My suggestion would be to do the best you can with the existing APIs and mqtt topics because I think you are likely to end up with different but similar challenges anyway.

[tve]

I have tried using the current APIs. It doesn't work. By the time Frigate produces the event, it goes through MQTT and I get around to issuing an HTTP request to fetch the current frame the bird has moved significantly and the bounding box is off and/or the bird pose it bad.

My back of the envelope calculation is as follows:

• UXGA frame: 1600 * 1200 * 1.5 bytes = 2.7MB
• An event per second, frames retained for 10 seconds means 10 frames in memory = 27MB

That doesn't sound excessive to me. Even if there were 10 cameras it wouldn't cause issues as far as I know. The only alternative I can come up with is to shove all these frames into mqtt and that seems worse to me. In the end, the primary purpose here is to enable experimentation with new stuff. If it works and efficiency is required then it will probably have to be integrated more tightly into frigate.

[blakeblackshear]

Have you tried using the snapshot.jpg endpoint instead of the current frame? That will return the frame that corresponds to the bounding box in the event from memory. You can even have it pre-cropped to the bounding box for you.

[tve]

Here's what I see when I use snapshot. I'm waiting for the "end" event and then fetch the snapshot and crop to the bbox in the event. The cropping is often too tight, like:

I know I can easily add space around the crop, that's OK.
Other times the pose is horrible:

Out of the last 18 events only 5 of the snapshot photos are good. I looked at a bunch of clips and there are definitely much better frames in there.

[blakeblackshear]

Use the region instead of the bounding box. Frigate automatically readjusts the region if the bounding box is close to the edge of the region and runs object detection again. Also, you don't have to wait for the end event. The snapshot is available before the event ends.

[tve]

I have different proposal (haven't looked at the code yet to confirm feasibility/difficulty). With each event, publish an array of metadata that allow reprocessing or visualizing the clip corresponding to the event. The metadata would contain an entry for each frame processed by frigate and each entry would consist of: exact frame timestamp, object type, detection score, object bounding box, plus anything else relevant that I'm forgetting. This would allow an external classifier to fetch the clip, extract the frames that frigate analyzed, and for each one crop out the detection bbox. This would also allow some other features for debugging, such as producing an annotated version of the clip that visualizes what frigate's analysis came up with.

[ozett]

Then my external classifier can fetch the recording and extract the relevant frames and crop as desired.
is it not the same as grabbing a video/clip-sequence?
with all the frames in the clip the second-classifier could extract the frames and analyse them,

there would just needed a AI-based startpoint, means the first frame for the clip should have detected the bird-object..
or what other benefits do you get creating that store array for mqtt?

[tve]

I'm not sure I understand your question correctly. The reason I want more than just the "best snapshot" from frigate is that the best snapshot is not necessarily the frame where the object is the most recognizable. The reason I want the full history array is that I need the bounding box for each frame so I can extract the relevant portion of each image. I'm including some additional info, such as region and zones, because I simply save the event data away.

[ozett]

ok, i see. a different thinking for second stage.

for me, a second stage AI could do some work again,
grab frames from a short sequence again and do its own AI on that frames, like object detection with another model.
i would only roughly use the first AI as a "trigger AI" and start grabbing an interesting video-sequence to use in second stage.
maybe to simple thinking..

[tve]

Sure, if you want to try out better/different object detection you can just grab the recorded segment and then re-run anything you want. You then get to implement motion detection, unless you don't want it, then extract regions to feed to the detector, etc. I didn't fancy doing all that, I want to get more specific information on the object detected and not redo work that frigate already did. Different goals, I guess :-).

[ozett]

you brought up a second-stag AI.
Thats a fantastic idea.

now lets see how that transfer to the second-level can be generally modelled ... 🤔

[tve]

Quick update. The initial camera I used was too crappy (the lens, specifically) and as a result the images extracted were too poor to yield decent results. I now have a better camera (which still needs some focus adjustment) and the results are better. Here's a screen shot of classifications at my bird bath this morning. There's still a lot of room for improvement, but at least it's not all garbage.

Note that all the images here are an exact representation of what is passed into the image classification model. The model takes a 240x240 image and that is a cropped & resized section of the full frame. That's why the quality of the original image is really important when trying to classify small things...

Behind each one of the images above is a page that shows all the frames that were analyzed. For example:

The image used by Frigate as snapshot is not the one used by the classification for the top score:

So far everything is strung together with duck tape and bailing wire... But it's showing signs of working! The change to frigate to export the event with the detailed info that allows the entire clip to be reprocessed in the image classification app is really minimal.

[ozett]

Behind each one of the images above is a page that shows all the frames that were analyzed.

why is the hit-percentage on the analysed frames missing? woudnt it be more informative than the frame-number? or is not the frame-nr but the inference-percentage?

[tve]

These numbers are not actually percentages, e.g. 84 doesn't mean "with 84% probability" or anything like that. They're just scores on an arbitrary scale from 0 to 100. For the classification I put the scores behind the label under the image.

[ozett]

i still like the idea of "chaining" object classification.

could potentially help to find and compare for models with somehow "better" detections capabilites.
maybe at least for a set of inididually choosen classes.
there is still no "chaining", but if it would, the question arises for me as a noob, if all this models were equally usable or suitable:

https://hub.tensorflow.google.cn/s?module-type=image-object-detection

[blakeblackshear]

There is some related discussion here.

[tve]

I saw that, thanks. I implemented the publishing of the metadata as I described. It's a pretty small change and it works extremely well for reprocessing the video clip taking advantage of all the work already done by Frigate. It sounded to me like there's little interest in merging a PR, so I haven't created one.

[ozett]

maybe the "publishing of the metadata" can be integrated in the request from @jakowenko from the above cited discussion ?

That would open up the possiblity to do a second-stage AI. But this still depends on first-stage AI with a succesfull first-stage detection with the first-stage AI-Model to get a "primary" event. doesnt it?

Could there also be at that place an interface to do other AI-Models in parallel?
Only based on frigates motion-trigger events?

[tve]

You are correct that the metadata only gets published when Frigate's model detects something. I don't know that publishing the motion detection makes sense because it often generates multiple bounding boxes per frame on each frame. That would result in a quite high MQTT message rate and then the receiver has to somehow get those video frames and decode them for analysis. At that point I believe it would be better for Frigate to have an A/B capability built-in so it can run two models and produce info. Overall, I wish Frigate was more modular/decoupled to make it easier to add/replace pieces, but that's a different discussion.

[ozett]

yeah. but i would be quite interessting to see detection results from different models (custom trained models?) and in comparison.
like in an A/B/C/... design=
i personally grab all the frigate-cam streams in parallel and use yolo3 for object-detection too.
the differences are quite interesting and helps me to optimize my needs for surveillance.

so one has to figure out a good junction-point to feed a second-model?
in the same containter? or over the network? with an api? how to get the (decoded?) streams into another AI? and who runs that AI?

[tve]

so one has to figure out a good junction-point to feed a second-model?

Yeah, that's the issue... Frigate is pretty decoupled architecturally but the code is quite intertwined and the interfaces between the pieces are not designed to make it easy to plug something else in. At least IMHO so far... The decoded frames are placed into a shared memory area so multiple processes can access them, that's great. But their lifetime is very tied to Frigate's current processing. There are message queues between the processes, which is great, but they're point-to-point, not pub-sub. Ideally you could have an extra process subscribe to the right queue, grab frames from shared memory, and publish images to the CloudTPU process for it to run a second model on, then subscribe to the results coming back.

To be honest, at some level I don't know whether the above is actually worth it. If you want to dev a different object detection model you're probably better off grabbing a bunch of recorded scenes and run them offline through the different detector variants. This way, each time you tweak the model you can re-run the same scenes and really compare. I'm getting pretty close to working on this 'cause I need to get more wildlife labels into the object detection model.

What I'm currently doing is different 'cause I'm chaining an image classification model after Frigate to get more precision on the object detected. As long as I'm not trying to share the cloudtpu with frigate's model I can easily run the classification outside of Frigate's container in a very decoupled manner. It also runs quite rarely, hence using a bigger CPU model works OK.

[scooper1]

I would be interested in being able to chain frigate classification to another system for different model detection

[ozett]

anybody seen already this birdfeeder-project on google-coral ?
https://github.com/google-coral/project-birdfeeder

[ozett]

@tve
is your code for the meta-data available?

i followed the idea of the bird-classifier and also needs better object-crops.
#2574 (comment)

as the doods2 developer will build the bird-object-classifier into doods2,
you could use this as a classifier service.
snowzach/doods2#3 (comment)

still good (croped) images of the objects are needed...

after that is done, one could think of lots of other classification ideas...