Facial Recognition UX

[briantopping]

Hi, just getting back into Frigate and loving the changes that have come together while I've been gone over the last few years! Thank you for keeping it alive!!

Regarding the facial recognition, I like the Apple Photos UX a lot. In that, they search for faces and leave them anonymous. When it can, it groups the same face recognitions together, but they remain unnamed until the user gives them a name. As new pictures arrive and the recognizer has a high probability, they are automatically added.

What would be nice about this in a security context is if there is a problem on our street at 2am and I have a face shot, it should be simple for the recognizer to recover other video it has of the person from the past. In a situation like that, I surely don't know the person to give them a name, but it might be that they have been casing my block before during the daytime and a better shot of them might exist. Having multiple times and dates makes all the difference for investigators using IMEI data to get further.

[NickM-27]

Hi, so to be clear this isn't really a UX change, this would be an entirely different feature / use case that would include a retention period for the faces that are detected.

With that said, based on how the face recognition feature works, I don't think this would actually work that well. To have a fairly accurate face recognition many photos are needed of the same face, in at least decent quality and resolution. If we have one face of a person and are essentially saying "find all faces that look similar to this" odds are you would get a random array of results of unrelated people. Especially as crimes happen more often at night / low light, which would make the quality of those faces even worse.

[briantopping]

Sure, I was focusing on the overall user experience, I recognize that it is not just a UI change.

Regarding whether it would work, wouldn't you agree that if we had long distance sensors that rivaled photos taken at typical consumer grade phone distances that there would be no difference in the ability to perform this feature at the same level of accuracy as other commercial products? I am not saying we can do that today, but a PTZ with sufficient optical zoom would work. Those probably exist today for some level of cost that most can't afford, but the costs will come down. Recognizers will improve over time as wll. Having a good UX that supports such cameras seems to be a sensible move.

[NickM-27]

Having a good UX that supports such cameras seems to be a sensible move.

It is entirely separate though, it has nothing to do with the face recognition feature, which is an entirely different use case. The parts of the code that would used to perform such a search wouldn't even be the same except the part that runs the model.

And trying to combine the use cases into a single UI would just make the current face recognition feature more confusing and work in a less efficient way than it does now.

[briantopping]

What I offered was subjective experience of how I think it would work best, what I think would be helpful. I am not asking you to implement it or even agree with my perspective. If it influences some future version, it was worth the time to contribute it. Cheers!

[NickM-27]

Right, I get what you're saying, but what you're suggesting is a fundamentally different behavior, that would require a ground-up rewrite. I think it is important to provide context for why things are done a certain way instead of just saying it is the way it is.

A photo library and an NVR face recognition are fundamentally different because photos in a photo library are inherently assumed to be persistent. You save a photo and that photo exists as long as the library does (in most cases, obviously you can delete a photo). So in that case the grouping of people makes sense, and this is actually an entirely different way of doing face recognition, as you essentially create a graph of faces and group faces that are in a similar position. You can read more about how photo libraries like Immich do this

But face recognition in the context of an NVR is entirely different, you are taking a known dataset and trying to ascertain if a single face is related to that dataset. It is looking for known faces to see if a new face is one of those known faces, not grouping all similar faces together and storing them indefinitely.

[briantopping]

Thanks, makes sense. Appreciate you taking the time to explain.

What caused me to believe this is eventually possible and a worthy target state is the recognizers typically work on an n-dimensional vector distance formula, in this case from facial geometry pulled from 2d space. I understand completely what you are saying that the camera is generally square to the face and a camera lofted 12 feet in the air (and everything in between) will provide different geometry than a squared nearby shot. Most people don't have PTZs, although the fact that Frigate supports them now (with tracking and zoom even) tells me that getting a good face shot is not technically outrageous.

Even if we have a concept of camera angle and height (some of that provided by the PTZ initialization vectors in Frigate?), we still have a lot of variables that are impossible to find except on military grade cameras because the encoders suck with consumer grade or their calibration is unknown. Programming such a system with today's resources for the few people that might have such cameras is not an option. At some future point, sure... feed it to an "AI" and let it work out the correct interpolation scheme for a picture given constants such as absolute location and variables such as outputs from precise encoders. Frigate is going to be amazing in 20 years!!! What if I was going to start a Kickstarter for such cameras? Do we have to cut off this path so early?

What I don't think we know at this point is where the middle ground is. I respect (for lack of other input) you have enough experience with what's possible in the community to say "this ain't it". And I hear you loud and clear. But the first rule of getting to a destination is knowing where it's at. What I am talking about in this FR is defining that point and saying "someday we'll be able to do this". If it's going to take twenty years, then yes, let's close this out and act like the idea never occurred to anyone. I don't think it will. Maybe five, but that's not too long to start thinking about the incremental steps that are required to get there. And when I say that, I am not talking about tasks that I expect you or anyone else on the team to take on. I was just trying to define a destination. That's it.

Hope that helps provide additional clarity from my end as well!

EDIT: I added all that clarity because I don't think we need to capture an infinite history of pictures, rather just a few of them for human consumption and some form of the individual facial model, maybe including some evolution of it over time. Again, five years from now, we might have storage that makes what we have today look laughable. A lot of things will change, getting too deep in the details at this point is probably not overly constructive.

[keteflips]

#19940 (comment)