[Feature] Facial Recognition Enhancements

[Dental1347]

(Parent discussion for FR enhancements)

Summary:

• 1. Ability to show/hide faces (similar to google photos) (feat(server,web): hide faces #3262)
• 2. Option to select an alternative person thumbnail (similar to google photos) (feat(web/server): Face thumbnail selection #3081)
• 3. Explore having a person "preview" when view people in a shared album, link, partner timeline, etc.
• 4a. Improve performance of viewing all people
• 4b. Improve performance of viewing all assets for a person (feat(web): use time buckets of person detail page (3) #3557)
• 5. Add the option to link/merge two people into one (feat(web/server): merge faces #3121)
• 6. Add a confirmation modal before running recognize faces for all assets (feat: confirm before deleting all faces and people #2496)
• 7. Add FR related features into the mobile app
• 8. Possibility of a dedicated People sidebar item (feat(web): people sidebar link #4257)
• 9. Explore reducing required clicks to tag people (feat(web): add better face management UI action #3328)
• 10. Add the option to remove/update a wrongly tagged person (feat: re-assign person faces (2) #4949)
• 11. Back arrow is broken when navigating to a person from the View All (people) page. (fix(web): back button on person page #2566)
• 12. Run the FR pipeline again for a specific image.

(Original Post Below)

The feature

Hi,

Awesome that the FR is implemented, it works awesome! After making use of the UI if felt a bit clunky and could perhaps use a bit of work. Down below I have listed a few things that I've noticed.

1. Currently within the UI there are 12 spots, which I presume are reserved for people with the most pictures. Although, I doubt this a bit. There are various things that came to mind when I saw this. It would be a fantastic feature to favorite certain people, this could be put under the Explore tab. At the same time adding a separate tab for Facial recognition, or People, could also be a possibility.
2. It would be nice if there is a possibility to add more than 12 people within the people tag within explore.
   
3. When you want to assign a name to someone that is not within the Explore tab you have to click on view all, then click on the person, and then you're able to add the name. A few things came to mind, add the possibility to add names to the image it self, as per shown below. Second
   
4. When clicking on a photo, the faces that are within the photo are displayed. I don't know if this is possible or even feasible, but it would be nice if the faces are displayed that are actually within the picture, instead of the general image which is associated with the person. This can help with improper matching, although I have not yet encountered this.

If there is more, I will edit it below.

EDIT:
13. When clicking on view all and selecting a person, after clicking on the back arrow in the top left. It takes you back to the explore page, resulting in needing to click on view all again. When clicking on back arrow within the browser, it does not do this. Although, I have also noticed this not working 100% of the time. This just adds another click.
14. The possibility to remove faces or hide them.
15. The possibility to unmark someone if they are incorrectly tagged.
16. A rescan on a per image basis if facial recognition does not recognize a particular face in the image. (I've noticed this happening with people who have a mouctache)

Platform

• Server
• Web
• Mobile

[jrasm91]

Thanks for documenting these!

[colin715]

I would appreciate a separate page/menu icon for People

[jrasm91]

We're pretty much sticking to the layout that Google uses right now, although we could certainly consider alternatives.

[dominikheinz]

I agree, a dedicated tab to manage everything related to people and FR would be nice.

[jrasm91]

An option to enable a dedicated "People" page link (number 8 in the list) has been added and will be in the next release.

[dominikheinz]

Great list. Another feature that came to mind after experimenting with the FR feature is the option to "link" people. For example, I have two pictures of myself, clearly in frame, but they are detected as different people. (Although to the human eye they look very similar).

So, it would be great if I could merge these "two" people into one, or link them.

I feel like a dedicated section in the settings for everything related to people and face recognition would be a good idea.

[Belphemur]

Link or merge them. I think they should be one "person".
One easy way I saw it done was to merge the people when we use the exact same name.

[bjrne]

I'd prefer same names to not be merged automatically, otherwise you have to keep in mind who you tagged already if you're not using family names...

[bread22]

"merge" is a great feature to have, my son shows up as 3 different people...

[ShinasShaji]

Another feature I can suggest is having the ability to set a 'Key' photo of a person that would then be the image displayed in most of the UI except where 4 above comes into play. This is possible in Apple's photos app as 'Make key photo', and in Google Photos as 'Use as feature photo'.

I think this would be great because in the current UI it is often possible for blurry or yawed faces to be key photos.

[DrSpaldo]

Some of the ideas for tweaking the UI are good ones. Having to do the view all, then click the photo, then name it, then back to the main list and start again is a little cumbersome

[Vcele]

In Google Photos there's also a way to improve recognition by manually assigning select photos to a person. Also, like in Photoprism, single faces could be moved into their own overview to prevent clutter.

[DrSpaldo]

I've just found my first person that was incorrectly linked to a people/person tag. Ie. There is 300 photos that have been tagged/linked to a particular person, however, one is incorrect and not that person. Is there a way I can unlink that photo from that person? If there is no way, can that please be added to the roadmap above?

[jrasm91]

Added!

[jeremywillans]

Is it possible to highlight in a photo where each identified face is? Scenario is when FR detects partial faces, if its possible to locate this in a photo in order to work out who it actually is to link to an existing user, or correctly identify them.

Also, the option to dismiss people/hide from the people list

[fabiannagel]

There's now a discussion for this

[Dental1347]

I've added two new suggestions. To my post, as I intended the first list to be edited by the team :)

[xyxzxyz]

This is an excellent list. However, any chance you can move the additional requests above "(Original Post Below)"? I almost overlooked the #12...#15. I scrolled past Original Post and missed those additional requests.

[Dental1347]

You want me to move them to the list above the post?

[Harris-KA]

My thoughts exactly! 5 would be great for me as anyone with sunglasses seems to be a different person!

[pablo03v]

5 please :)

[henley-regatta]

I'm finding the "All People" view very slow to load, because FR has found >2000 unique faces; that's a lot of thumbnails to load.
Would it be possible to add a "Named People" and "Unknown People" (without tags) filters to the interface to help with this in everyday use?

[henley-regatta]

Thanks.
I've since discovered that the bulk of my own performance issue was due to a conflict between the way this page is being built and a Chrome extension I'm using (HoverZoom+). Time to first thumbnail with that active was >10 seconds; if I disable it it's ~1.5seconds.
(I'm writing this here mainly in case anyone else is having this sort of performance issue)

[DrSpaldo]

Very interesting @henley-regatta , I should take a look at what extensions I have running as well to see if it would help my terrible loading times and partial freezing

[petersuba]

After running FR through my archive and ending up with 10000+ unique faces, most of them are small blurred faces of people in a crowd, I also think it would be useful to be able to specifically tag faces as "Unclear" or something similar, specifically to mean that a.) yes, these are correctly identified as faces (I don't want to drop their association with a face so that FR wouldn't come back offering them up again next time), b.) they are unusable for practical purposes and I don't care about them (similar but not the same as "hidden": a person may be recognised and his/her faces may be grouped but then the specific person would be hidden - in this case this is a collection of many faces that would likely be many persons also).

In terms of UI/UX:

• it would be useful to "tag" faces as this from multiple places, e.g.
  • when in "Edit faces" for a single photo, this option can be next to the "select new face" button
  • in the People page, this option can be added to the "hamburger" menu popping up for a face after "Hide person"
  • In the page for a single face's pictures, when selecting multiple photos, in can be on the context menu next to "Fix incorrect match" and/or as another button on the "Fix incorrect match" page next to "Create new person"
• faces with this "tag" should be treated in terms of their visibility the same way as those with "Hide person", except when using Show/hide hidden faces options, instead of just a list of hidden faces, these pictures could be shown as a separate category/grouping of photos (this is really just icing on the cake for fixing erroneous clicks)

[alextran1502]

@petersuba you can tune the number of similar faces to create a person in the machine learning settings. Default you need at least 3 similar faces to create a person. By increasing it to like 5-6 I think it will reduce the number of people get created significantly

[petersuba]

@alextran1502 I am aware of that, thank you. Even if I do that, I have many photos in from events where there is a crowd sitting in an audience and so there ends up being many pictures in a "person" that isn't real. My library overall has 200000+ photos over 2 decades. There is only so much one can do tweaking ML parameters as well before the valuable matches start to get dropped as well, not to mention the time it takes to run the model. If at least there was a way to re-run the model with keeping the named people/faces only and re-running the rest (an outstanding feature request) that would help losing the additional manual work, but even then running the model again and again takes time and processing power.

[radh21301]

Would it be possible to implement a feature to detect faces in videos? I know Google Photos does some magic and shows people in videos as well (not just people found in the video thumbnail). This is could be such a cool feature!

[Ziomal12]

It detects though. I even checked right now to make sure and surely when I click on one of the faces in first 3 lines of photos I've got 2 videos.

[Ziomal12]

Anyways, if anyone wants to play around with face detection in a video I'd suggest to analyze 1 frame per second of video or even less (maybe even 1 per 5 seconds)

[radh21301]

I checked again. I can see that people are detected if they are present in the video thumbnail (or in the very beginning of the video). But not when they come in the middle of the video. Maybe I am doing something wrong.

[xyxzxyz]

It does recognize faces in videos, at least in mine.

[tsemachh]

Perhaps consider allow having in the person view in video links to the positions this person appears , this would allow finding people on specific location on the video

[lukanowa]

I have some suggestions to hide/show faces ability:

1. Merging peoples option should not show hidden faces,
2. When we want to hide more faces all faces with no name on it should show on top below named faces (it hard to find missing faces to hide when we have over 300 faces)
3. When we try to name faces and this name is already use can Immich ask to merge this two people?

[martabal]

Additionally to point 2, I think it would make more sense to be able to select multiple faces and hide them all at once whitin the faces page and not the hidden faces. Could look like this:

At this point, just use the merge editor ?

[0rn0lf]

merge editor? Where can I find it?

[martabal]

The "Merge Faces" button.

[yoyotueur]

Would it be possible to add an autocomplete when faces have not been recognised and you type for the name so you make sure you don't input the wrong format for existing people

[martabal]

Yes, this is in my TODO list

[xyxzxyz]

Auto add person/pet to album is a must.

[thomasloven]

Some form of "mass retagging" would be awesome too.
I have a person with 200+ assets where perhaps 100 are wrong but of the same second person.
Right now it's six clicks to retag each.

I wish I had some idea on how a better way to do it might even look...

[tippfehlr]

select all wrong faces and click on the three dots -> ‘fix incorrect match’ :)

[jakob42]

I cannot find that on v1.94.1. Where did you find that? I've got a couple pictures selected and want to change one (wrongly) recognized face into another...

[tippfehlr]

Sorry, I wasn't specific. You have to select them on the page for a face.

[search] -> people -> click on the face the images are wrongly assigned to -> select images -> click the three dots at the top right

[thomasloven]

I have no idea how I could have missed that...

Thanks a lot!

[tippfehlr]

I have two people shown as one (because of similar faces) a few times.

Currently, there is no way of manually adding faces (?), so the wrong face can’t be changed manually.
So, first step would be to allow manually creating faces
But manually making FR more specific in these cases (by telling FR which face is which) would be great.

EDIT: I just found the (much more powerful) menu when clicking on ‘fix incorrect match’ in the page for a face.

[geoleven]

One great feature would be to add the ability for facial detection / recognition to be able to run on all assets without losing the previous tags.

Let me explain:
Right now there are two options, missing and all. Missing runs for images it has not already run, all will remove all the tags (for me that I have more than 600 named people, this is a week's worth of work). Right now, and depending on some conditions (bugged out facial recognition, etc which I probably have to file as a bug at some point), I end up in a lot of situations with obvious missing faces detected and only the already named people are tagged. I would like to have a middle option to rerun the face detection on the parts of the photographs that is not tagged already. I don't really know the technical background so I am not sure this is technically easy or viable. Another take would probably be (if the previous one is undoable), to enable a per photo - on demand rerun of the facial detection. This means that although I may lose the info for the already tagged people for a photo is easy to reassign this specific one adding the new ones.

[tuupola]

Something like this is needed. It is also currently bit too easy to loose all your named persons.

[jenda69]

It would be nice to have (or is it there somewhere?) option to rescan face in a photo, similarly to "Refresh metadata" and "Refresh thumbnails" commands.
I have an external gallery where I manage my photos externally, that especially means editing RAWs. And when I'm done editing, I run a script to refresh edited output JPG photos in Immich using API command runAssetJobs with regenerate-thumbnail and refresh-metadata parameters. I get new thumbnails etc. but if I cropped a photo, the faces are now tagged in wrong positions when hovering over them in details panel.
It doesn't really break anything, but I find it a bit annoying.

[waclaw66]

Really missing such feature. Currently using SQL workaround...

delete from asset_faces where "assetId"='DESIRED_ASSET_ID'

Following Refresh thumbnails recognizes asset faces again. With caution!

[jhit]

I would love to see an option to share FR Data with a partner account. It is a huge effort to maintain and correct recognized faces. I want to share the training data and already tagged persons with my family. Furthermore, I want my family to be able to see and search by people on my shared pictures.

[aviv926]

#7045

[tuupola]

I am also looking for this feature. We have a group of people recognizing people from old photos. Currently this is possible only by using one shared account, which is not optimal.

[isaacolsen94]

Hey all, I've seen manually being able to tag faces mentioned a few times, but I don't see it on the list of features...is this something the team is looking into? Or is there no plan in implement it?

[alextran1502]

it is something we want to do eventually for sure

[isaacolsen94]

Awesome! That's good to hear! We know there is a lot of stuff on your plate, thanks for confirming!

[tuupola]

Manually tagging a person to a face would be great. For example if I have four photos of a person, three of them are recognized and person is created. Currently there is no easy way to add the fourth photo to the person.

Only way I can make this work is to set the MIN RECOGNIZED FACES faces to 1 and then keep merging the single image persons. Or maybe I am missing something?

[MacBadger]

Manual tagging is my number one! It would be great. The only feature Immich doesn't have!

But just to highlight at the top, "remove/update a wrongly tagged person" is ticked as complete. This isn't the case as far as i can tell. You can reassign. If say a none-person (or stranger!) is detected as being someone, you can't remove the tag... only change it to someone else.

[whoindahouse]

Any way to increase the number of unnamed faces you can merge into a named face from 5? Even temporarily...very useful when first scanning a library

[filski]

I haven't tried yet, but: #3894 (reply in thread)

[porjo]

After adding a bunch of photos and now have a lot faces to sort through. Some things that could make this easier:

• merge faces by dragging a face thumbnail onto another face thumbnail (saves three clicks)
• double clicking on the face thumbnail would cause the 'change name' form to appear (saves one click)

Thankyou to everyone involved in creating Immich!!

[whoindahouse]

Or the ability to select multiple faces from the people page via CTRL+click and merge them into one

[porjo]

@whoindahouse yes I think CTRL+click might be better. Dragging and drop could get annoying if it becomes necessary to scroll the page up/down to find the target thumbnail.

[radh21301]

Detect 'person' using clothing and other visual cues across photos taken within a similar timeframe --- Like Google Photos

I this this is a super cool feature in google photos. I was sure surprised when I saw the results the first time. Would be really nice to do something like this in Immich as well!

https://www.androidauthority.com/google-photos-facial-recognition-back-3333108/

Edit: Looks like there is another feature request with the same improvements for facial recognition (#9810)

[themistymay]

In reference to this request: #3356
Unmerging faces (incorrectly detecting multiple kids as the same person). Match was done by FR not merge feature so this would not be a reverse of merge operation, but rather a retrain / rerun after pulling out bad matches.

The issue is that no other "person" is the other kid so there is nothing to reassign and it seems that adding a person is not correctly allowing a FR workflow.

[Germs2004]

Please allow the user to "confirm" an AI-recognized face so Immich will know it's recognized correctly. Immich should also have a view of all the unconfirmed faces, per person, so we can highlight a bunch at a time to mark them confirmed. If Immich ever rescans all the photos to update its guesses about who each face belongs to, it should keep all the user-confirmed faces and only edit the guesses of unconfirmed faces.

[Germs2004]

If I set a person's Date of Birth in Immich, the AI facial recognition should consider that information along with the photo's Date Photo Taken metadata when deciding which person that face belongs to. It should not recognize a face of an adult as being a person who has a date of birth 3 years ago.

[Germs2004]

If Immich assigns a face in a photo to the wrong person, and I edit that face to the correct person, I'd like a way to tell Immich to rescan all the faces with that new information I gave it so it will automatically correct any other mistakes that are similar to this one. This feature seems to need the "user-confirmed face" feature I mentioned above so it will know which faces are definitely correct and which ones are its own previous guesses. It should adjust its remaining unconfirmed guesses based on the photos I've since confirmed.

For example, I may have 1000 photos of John and 1000 of Sally, but 20 photos of Sally are incorrectly marked as John, so when I edit one of those mistakes from John to Sally, Immich should fix the other 19 automatically.

[bo0tzz]

I'm going to close this thread since most of the original points have been implemented and it's impossible to track dozens of feature requests in one thread like this. If there's a request in here that you want to see and that hasn't been implemented yet, please check for duplicates and then open a new FR if there isn't one yet.