search: make French in-page search accent-insensitive

[Pinzauti]

Hi,

I'm using the standard lunr configuration for search. Theme version 8.0.1 and Hugo version 0.150.0.
I would like search to be accent-insensitive in addition to case-insensitive. For example, if I have a page with the word "Éditeurs" and I search for "éditeurs" the result is found. However if I search for "Editeurs" or "editeurs" this is not the case.
Is there a way to achieve this result? Or could this feature be considered?

Thank you!

[McShelby]

This is an issue of the underlying french lunr-languages package. See MihaiValentin/lunr-languages#68

[Pinzauti]

Hi @McShelby, thank you for your answer. For your information, the issue is not related to the lunr-language package but to the theme itself. I was able to fix the issue by modifying the theme.js and search.js files.

On a general level, I created a function:

function normalizeAccents(str) {
  return str.normalize('NFD').replace(/[\u0300-\u036f]/g, '');
}

and then in the function highlightNode() added:

var normalizedData = normalizeAccents(node.data);
var match = normalizedData.match(re);

same thing in the highlight() and elementContains() function. I also changed a bit the logic in the renderItem(item) function but it's not essential.

It's working on my end as you can see below:

I know that for English this is irrelevant, but for French, Italian and many other languages I believe this feature is essential. I can create a PR if you want.

[McShelby]

Now I understand what you mean.

Search contains of two parts: search inside the currently displayed page (your use case, that highlights your search text on a page) and search over your whole site (displaying a weighted list of result pages, in the search box or the dedicated search page).

I am reluctant. Reason is, that this will cause in-page search to behave differently to site-search. To stay with your example "Editeurs" would be marked on a page but would not be found via site-search.

For German (my native language) removing diacritics would also be kind of non-sensical.

@postb99 What are your thoughts on this?

[Pinzauti]

This is not true though, as you can see from the screenshots I attached above this is working also for searching in the whole website. So in-page search would behave exactly the same as site-search.

It's actually in the current situation that in-page search behaves differently than site-search. I took the liberty of visiting the website of @postb99.
Let's say you want to search the word "decide". The actual word is "décide". As you can see you have several results. The first one is this. But when you land on that page, no word is highlighted. Why? Because in site-search "decide" is the same as "décide", while in in-page search this is not the case.

I can't comment for German language as I don't know it, but regarding French, Italian, Spanish and Romanian what I'm proposing is the default behaviour of almost every search tool I have ever encountered. Maybe it could be an option that users can activate through a parameter?

[McShelby]

You are right, here German and French behave differently.

The reason is, that the in-page search uses different code as the site-search. Best would be to let in-page search use the site-search somehow. I think, this would be the best solution but may be hard to achieve.

[postb99]

I feel blessed that my site is a nice area to perform searches and investigate.

I agree that in-page search would be best similar to site-search but it seems difficult to achieve unless you get a clue how site-search returns the words somewhat around the searched term. Just thinking out loud right now, this could help.

Pinzauti's code hack looks fine at first glance for his need and French support.

I also hacked search code, but for site-search... However it was hard to understand how to plug my custom code, so it's not very nice, but it works for my need. I'm not proud of this. In assets/js/search-lunr.js.

I had the day to think over user intend when he lookups for some term and about diacritics being essential in German. Sometimes German diacritics are replaced by unaccented letter followed by "e", when you are unable to easily type the desired diacritic. But what about searches? Maybe Lunr internally does some equivalencies lookup, i.e. "ü" is similar to "ue"? And German-specifically?

About casing, maybe search works well, but if I think over:

• when user types "Robert" he may surely lookup for such a proper name, but won't be scared "robert" matches too
• when user types "éditeur" he expects "Éditeur" to match too, because an accented letter can be capitalized, especailly in some kind of texts like poetry. I even don't know how to type an uppercased "é", haha. So I expect lookup to be broad.
• for French, indeed it's best when "editeur" and "éditeur" give results without the same diacritic or case (broader search, same as preceding point, I'm able to type "Editeurs" and lookup for "éditeurs").
• last thought, Lunr is configured in search.js to calculate a matching score for search term taking into account some fuzziness, and that could give 100 matches on my website because it scores high for words that look the same excepted one letter... I think I'll give a try a day to Orama alternative search engine McShelby tried.

Sorry I think my rambling does not help much but I enjoyed this subject being discussed this way.