Improve search (make fuzzy again and more robust) #932

nikku · 2024-09-23T16:27:21Z

Proposed Changes

We found various issues with our existing search, both in terms of robustness and behavior:

Under certain circumstances searching loops infinitely (https://github.com/camunda/web-modeler/pull/10940)
Search is no longer fuzzy, i.e. matches terms across all fields, and hence is overly strict (Improve search (make fuzzy again and more robust) #932 (comment))
Search does not return all tokens for matched entries, making it hard to integrate it properly (Use search bpmn-js#2235)
Lack of test coverage around contract and other crucial aspects

This PR solves these issues, while keeping the generally desirable properties:

We sort search results semantically, preferring
- start of word
- start of term (in word)
- we sort alphabetically (where items are equal in semantics)

The overall goal is to offer a snappy, robust, and intuitive search behavior, and I'd like to put up to a test if we accomplished that:

Try it out via

npx @bpmn-io/sr bpmn-io/bpmn-js#use-search -l bpmn-io/diagram-js#fix-search

Checklist

To ensure you provided everything we need to look at your PR:

Brief textual description of the changes present
Visual demo attached
Steps to try out present, i.e. using the @bpmn-io/sr tool
Related issue linked via Closes {LINK_TO_ISSUE} or Related to {LINK_TO_ISSUE}

nikku · 2024-09-23T17:45:42Z

Additional finding, debugging the implementation:

We do no longer tokenize between spaces, this makes it impossible to match two works, such as this:

fix: correct popup menu search on duplicate entries Related to bpmn-io/diagram-js#932

Related to bpmn-io/diagram-js#932 and friends.

lmbateman · 2024-10-10T14:33:24Z

Is there a playground where I can try this out? Alternatively, if you need my review, could you please do a quick screen capture to demonstrate the changes? Thanks!

nikku · 2024-10-10T16:09:44Z

Cf. bpmn-io/bpmn-js#2235 (comment). I'll give you an E2E demo once I'm done with implementing the search, unless you want a demo earlier, to discuss what we do (did), and why.

nikku · 2024-10-31T17:22:53Z

I updated this PR to be fuzzy again, and solve all other outstanding issues.

Just now tests revealed that we're not fully fuzzy, actually match too many things, cf. original comment in the popup menu that I subscribe to:

// every word of `search term` should be included in one of the searchable fields

Our current behavior, leading to more search results popping up is:

// at least one word of `search term` should be included in one of the searchable fields

Need to do another round tomorrow.

nikku · 2024-10-31T18:15:49Z

Now ready to be reviewed.

lib/features/search/search.js

philippfromme

Great work! 🎖️

misiekhardcore

I did some review up to my limited knowledge. Mostly catching some smaller things

misiekhardcore · 2024-11-01T15:10:45Z

lib/features/search/search.js


-      results.splice(index, 0, result);
+  return items.flatMap((item, idx) => {


idx seems not to be used anywhere

Fixed via 5d1d756.

misiekhardcore · 2024-11-01T15:23:57Z

lib/features/popup-menu/PopupMenuComponent.js

@@ -54,6 +54,7 @@ export default function PopupMenuComponent(props) {
    scale,
    search,
    emptyPlaceholder,
+    searchFn,


can we have some type definition for this?

Also missing here, good catch.

Fixed via f1a8efe.

lib/features/search-pad/SearchPadProvider.ts

misiekhardcore · 2024-11-01T15:33:07Z

lib/features/search-pad/SearchPad.js

    } else {
-      htmlText += escapeHTML(t.normal);


so the normal property is not used anymore? What was the purpose of it before?

Good catch, that is a bug. 👍

Of course normal is still used, but we did not properly test cover it to date.

Bug fix: 32d8f0e

Test coverage added for rendering (different cases): 1fd6df6

test/spec/features/popup-menu/PopupMenuComponentSpec.js

test/spec/features/search-pad/SearchPadSpec.js

misiekhardcore · 2024-11-01T15:52:10Z

One general question that crossed my mind when I was playing with it:
Do we want to match partial searches? What I mean
There is an element named User task and user types in the search Ustk so few letters taken from the name in the order of occurance "User task".

nikku · 2024-11-04T08:59:45Z

Do we want to match partial searches?

This initiative aims to unify search across all places where we use it.

Given the new infrastructure in place (global search utility) integrators can choose what kind of search experience they want.

At the core we want to offer a search that is simple enough, downstream you may plug-in your fuzzy search provider, cf. camunda/camunda-bpmn-js#367.

The implementation for comparing entries did not properly account for entries that are "equal", and whenever they occur it ran into a loop. The implementation proposed does two things: * First we simplify the existing logic to be a two step filter + sort mechanism * Second, the search comparison is simple and robust, and handles equality well. This should fix the loop issues we're seeing. Related to camunda/web-modeler#10940

Reverts 34a65d32abc0c80cc822be13c440c61757ae6376

We ensure that for every entry that is a match we _always_ provide tokens. This ensures that a user can rely on these tokens being there, i.e. to render the results. Related to bpmn-io/bpmn-js#2235

This ensures we properly handle fuzzy results (again), where parts of the search is matched across different keys. This ensures we're not overly strict in filtering things. Related to bpmn-io/bpmn-js#2235

Tokens returned by `search` are now recognized by the component, this makes conversion simpler.

Co-authored-by: Michał Konopski <[email protected]>

`search` is now generic, items returned will match the type closure of searchedEntries (whatever that is).

nikku · 2024-11-04T16:54:50Z

Squashed the changes, @misiekhardcore ready for another look.

bpmn-io-tasks bot added the needs review Review pending label Sep 23, 2024

nikku requested a review from philippfromme September 23, 2024 16:27

bpmn-io-tasks bot assigned nikku Sep 23, 2024

nikku requested a review from barmac September 23, 2024 16:27

nikku mentioned this pull request Sep 23, 2024

chore: unbuild search integration in popup menu #933

Merged

4 tasks

nikku force-pushed the fix-search branch from 187d605 to ebbe69c Compare September 23, 2024 18:24

This comment was marked as outdated.

Sign in to view

nikku added a commit to camunda/camunda-modeler that referenced this pull request Sep 23, 2024

deps: update to [email protected]

a0de2c5

fix: correct popup menu search on duplicate entries Related to bpmn-io/diagram-js#932

nikku mentioned this pull request Sep 23, 2024

Release camunda-modeler v5.28.0 camunda/camunda-modeler#4521

Closed

41 tasks

nikku added a commit to camunda/camunda-bpmn-js that referenced this pull request Sep 23, 2024

deps: bump to [email protected]

2fc3987

Related to bpmn-io/diagram-js#932 and friends.

nikku added the in progress Currently worked on label Sep 25, 2024 — with bpmn-io-tasks

nikku removed the needs review Review pending label Sep 25, 2024

barmac marked this pull request as draft September 27, 2024 13:07

nikku mentioned this pull request Oct 2, 2024

Use search bpmn-io/bpmn-js#2235

Merged

4 tasks

nikku requested a review from lmbateman October 10, 2024 14:03

nikku added the ready Ready to be worked on label Oct 11, 2024 — with bpmn-io-tasks

nikku removed the in progress Currently worked on label Oct 11, 2024

nikku force-pushed the fix-search branch 2 times, most recently from 9b3228e to 1fbd403 Compare October 31, 2024 16:44

nikku changed the title ~~fix(search): correctly handle duplicate entries~~ Improve search (make fuzzy again and more robust) Oct 31, 2024

nikku marked this pull request as ready for review October 31, 2024 16:55

bpmn-io-tasks bot added needs review Review pending and removed ready Ready to be worked on labels Oct 31, 2024

nikku requested a review from GeethaParthasarathy October 31, 2024 16:56

nikku force-pushed the fix-search branch 2 times, most recently from 6c502b7 to e1fe8ef Compare October 31, 2024 17:04

nikku force-pushed the fix-search branch 2 times, most recently from a68c85e to 08996a2 Compare November 1, 2024 12:38

philippfromme reviewed Nov 1, 2024

View reviewed changes

lib/features/search/search.js Outdated Show resolved Hide resolved

philippfromme self-requested a review November 1, 2024 15:08

philippfromme approved these changes Nov 1, 2024

View reviewed changes

nikku force-pushed the fix-search branch from 08996a2 to 688cc11 Compare November 1, 2024 15:19

misiekhardcore reviewed Nov 1, 2024

View reviewed changes

nikku force-pushed the fix-search branch from 5d1d756 to 120b93d Compare November 4, 2024 16:45

nikku and others added 12 commits November 4, 2024 17:51

test(search): verify partial match

39c2ca3

feat(popup-menu): integrate popup menu into search component

8be7291

Reverts 34a65d32abc0c80cc822be13c440c61757ae6376

test(popup-menu): restore fuzzy search test

38e72e0

fix(search): search with whitespace

147fd00

feat(search): always provide tokens

cd70b64

We ensure that for every entry that is a match we _always_ provide tokens. This ensures that a user can rely on these tokens being there, i.e. to render the results. Related to bpmn-io/bpmn-js#2235

feat(search): find search terms across all keys

3b95628

This ensures we properly handle fuzzy results (again), where parts of the search is matched across different keys. This ensures we're not overly strict in filtering things. Related to bpmn-io/bpmn-js#2235

test: correct search spec title

5f9767b

style: simplify

2278d1f

feat(search-pad): recognize modern tokens

4e235fd

Tokens returned by `search` are now recognized by the component, this makes conversion simpler.

test(popup-menu): simplify spec

bc549ac

Co-authored-by: Michał Konopski <[email protected]>

feat(search): improve types

5c8a253

`search` is now generic, items returned will match the type closure of searchedEntries (whatever that is).

nikku force-pushed the fix-search branch from 120b93d to 5c8a253 Compare November 4, 2024 16:54

nikku requested a review from misiekhardcore November 4, 2024 16:54

nikku merged commit 3ab3be3 into develop Nov 4, 2024
12 checks passed

nikku deleted the fix-search branch November 4, 2024 20:32

bpmn-io-tasks bot removed the needs review Review pending label Nov 4, 2024

nikku mentioned this pull request Nov 4, 2024

Create/append anything should display exact match first camunda/camunda-modeler#4122

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve search (make fuzzy again and more robust) #932

Improve search (make fuzzy again and more robust) #932

nikku commented Sep 23, 2024 •

edited

Loading

nikku commented Sep 23, 2024

This comment was marked as outdated.

lmbateman commented Oct 10, 2024

nikku commented Oct 10, 2024

nikku commented Oct 31, 2024 •

edited

Loading

nikku commented Oct 31, 2024

philippfromme left a comment

misiekhardcore left a comment

misiekhardcore Nov 1, 2024

nikku Nov 4, 2024

misiekhardcore Nov 1, 2024

nikku Nov 4, 2024

nikku Nov 4, 2024 •

edited

Loading

misiekhardcore Nov 1, 2024

nikku Nov 4, 2024

nikku Nov 4, 2024

misiekhardcore commented Nov 1, 2024

nikku commented Nov 4, 2024

nikku commented Nov 4, 2024


		results.splice(index, 0, result);
		return items.flatMap((item, idx) => {

Improve search (make fuzzy again and more robust) #932

Improve search (make fuzzy again and more robust) #932

Conversation

nikku commented Sep 23, 2024 • edited Loading

Proposed Changes

Checklist

nikku commented Sep 23, 2024

This comment was marked as outdated.

lmbateman commented Oct 10, 2024

nikku commented Oct 10, 2024

nikku commented Oct 31, 2024 • edited Loading

nikku commented Oct 31, 2024

philippfromme left a comment

Choose a reason for hiding this comment

misiekhardcore left a comment

Choose a reason for hiding this comment

misiekhardcore Nov 1, 2024

Choose a reason for hiding this comment

nikku Nov 4, 2024

Choose a reason for hiding this comment

misiekhardcore Nov 1, 2024

Choose a reason for hiding this comment

nikku Nov 4, 2024

Choose a reason for hiding this comment

nikku Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

misiekhardcore Nov 1, 2024

Choose a reason for hiding this comment

nikku Nov 4, 2024

Choose a reason for hiding this comment

nikku Nov 4, 2024

Choose a reason for hiding this comment

misiekhardcore commented Nov 1, 2024

nikku commented Nov 4, 2024

nikku commented Nov 4, 2024

nikku commented Sep 23, 2024 •

edited

Loading

nikku commented Oct 31, 2024 •

edited

Loading

nikku Nov 4, 2024 •

edited

Loading