Search: Migrate to Elastic Search (instead of Elastic App Search)

[wei2912]

We've been using Elastic Search UI for a while, but we're now running into the following limitations:

1. No infinite scroll support ([MPT-45] Mentors: Switch to single-page scroll #645)
2. Difficult to link to individual mentors (see [MPT-65] Attempt to create shareable link #731)
3. Customisation of Elastic Search queries
4. [MPT-13] Search: Filtering by certain combinations removes facets from the screen #565

A possible alternative might be https://www.searchkit.co/, but more research needs to be done first.

[linear]

MPT-125 Search: Migrate to alternative search library

[dabby9734]

• searchkit seems to offer much of the same functionality of Elastic Search UI
  .
• There doesn't seem to be an easy way to implement infinite scroll (and it does not appear practical) due to reasons shared in [MPT-45] Mentors: Switch to single-page scroll #645
• The issue with linking to specific mentors seems to be related to Elastic Search UI performing its own transformations on the URL (which overwrites our extra query uid=...). While investigating the possibility of customising Elastic Search UI's transformation of URLs, I came across a NextJS Integration Guide. However, it's written for the NextJS Pages router, so I've been having some trouble adapting it to the App Router so that I can experiment with it using our setup.
• Need more elaboration on the 'customisation of Elastic Search queries'. What kind of customisation are we looking at?

[wei2912]

There doesn't seem to be an easy way to implement infinite scroll (and it does not appear practical) due to reasons shared in #645

Searchkit seems to have an implementation at https://www.searchkit.co/docs/components/results/infinite-hits. This should be theoretically possible, just that Elastic Search UI doesn't support it. https://discuss.elastic.co/t/infinite-scroll-best-practices-with-es/20490 lays out some lays out some potential issues, but this is more to do with the use of scroll queries. On the UI side, infinite scroll would be equivalent to just appending the next page's results rather than direct to the page.

Need more elaboration on the 'customisation of Elastic Search queries'. What kind of customisation are we looking at?

TBC, but we're looking into implementing semantic search. The current proposal requires Reciprocal Rank Fusion (RRF) which basically combines the results of different searches to form a final search list.

[wei2912]

Based on discussion with @IvanFengJK:

1. Might want to change modal to page? (see how the routing works)
2. See other search solns that might possibly be more suitable
3. OR instead of shareable link, consider "bookmarking" mentors instead?

[wei2912]

@dabby9734 @Unknownflow I've also added #565 to the list of things we'd want to tackle with the migration.

[dabby9734]

It appears that AppSearch (which we are using) is being deprecated. See this link. This may affect how we configure our connection to our Elasticsearch instance.

[dabby9734]

Some comments post SPRINT on 4 Jan 2025:

Adding on a breakdown of the features we are looking at for our search engine.

Currently implemented in Elastisearch

• Searchbox (to search for: 1) name, 2) company (=organisation), 3) role, 4) school, 5) full_bio, 6) course_of_study)
• Autocomplete for the search box
• Highlighting of matched keywords from search in the relevant fields
• Sort by 1) relevance and 2) name
• Search facets: 1) industries, 2) organisations, 3) course of study
• Clear all filters button
• Show N number of results dropdown
• Pagination

Potential enhancements:

• Infinite scroll
• Semantic search implementation with Reciprocal Rank Fusion (RRF)

cc @wei2912 @jcc-kh, please comment if I've missed anything or if there's something y'all want to add