Noob question: what sets Inx and MeiliSearch apart?

[dsaghliani]

I found Inx through Datacake and Datacake through a link on MeiliSearch's Discord. The two sell themselves as the same thing—a search engine written in Rust—so I'm a bit confused. Are the two in truth different things? If not, if you had a give a ballpark estimate, how do they compare in different metrics, such as performance and maturity?

This isn't meant as a criticism, by the way. ☺️

[ChillFish8]

Hi 👋

They do loosely target the same thing, although lnx can in theory be used for a wider range of tasks in situations outside of fuzzy searching unlike meili.

From a high level, if you only have a couple million docs at most (say ~20GB or less for decent performance, the higher you go beyond that the harder it'll be to scale) then you can probably get away with using Meilisearch, it'll be a simple, convenient system to use. The flip side is sizes beyond that or situations where you want the better search and indexing performance.

Implementation differences:

• Meilisearch uses a custome engine and produces a single backed by LMDB.
• lnx is built upon tantivy which is a very mature implementation akin to lucene.
• lnx's fast-fuzzy mode is a much different algorithm for user facing search than the traditional levenstein distance on an FST which gives it much better performance.
• Meilisearch probably has slightly better relevancy although it depends on what you're looking for I suppose, fast-fuzzy should give you pretty good results.

Performance differences:

• Meilisearch has some limitations still especially around compression which leads to incredibly high write amplification.
• lnx has incremental indexing which has a fairly constant rate of ingestion vs Meilisearch's almost exponential indexing time increase as the index gets bigger. For reference lnx can process about 8-10MB/s per CPU core if your disks are good, which puts it on the side of being one of the fastest engines for ingestion rate overall, a few times faster than elasticsearch and depending on size, potentially and order of magnitude faster than Meilisearch.
• lnx has much better index compression, taking less space up on disk, under normal circumstances you can probably expect size_of_data_to_ingest = size_of_compressed_files_total so it's fairly predictable to work out how much disk space you need, i.e. a 20GB index probably wants slightly over 20GB of disk space to keep everything.
• Search wise lnx has a much better search performance compared to meili (or most other engines for that matter) with higher throughput and lower latency per core.

A benchmark I did a while ago on the Meilisearch movies dataset (only 20k docs so very small) got the following results:

It's worth noting here however that these benchmarks were done about 6 months ago or so and in the meantime all the engines mentioned in those graphs have had optimisations and improvements done in the mean time so I encourage you to only take them as a grain of salt for the most part unless you run the benchmark yourself.

[dsaghliani]

Thanks for the detailed breakdown! Inx looks really impressive.