Request: exact search

[saifulmuhajir]

Hi!

It looks like this extension doesn't support exact search, for a keyword. I see it has equality but only for bm25vector data type.

I think it would be a good feature since sometimes we also want to search for an exact word.

Thanks

[VoVAllen]

Hi, can you describe what kind of exact search are you looking for, something like tsvector @@@ search? Do you need BM25 ranking for the exact search?

[VoVAllen]

For now, you can try SET bm25_catalog.bm25_limit=-1, which will return all the documents whose scores are great than zero, like

SET bm25_catalog.bm25_limit = -1;
SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL', 'tokenizer1')) AS rank
FROM documents
ORDER BY rank
LIMIT 100000; -- large enough to get all the results

[saifulmuhajir]

I guess my question is for this case:

I need to return data matching foo_3prudu83hdDhT but NOT foo_3prudu83hdDCc (last two letter difference). This is needed because they could contain sensitive data. Using ranking, the difference is quite small, -1 difference or so. So, I'm not sure what's the best way to do it, hence my thought was maybe exact search of a keyword.

[VoVAllen]

You should use bigram or trigram tokenizer for this. @silver-ymz Can we do ngram tokenizer now?

[VoVAllen]

like https://www.postgresql.org/docs/current/pgtrgm.html

[silver-ymz]

Can we do ngram tokenizer now?

No, but we can simply update pg_tokenizer to support the ngram tokenizer.

[VoVAllen]

@saifulmuhajir I think I misunderstood what you want. For exact match, does btree index on text column work for your?