Support a <=> (cosine similarity) operator for bm25vector

[mfolnovic]

Summary

It would be great to add a <=> operator that computes cosine similarity between two bm25vector, so that users can both rank and filter results based on a similarity threshold.

Motivation

• Filter out irrelevant docs: BM25 always returns some results, even if the query and documents are completely unrelated.
• Combine ranking + thresholding: We want to both sort by similarity and exclude anything below a given cosine-similarity cutoff in a single pass.

Example

select document_id,
    embedding <&> to_bm25query('embedding_bm25', tokenize('document content', 'default')) as rank,
    from embedding
    where (embedding <=> tokenize('document content', 'default')) >= 0.8
    order by rank
    limit 5

Alternate solution

We currently have an implementation that runs a query to get embedding of query document, then manually computes cosine similarity between pairs. This requires parsing the outputs of tokenize and the embedding column and implementing sparse‑vector cosine similarity ourselves.

Thank you for awesome product! :)

[VoVAllen]

Hi, I don't think cosine similarity has meaningful usage with bm25vector. Can you elaborate your scenarios?