Get "not enough values to unpack" error from BM25EmbeddingFunction with scipy v1.15.0

[yhmo]

Script to reproduce:

from langchain_milvus.utils.sparse import BM25SparseEmbedding
from typing import List

from milvus_model.sparse import BM25EmbeddingFunction


class MyBM25SparseEmbedding(BM25SparseEmbedding):
    def __init__(self, corpus: List[str] = None):
        if corpus is not None:
            super().__init__(corpus=corpus)

    def load(self, save_path: str):
        self.bm25_ef = BM25EmbeddingFunction()
        self.bm25_ef.load(save_path)

    def save(self, save_path: str):
        self.bm25_ef.save(save_path)


bm25_store_file = 'bm25.json'  # Name of the file to save the sparse embeddings

docs = ["hello hello world", "welcome to wonderful wonderful world"]

sparse_embedding_func = MyBM25SparseEmbedding(docs)
sparse_embedding_func.save(bm25_store_file)
print(sparse_embedding_func.bm25_ef.idf)
print("=====================================")
print(docs)
print("=====================================")
print(sparse_embedding_func.embed_documents(docs)) # Error here

With scipy version== 1.14.1, it works fine.
With scipy version==1.15.0, it throws this error:

Traceback (most recent call last):
  File "/Users/test.py", line 28, in <module>
    print(sparse_embedding_func.embed_documents(docs))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test_venv/lib/python3.12/site-packages/langchain_milvus/utils/sparse.py", line 45, in embed_documents
    return [self._sparse_to_dict(sparse_array) for sparse_array in sparse_arrays]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test_venv/lib/python3.12/site-packages/langchain_milvus/utils/sparse.py", line 48, in _sparse_to_dict
    row_indices, col_indices = sparse_array.nonzero()
    ^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

[yhmo]

User reported this problem from here: milvus-io/milvus#40954

[yhmo]

I think the root cause is in scipy. The milvus_model.sparse.BM25EmbeddingFunction calls scipy.sparse.vsstack.tocsr() to generate a sparse array. There must be a behavior change for scipy.sparse.vsstack in scipy 1.15.0. The output of scipy.sparse.vsstack.tocsr() is changed.

A simple script to verify the scipy. With scipy version== 1.14.1, it works fine. With scipy version==1.15.0, it throws error "not enough values to unpack":

import numpy as np
from scipy.sparse import csr_array, vstack

sparse_embs = []

values = [1.0687022900763359, 1.4973262032085561]
rows = [0, 0]
cols = [1, 0]
sparse = csr_array((values, (rows, cols)), shape=(1, 4)).astype(np.float32)
sparse_embs.append(sparse)

values = [1.3658536585365855, 0.9395973154362417, 0.9395973154362417]
rows = [0, 0, 0]
cols = [3, 1, 2]
sparse = csr_array((values, (rows, cols)), shape=(1, 4)).astype(np.float32)
sparse_embs.append(sparse)

kk = vstack(sparse_embs).tocsr()
print(kk)
print("==========================")
for s in kk:
    row_indices, col_indices = s.nonzero()
    print(row_indices, col_indices)

[xiaofan-luan]

should we fix this and upgrade scipy to later version?

[zc277584121]

I will take a look

[zc277584121]

It's a compatibility issue in langchain_milvus, fixed at: https://github.com/langchain-ai/langchain-milvus/pull/57/files
Also, added deprecated warning at: langchain-ai/langchain-milvus@300cf23