API for reading/converting large dense/sparse format in Finch

[SEICS]

Hi,

Thanks for the great work on Finch.jl!

I’m trying to use Finch.jl as the execution backend for my project. My input tensors are model parameters trained in Python:

• Sparse tensors: stored in .pkl using Python’s sparse.COO format
• Dense tensors: stored in .npz using NumPy arrays

My current idea is to store all raw arrays in .npz and keep tensor type metadata in a .json manifest, so after reading into Julia, I can convert them into desired tensor formats in Finch.

However, the existing Finch tensor constructors seem to create tensors from scratch, and I couldn’t find an API that directly wraps or converts external tensor formats (not already Finch tensors) into Finch-compatible formats—especially for large sparse data.

Question:
Is there an existing or recommended way to read large sparse tensors from formats like .npz/.json (or even Python’s sparse.COO) and convert them efficiently into Finch tensor formats?

Any tips, helper functions, or example code for this workflow would be greatly appreciated.

Thanks!

[willow-ahrens]

Thanks for your interest in the project!

You may find the binsparse file format helpful: https://finch-tensor.github.io/Finch.jl/dev/docs/fileio/ https://github.com/GraphBLAS/binsparse-specification You could write the file to binsparse using pydata/sparse or https://github.com/ivirshup/binsparse-python, and read with julia.

You may also wish to use the python bindings we have for finch here: https://github.com/finch-tensor/finch-tensor-python. These bindings support both fileio and high-level operators in the array api.

If you're interested, I'd be happy to discuss more and learn about your use case.

[SEICS]

Hi,

Thanks for the help!

I have looked into these repositories and the corresponding papers, but I am quite confused on how to use the Binsparse with [its python API (i.e., Binsparse)](https://github.com/ivirshup/binsparse-python, since I have no experience with HDF5 format. I installed the packages following the instructions, and tried to run the given example:

csr = scipy.sparse.random(1000, 100, format="csr", density=0.1, random_state=np.random.default_rng())
z = zarr.group()
binsparse.write(z, "csr", csr)
binsparse.read(z["csr"])

but got the error below:

>>> binsparse.write(z, "csr", csr)
Traceback (most recent call last):
  File "<python-input-8>", line 1, in <module>
    binsparse.write(z, "csr", csr)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/jw/Desktop/binsparse-python/src/binsparse/_io/api.py", line 52, in write
    write_csr(store, key, x, dataset_kwargs=dataset_kwargs)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jw/Desktop/binsparse-python/src/binsparse/_io/methods.py", line 55, in write_csr
    write_attrs(group, "binsparse", metadata)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jw/anaconda3/envs/py313/lib/python3.13/functools.py", line 934, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/jw/Desktop/binsparse-python/src/binsparse/_io/utils.py", line 10, in write_attrs
    raise NotImplementedError(f"no implementation for type {type(element)}")
NotImplementedError: no implementation for type <class 'zarr.core.group.Group'>
>>> 

Based on the documentation, I should expect to get:

<1000x100 sparse matrix of type '<class 'numpy.float64'>'
	with 10000 stored elements in Compressed Sparse Row format>

I am using this version but not sure if this is the right one to use, and I am using Python3.13:

>>> binsparse.__version__
'0.0.1.dev0'

Also, I have tried to use Pydata's sparse library but API seems to be not really there, where z = zarr.group():

Traceback (most recent call last):
  File "/Users/jw/Desktop/bayestensor/progs/aqp/ssb/finch/store_cpds.py", line 22, in <module>
    binsparse.write(z, "customer_c_custkey", customer_c_custkey)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jw/Desktop/binsparse-python/src/binsparse/_io/api.py", line 58, in write
    raise NotImplementedError(f"no implementation for type {type(x)}")
NotImplementedError: no implementation for type <class 'sparse._coo.core.COO'>

Apart from above issues, I tried to store several scipy.sparse.csc_matrix matrices in .bsp.h5 format under the same group with default path. I didn't have any error reported during writing:

with h5py.File("progs/aqp/ssb/finch/cpds.bsp.h5", "w") as f:
    for (k, v) in ssb_cpds.items():
        if isinstance(v, np.ndarray):
            continue
        else:
            spt = scipy.sparse.csc_matrix(v.todense()) # v is a Pydata's sparse.COO matrix
            binsparse.write(f, k, spt)   # Writes into file
f.close()

but when I tried to load them in Finch:
cpds = Finch.bspread(h5open("progs/aqp/ssb/finch/cpds.bsp.h5","r")).
it reports:

ERROR: LoadError: KeyError: key "binsparse" not found
Stacktrace:
 [1] getindex(x::HDF5.Attributes, name::String)
   @ HDF5 ~/.julia/packages/HDF5/Z859u/src/attributes.jl:379
 [2] bspread_header(f::HDF5.File)
   @ HDF5Ext ~/.julia/packages/Finch/tUiDI/ext/HDF5Ext.jl:22
 [3] bspread(f::HDF5.File)
   @ Finch ~/.julia/packages/Finch/tUiDI/src/interface/fileio/binsparse.jl:330
 [4] top-level scope
   @ ~/Desktop/bayestensor/progs/aqp/ssb/finch/read_tensors.jl:4
in expression starting at /Users/jw/Desktop/bayestensor/progs/aqp/ssb/finch/read_tensors.jl:4

To my understanding, the metadata .json file with key "binsparse" for .bsp.h5 should be automated by binsparse-python itself with binsparse.write(store, key, x, *, dataset_kwargs=mappingproxy({})). I am not sure where I did wrong. A brief example with either SciPy.sparse.coo_matrix or Pydata's sparse.COO would be really helpful. Thanks.

[willow-ahrens]

I think the python binsparse writer may not be up to date, we added the binsparse tag in a later version. I'll file an issue for that parser, and bring it up at the binsparse meeting series.

[SEICS]

That would be great! Any update would be much appreciated.

[hameerabbasi]

From what I can tell, that binsparse parser doesn't support Finch or N-D arrays out of the box. It may have extension points we can use; I have to check.