write_dataset only writes the first batch

[mesejo]

I have this snippet of code:

from pathlib import Path

import pyarrow as pa
import pyarrow.dataset as ds


tempdir = Path(__file__).parent / 'temp'

table = pa.table({"a": range(1024)})
batches = table.to_batches(max_chunksize=2)
ds.write_dataset(batches, tempdir, format="parquet", preserve_order=True, use_threads=True)

I was expecting it to write all the batches, but it is only writing the first batch:

$ ls temp
part-0.parquet

Is this the expected output? I'm working with Python 3.11 and pyarrow 21.0.0

$ ipython
Python 3.11.11 (main, Feb  5 2025, 19:11:07) [Clang 19.1.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can change the editing mode of IPython to behave more like vi, or emacs.

In [1]: import pyarrow as pa

In [2]: pa.__version__
Out[2]: '21.0.0'

In [3]: 

[adamreeve]

Hi @mesejo. What's the contents of part-0.parquet? I'd expect it to contain all 1024 rows. write_dataset doesn't partition data into separate files for each input batch. Maybe you want to use the max_rows_per_file parameter if you're trying to reduce the size of files?

[mesejo]

Indeed, the file contains all the data.