`ArrowTypeError` when reading a parquet dataset that's partitioned on a `large_string` column

[TomAugspurger]

Describe the bug, including details regarding any error messages, version, and platform.

When using pyarrow.dataset.dataset to read a partitioned parquet dataset that's partitioned on a large_string field, I see a ArrowTypeError: Unable to merge: Field part has incompatible types: large_string vs string:

import pyarrow as pa
import pyarrow.dataset
import pathlib
import pyarrow.parquet


t = pa.table(
    {"part": pa.array(["a", "a", "b", "b"], type=pa.large_string()), "col": [1, 2, 3, 4]}
)
root = pathlib.Path("string.parquet")
a = root / "a/data.parquet"
b = root / "b/data.parquet"

a.parent.mkdir(parents=True, exist_ok=True)
b.parent.mkdir(parents=True, exist_ok=True)

pyarrow.parquet.write_table(t[:2], a)
pyarrow.parquet.write_table(t[2:], b)

source = list(root.glob("**/*.parquet"))
t = pyarrow.dataset.dataset(root, partitioning=["part"], partition_base_dir=str(root))

errors with

Traceback (most recent call last):
  File "/Users/toaugspurger/gh/dask/dask/debug.py", line 21, in <module>
    t = pyarrow.dataset.dataset(root, partitioning=["part"], partition_base_dir=str(root))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/dataset.py", line 790, in dataset
    return _filesystem_dataset(source, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/dataset.py", line 482, in _filesystem_dataset
    return factory.finish(schema)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 3196, in pyarrow._dataset.DatasetFactory.finish
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Unable to merge: Field part has incompatible types: large_string vs string

I don't observe that failure with type=pa.string(), just large_string().

Component(s)

C++, Python

[raulcd]

Hi @TomAugspurger thanks for opening the issue, is this something new for Arrow / PyArrow v 21.0.0 or is this something that was already happening on Arrow v20.0.0?

[TomAugspurger]

Thanks @raulcd. I see the error going back to 18.1.0 with uv run --with "pyarrow==18.1.0" debug.py (I think there might not be a pyarrow wheel for my version of python going further back, so I stopped there).

We're only seeing it in Dask now because pandas is changing its default string type to use pyarrow arrays with type=pyarrow.large_string().

And I should forgot to mention, I think that this is a supported use case, but I don't think I could reproduce this when using pyarrow's built-in partitioned parquet writing. I'll put up an example with that.

[TomAugspurger]

This script shows that using pyarrow.parquet.write_to_dataset works fine:

import pyarrow as pa
import pyarrow.dataset
import pathlib
import pyarrow.parquet
import shutil


shutil.rmtree("string.parquet", ignore_errors=True)
shutil.rmtree("ds.parquet", ignore_errors=True)
t = pa.table(
    {"part": pa.array(["a", "a", "b", "b"], type=pa.large_string()), "col": [1, 2, 3, 4]}
)
root = pathlib.Path("string.parquet")
a = root / "a/data.parquet"
b = root / "b/data.parquet"
a.parent.mkdir(parents=True, exist_ok=True)
b.parent.mkdir(parents=True, exist_ok=True)

# Manually write the two parts to disk using `write_table`

pyarrow.parquet.write_table(t[:2], a)
pyarrow.parquet.write_table(t[2:], b)

source = list(root.glob("**/*.parquet"))

# Use write_to_dataset to let pyarrow handle the partitioning
ds_root = pathlib.Path("ds.parquet")
pyarrow.parquet.write_to_dataset(t, ds_root, partition_cols=["part"])

ds_source = list(ds_root.glob("**/*.parquet"))
ds = pyarrow.dataset.dataset(ds_source, partitioning=["part"], partition_base_dir=str(ds_root))

print("manual")
print(pyarrow.parquet.read_table(source[0]))

print("\n\ndataset")
print(pyarrow.parquet.read_table(ds_source[0]))

That prints out

manual
pyarrow.Table
part: large_string
col: int64
----
part: [["a","a"]]
col: [[1,2]]


dataset
pyarrow.Table
col: int64
part: dictionary<values=string, indices=int32, ordered=0>
----
col: [[1,2]]
part: [  -- dictionary:
["a"]  -- indices:
[0,0]]

So the big difference is that pyarrow.parquet.write_to_dataset(..., partition_on=) will dictionary encode the partition keys.