How to migrate to 0.3 for APIs that expect PathLike

[Zan-L]

As an example, Polars used to work perfectly with UPath < 0.3:

path = UPath('s3a://...')
df = pl.read_parquet(path)

But after 0.3 it stops working. The problem is, none of the 3 solutions works from the migration guide. Any suggestions?

[ap--]

Hi @Zan-L

cross-referencing: pola-rs/polars#24921

In the example you provided, they converted remote UPaths to a string via indirectly calling os.fspath() via os.path.expanduser() see here: https://github.com/pola-rs/polars/blob/77254fce44d7b848aa9aa8bb3a0f01f5d2e63d1c/py-polars/src/polars/_utils/various.py#L235
It's better to be explicit about how you're calling this API, instead of relying on the UPath being cast to string (and losing its storage options).

Your example code is effectively equivalent to:

path = UPath('s3a://...')
df = pl.read_parquet(path.as_uri())

Warning

But be aware !! you're dropping all storage_options in the general case (in your example there are none).

I would recommend to use it like this, if you want to use fsspec and s3fs as the read backend (probably slower than polars default):

import upath
import polars

p = upath.UPath("s3://bucket/file", anon=True)

polars.scan_parquet(p.open('rb'))

Or if you want to use the polars rust backends

import upath
import polars

p = upath.UPath("s3://bucket/file", anon=True)

object_store_storage_options = ...  # convert your fsspec p.storage_options to the object store style

polars.scan_parquet(p.as_uri(), storage_options=object_store_storage_options)

Warning

Because polars uses object-store instead of fsspec and s3fs, the storage options have different names and you have to convert: https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html

I'll add this to the migration guide when I find some time.

Cheers,
Andreas