Can I define a filesystem using fsspec?

[pawarbi]

Similar to pyarrow and polars, I want to define a local filesystem using fsspec. How do I do that in daft? e.g. in pyarrow
`filesystem_code = "xxx"

storage_options = {
"account_name": "xxx",
"account_host": "xxxxx.com"
}

fs = fsspec.filesystem(filesystem_code, **storage_options)

table2 = pq.read_table(local_file_path, filesystem=fs)`

can I do the same in daft ?

[samster25]

In Daft, we actually have native read support for the major cloud vendors built in via Rust! Take a look at our S3 example: https://www.getdaft.io/projects/docs/en/latest/user_guide/integrations/aws.html

[kevinzwang]

Hi @pawarbi, are you trying to access a Fabric endpoint that does not need authentication? In which case, it should just work if you do not specify any credentials, but if not, you can also try setting anonymous=True in the AzureConfig to force it to not use any credentials.

This is what it would look like:

from daft.io import IOConfig, AzureConfig

io_config = IOConfig(
    azure=AzureConfig(
        storage_account="xxx",
        use_fabric_endpoint=True, # or you can specify a custom endpoint using `endpoint_url="xxxxx.com"`
        anonymous=True
    )
)

df = daft.read_parquet(..., io_config=io_config)

[pawarbi]

Thank you @kevinzwang . I can access if I use the relative file path but abfss still does not work. Below is the error I get. Could it be that the endpoint url you are using should be onelake.dfs.fabric.microsoft.com instead of onelake.blob.fabric.microsoft.com. You can see here . If I use endpoint_url instead and set it to onelake.dfs.fabric.microsoft.com, I get an error :

DaftCoreException: DaftError::External Unable to open file abfss://[email protected]/Files/customer.csv: Error { context: Custom(Custom { kind: DataConversion, error: RelativeUrlWithoutBase }) }

[kevinzwang]

Fabric files typically can be queried via their onelake.blob.fabric.microsoft.com endpoint, but could you try setting endpoint_url to https://onelake.dfs.fabric.microsoft.com (I added the https:// to the beginning).

I'm not sure why there would be an SSL error though, could you make sure the root certificate on your machine is up-to-date?

[pawarbi]

Thanks. Adding https:// led to same error.
There is no certificate to handle in Fabric, auth is done automatically. I am no expert in it but if you are familiar with fsspec and arrow, can you see how they are handling it?
Thanks again for your help. I am going to ask another user to try just to make sure it's not just me.

[kevinzwang]

@pawarbi can you also verify that querying the same URL on the same machine works with other libraries?