buildin_httpfs true is slow as it seems to fetch all the data

[fcomte]

What happens?

I notice that the behaviour of http request on parquet file is very slow as each request download all the file even for counting row of a parquet file.

To Reproduce

use https://duckdb.org/docs/stable/clients/wasm/overview

test with :
SET builtin_httpfs = false; => very efficient (partial fetch)
SET builtin_httpfs = true; => very slow (full download)

to access this king of file :

select count(*) from 'https://my-domain/myfile.parquet';

Browser/Environment:

Device:

DuckDB-Wasm Version:

1.4

DuckDB-Wasm Deployment:

shell duckdb

Full Name:

Frédéric Comte

Affiliation:

Insee

[carlopi]

Can you share a link to a file that exibits this behaviour?
And check how DuckDB native behaves in that case?

[fcomte]

https://minio.lab.sspcloud.fr/tm8enk/diffusion/RP/2016/individu_reg/individu_reg.parquet

no pb on duckdb native

[fcomte]

we develop onyxia and notice the regression on our open instance https://datalab.sspcloud.fr/data-explorer?source=https%3A%2F%2Fminio.lab.sspcloud.fr%2Ftm8enk%2Fdiffusion%2FRP%2F2016%2Findividu_reg%2Findividu_reg.parquet

[carlopi]

Could you try with:

LOAD httpfs;
SELECT count(*) FROM 'https://minio.lab.sspcloud.fr/tm8enk/diffusion/RP/2016/individu_reg/individu_reg.parquet';

this to me does the exact same request as native, that is single HEAD + single GET.

[carlopi]

Shareable link

[fcomte]

yes this is working , we did not use the LOAD httpfs
have an idea about the behaviour without it ?

[carlopi]

No, it looks indeed wrong, as a workaround I'd say just LOAD httpfs explicitly, that is the direction where this will move, but I'll look into why this is not autoloaded.