Skip to content

Fix polars remote csv panic #744#754

Merged
mgrover1 merged 5 commits intomainfrom
744-remote-panic
Dec 11, 2025
Merged

Fix polars remote csv panic #744#754
mgrover1 merged 5 commits intomainfrom
744-remote-panic

Conversation

@charles-turner-1
Copy link
Collaborator

@charles-turner-1 charles-turner-1 commented Oct 23, 2025

Change Summary

  • Wrap pl.scan_csv in fsspec context handler.

Related issue number

#744

Checklist

  • Unit tests for the changes exist
  • Tests pass on CI
  • Documentation reflects the changes where applicable N/A

@charles-turner-1 charles-turner-1 changed the title Fix polars remote csv panic: Fix polars remote csv panic #744 Oct 23, 2025
@chiaweh2 chiaweh2 mentioned this pull request Nov 26, 2025
1 task
@mgrover1
Copy link
Collaborator

@charles-turner-1 - you good if I merge this?

@charles-turner-1
Copy link
Collaborator Author

Go for it!

@mgrover1 mgrover1 merged commit 347205c into main Dec 11, 2025
9 of 10 checks passed
@mgrover1 mgrover1 deleted the 744-remote-panic branch December 11, 2025 02:30
mgrover1 added a commit that referenced this pull request Dec 12, 2025
* add kerchunk engine ability provided by xarray

* add lazy load option in `read_kwarg`

polar not supporting http csv link that does not have content length but pandas dose. the kwarg is popped to avoid interfering with the later use in read_csv

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add needed packages

kerchunk for kerchunk engine
url access need request and aiohttp
fastparquet for parquet reference file read

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add lazy read description in the esm_datastore docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove changes that can be solved with #754

* add the PR fix to the newest version

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Max Grover <mgroverwx@gmail.com>
charles-turner-1 pushed a commit to ACCESS-NRI/intake-esm that referenced this pull request Jan 9, 2026
* add kerchunk engine ability provided by xarray

* add lazy load option in `read_kwarg`

polar not supporting http csv link that does not have content length but pandas dose. the kwarg is popped to avoid interfering with the later use in read_csv

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add needed packages

kerchunk for kerchunk engine
url access need request and aiohttp
fastparquet for parquet reference file read

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add lazy read description in the esm_datastore docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove changes that can be solved with intake#754

* add the PR fix to the newest version

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Max Grover <mgroverwx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic in Polars when loading ESM catalog from S3-hosted CSV (begin <= end (10 <= 9) slicing error)

2 participants