Panic in Polars when loading ESM catalog from S3-hosted CSV (begin <= end (10 <= 9) slicing error)

[BaptisteFrancois]

Description

When attempting to load an ESM catalog from a CSV file hosted on S3 using intake-esm, the process fails with a panic from the polars backend. The error appears to be related to slicing logic in polars-io.

What I Did

import intake
cat = intake.open_esm_datastore("https://cadcat.s3.amazonaws.com/cae-zarr.csv")

Traceback

thread '<unnamed>' (14708) panicked at crates\polars-io\src\path_utils\mod.rs:427:63:
begin <= end (10 <= 9) when slicing `cadcat.s3.amazonaws.com/cae-zarr.csv`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\core.py", line 124, in __init__
    self.esmcat = ESMCatalogModel.load(
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 265, in load
    cat._cast_agg_columns_with_iterables()
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 369, in _cast_agg_columns_with_iterables
    self.columns_with_iterables.intersection(
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 354, in columns_with_iterables
    return self._frames.columns_with_iterables  # type: ignore[union-attr]
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 582, in columns_with_iterables
    if (trunc_df := self.lazy.head(1).collect()).is_empty():
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\_utils\deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\lazyframe\opt_flags.py", line 330, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\lazyframe\frame.py", line 2407, in collect
    return wrap_df(ldf.collect(engine, callback))
pyo3_runtime.PanicException: begin <= end (10 <= 9) when slicing `cadcat.s3.amazonaws.com/cae-zarr.csv`

Version information: output of intake_esm.show_versions()

INSTALLED VERSIONS

cftime: 1.6.4
dask: 2025.7.0
fastprogress: 1.0.3
fsspec: 2025.9.0
gcsfs: None
intake: 2.0.8
intake_esm: 2025.7.9
netCDF4: 1.7.2
pandas: 2.3.2
requests: 2.32.5
s3fs: 2025.9.0
xarray: 2025.6.1
zarr: 2.18.3

[charles-turner-1]

What I Did

import intake
cat = intake.open_esm_datastore("https://cadcat.s3.amazonaws.com/cae-zarr.csv")
                                                                          ^^^

It looks like you're trying to open the csv file associated with the datastore here. There should be an .json file (based on a quick google it looks like https://cadcat.s3.amazonaws.com/cae-collection.json?) file associated with this, which is what you want to open.

So assuming I've found the correct file,

import intake
intake.open_esm_datastore('https://cadcat.s3.amazonaws.com/cae-collection.json')

[BaptisteFrancois]

Hi
Thanks for your answer.
I messed up my ticket actually.

The code I am trying to run is indeed using the .json

import intake
cat = intake.open_esm_datastore('https://cadcat.s3.amazonaws.com/cae-collection.json')

Yet, I am getting the error mentioned above, which I re copy/pasting below.

thread '<unnamed>' (29816) panicked at crates\polars-io\src\path_utils\mod.rs:427:63:
begin <= end (10 <= 9) when slicing `cadcat.s3.amazonaws.com/cae-zarr.csv`
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\core.py", line 124, in __init__
    self.esmcat = ESMCatalogModel.load(
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 265, in load
    cat._cast_agg_columns_with_iterables()
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 369, in _cast_agg_columns_with_iterables
    self.columns_with_iterables.intersection(
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 354, in columns_with_iterables
    return self._frames.columns_with_iterables  # type: ignore[union-attr]
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\intake_esm\cat.py", line 582, in columns_with_iterables
    if (trunc_df := self.lazy.head(1).collect()).is_empty():
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\_utils\deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\lazyframe\opt_flags.py", line 330, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\bfrancois\AppData\Local\anaconda3\envs\ebmud\lib\site-packages\polars\lazyframe\frame.py", line 2407, in collect
    return wrap_df(ldf.collect(engine, callback))
pyo3_runtime.PanicException: begin <= end (10 <= 9) when slicing `cadcat.s3.amazonaws.com/cae-zarr.csv`

[charles-turner-1]

Curious - I'm able to run the first snippet (import intake; cat = intake.open_esm_datastore('https://cadcat.s3.amazonaws.com/cae-collection.json')) without any issues.

It looks from your error messages that you're on Windows - can you additionally let me know which version of polars you're using? (import polars; polars.__version__) and I'll see if I can figure out what's going on.

Cheers!

[BaptisteFrancois]

Yes I am on Windows, and using polars version 1.33.1

[charles-turner-1]

Cheers, I was able to reproduce that bug. Downgrading to polars version 1.32 (conda install polars <1.33) should fix the bug. I'll open a PR fixing this for polars >=1.33 (or pinning to <=1.32).

[BaptisteFrancois]

Thanks. That worked!

[charles-turner-1]

Reopening as pinning to polars <=1.33 is not really a fix.

[charles-turner-1]

Okay, discovered the source of the issue:

import polars as pl
import fsspec
with fsspec.open("https://cadcat.s3.amazonaws.com/cae-zarr.csv") as f:
    lf = pl.scan_csv(f)
print(lf.head(3).collect())
shape: (3, 9)
┌───────────┬───────────┬───────────┬───────────┬───┬──────────┬───────────┬───────────┬───────────┐
│ activity_ ┆ instituti ┆ source_id ┆ experimen ┆ … ┆ table_id ┆ variable_ ┆ grid_labe ┆ path      │
│ id        ┆ on_id     ┆ ---       ┆ t_id      ┆   ┆ ---      ┆ id        ┆ l         ┆ ---       │
│ ---       ┆ ---       ┆ str       ┆ ---       ┆   ┆ str      ┆ ---       ┆ ---       ┆ str       │
│ str       ┆ str       ┆           ┆ str       ┆   ┆          ┆ str       ┆ str       ┆           │
╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪══════════╪═══════════╪═══════════╪═══════════╡
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ hursmax   ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ hursmin   ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ huss      ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
└───────────┴───────────┴───────────┴───────────┴───┴──────────┴───────────┴───────────┴───────────┘

or

df = pl.read_csv("https://cadcat.s3.amazonaws.com/cae-zarr.csv").head(3)
print(df)

Both work fine, but

lf = pl.scan_csv("https://cadcat.s3.amazonaws.com/cae-zarr.csv").head(3)

Triggers the error. It's a simple fix, just needs an fsspec wrapper. Working on the PR now!

with fsspec.open("https://cadcat.s3.amazonaws.com/cae-zarr.csv") as f:
    lf = pl.scan_csv(f)
print(lf.head(3).collect())
shape: (3, 9)
┌───────────┬───────────┬───────────┬───────────┬───┬──────────┬───────────┬───────────┬───────────┐
│ activity_ ┆ instituti ┆ source_id ┆ experimen ┆ … ┆ table_id ┆ variable_ ┆ grid_labe ┆ path      │
│ id        ┆ on_id     ┆ ---       ┆ t_id      ┆   ┆ ---      ┆ id        ┆ l         ┆ ---       │
│ ---       ┆ ---       ┆ str       ┆ ---       ┆   ┆ str      ┆ ---       ┆ ---       ┆ str       │
│ str       ┆ str       ┆           ┆ str       ┆   ┆          ┆ str       ┆ str       ┆           │
╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪══════════╪═══════════╪═══════════╪═══════════╡
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ hursmax   ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ hursmin   ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
│ LOCA2     ┆ UCSD      ┆ ACCESS-CM ┆ historica ┆ … ┆ day      ┆ huss      ┆ d03       ┆ s3://cadc │
│           ┆           ┆ 2         ┆ l         ┆   ┆          ┆           ┆           ┆ at/loca2/ │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ucsd/acce │
│           ┆           ┆           ┆           ┆   ┆          ┆           ┆           ┆ ss-…      │
└───────────┴───────────┴───────────┴───────────┴───┴──────────┴───────────┴───────────┴───────────┘