accessing nodes from the reader causes a lot of recursion but only on github CI

[kushalkolar]

Hi! We're using one of the multi-channel zarr examples in the CI for fastplotlib as both verification for multi-channel images and as an example in the docs. However today this stopped working but only on github CI, there's a recursion depth error if you do list(reader()). I also tried next(reader()) but get the same thing:

examples/image_volume/image_volume_multi_channel.py:22: in <module>
    image_node = next(reader())
                 ^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:574: in __call__
    node = Node(self.zarr, self)
           ^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:53: in __init__
    self.specs.append(Multiscales(self))
                      ^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:316: in __init__
    node.add(child_zarr, visibility=False)
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:139: in add
    node = Node(zarr, self, visibility=visibility, plate_labels=plate_labels)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:49: in __init__
    self.specs.append(Labels(self))
                      ^^^^^^^^^^^^
...

/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:51: in __init__
    self.specs.append(Label(self))
                      ^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:221: in __init__
    node.add(parent_zarr, prepend=True, visibility=False)
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:139: in add
    node = Node(zarr, self, visibility=visibility, plate_labels=plate_labels)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:53: in __init__
    self.specs.append(Multiscales(self))
                      ^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:297: in __init__
    data: da.core.Array = self.array(resolution)
                          ^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/reader.py:320: in array
    return self.zarr.load(resolution)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/ome_zarr/io.py:145: in load
    return da.from_zarr(self.__store, subpath)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/array/core.py:3795: in from_zarr
    name = "from-zarr-" + tokenize(z, component, storage_options, chunks, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:76: in tokenize
    return _tokenize(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:34: in _tokenize
    token: object = _normalize_seq_func(args)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:154: in _normalize_seq_func
    return tuple(map(_inner_normalize_token, seq))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:147: in _inner_normalize_token
    return normalize_token(item)
           ^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:207: in normalize_object
    return _normalize_dataclass(o)
           ^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:268: in _normalize_dataclass
    (field.name, normalize_token(getattr(obj, field.name, None)))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:207: in normalize_object
    return _normalize_dataclass(o)
           ^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:268: in _normalize_dataclass
    (field.name, normalize_token(getattr(obj, field.name, None)))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:207: in normalize_object
    return _normalize_dataclass(o)
           ^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:268: in _normalize_dataclass
    (field.name, normalize_token(getattr(obj, field.name, None)))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:161: in normalize_seq
    return type(seq).__name__, _normalize_seq_func(seq)
                               ^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:154: in _normalize_seq_func
    return tuple(map(_inner_normalize_token, seq))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:147: in _inner_normalize_token
    return normalize_token(item)
           ^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:207: in normalize_object
    return _normalize_dataclass(o)
           ^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:268: in _normalize_dataclass
    (field.name, normalize_token(getattr(obj, field.name, None)))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/utils.py:781: in __call__
    return meth(arg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/dask/tokenize.py:161: in normalize_seq
    return type(seq).__name__, _normalize_seq_func(seq)
                               ^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

seq = (1, 1, 256, 256)

    def _normalize_seq_func(seq: Iterable[object]) -> tuple[object, ...]:
        def _inner_normalize_token(item):
            # Don't go through Dispatch. That's slow
            if isinstance(item, _IDENTITY_DISPATCH):
                return item
            return normalize_token(item)
    
        with tokenize_lock:
            if id(seq) in _SEEN:
                return "__seen", _SEEN[id(seq)][0]
            _SEEN[id(seq)] = len(_SEEN), seq
            try:
>               return tuple(map(_inner_normalize_token, seq))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E               RecursionError: maximum recursion depth exceeded

Not sure how to troubleshoot this. Here's the error log from the CI action: https://github.com/fastplotlib/fastplotlib/actions/runs/18955626439/job/54131575160#step:8:183

Could it be that the EBI server hosting the data is sending weird data when it sees that the github actions IP is requesting it?

[will-moore]

I have no idea what's going on there, but I see that we have the same issue now at https://github.com/ome/ome-zarr-py/actions/runs/18946934905/job/54100792429?pr=492