exists calls much slower between 2025.3.0 and 2025.9.0

[APatrickJ]

After #696, we're still seeing slow performance for exists calls. Version 2025.9.0 performs significantly slower than 2025.3.0.

I have a small benchmarking script to show the difference (see attached).

Here's the benchmark I'm running:

python benchmark.py gs://earth-system-public-tiles-a86c/lima/tiles/10

With 2025.3.0:

Benchmarking: gs://earth-system-public-tiles-a86c/lima/tiles/10
--------------------------------------------------

==================================================
RESULTS
==================================================
Path: gs://earth-system-public-tiles-a86c/lima/tiles/10
Exists: True
Time: 2.877585 seconds
Time: 2877.58 milliseconds

With 2025.9.0:

Benchmarking: gs://earth-system-public-tiles-a86c/lima/tiles/10
--------------------------------------------------

==================================================
RESULTS
==================================================
Path: gs://earth-system-public-tiles-a86c/lima/tiles/10
Exists: True
Time: 37.475832 seconds
Time: 37475.83 milliseconds

And updating the command with the --debug flag is still showing this log output repeatedly with 2025.9.0 that isn't appearing in the log output for 2025.3.0:

2025-10-16 11:41:43,261 - gcsfs - DEBUG - GET: b/{}/o, ('earth-system-public-tiles-a86c',), None
2025-10-16 11:41:43,300 - gcsfs - DEBUG - _call -- GET: b/{}/o, ('earth-system-public-tiles-a86c',), None

benchmark.py

[Mahalaxmibejugam]

@APatrickJ I tried to run the provided script for 2025.3.0 and 2025.9.0 versions of gcsfs and I observed that execution time and DEBUG logs are similar between these two versions for the dataset that I used (40k objects with nested structure of 3 levels). I ran it for multiple paths and observed similar results for both versions, attaching one instance of the results below.

2025.3.0 results

gcsfs version: 2025.3.0
fsspec version: 2025.3.0
Benchmarking: gs://gcsfs-hns-testing/benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_89/l2_104
--------------------------------------------------
2025-10-22 17:43:49,118 - asyncio - DEBUG - Using selector: EpollSelector
2025-10-22 17:43:49,119 - google.auth._default - DEBUG - Checking None for explicit credentials as part of auth process...
2025-10-22 17:43:49,119 - google.auth._default - DEBUG - Checking Cloud SDK credentials as part of auth process...
2025-10-22 17:43:49,725 - gcsfs.credentials - DEBUG - connect -- Connected with method google_default
2025-10-22 17:43:49,725 - gcsfs.credentials - DEBUG - connect -- Connected with method google_default
2025-10-22 17:43:49,725 - gcsfs.credentials - DEBUG - Connected with method google_default
2025-10-22 17:43:49,726 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('gcsfs-hns-testing', 'benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_89/l2_104'), None
2025-10-22 17:43:49,726 - gcsfs - DEBUG - GET: b/{}/o/{}, ('gcsfs-hns-testing', 'benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_89/l2_104'), None
2025-10-22 17:43:49,726 - gcsfs.credentials - DEBUG - maybe_refresh -- GCS refresh
2025-10-22 17:43:49,726 - gcsfs.credentials - DEBUG - maybe_refresh -- GCS refresh
2025-10-22 17:43:49,726 - gcsfs.credentials - DEBUG - GCS refresh
2025-10-22 17:43:49,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): oauth2.googleapis.com:443
2025-10-22 17:43:49,792 - urllib3.connectionpool - DEBUG - https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
2025-10-22 17:43:50,000 - gcsfs - DEBUG - _call -- GET: b/{}/o, ('gcsfs-hns-testing',), None
2025-10-22 17:43:50,000 - gcsfs - DEBUG - GET: b/{}/o, ('gcsfs-hns-testing',), None


==================================================
RESULTS
==================================================
Path: gs://gcsfs-hns-testing/benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_89/l2_104
Exists: True
Time: 1.080780 seconds
Time: 1080.78 milliseconds

2025.9.0 results

gcsfs version: 2025.9.0
fsspec version: 2025.9.0
Benchmarking: gs://gcsfs-hns-testing/benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_49/l2_194
--------------------------------------------------
2025-10-22 17:45:18,818 - asyncio - DEBUG - Using selector: EpollSelector
2025-10-22 17:45:18,819 - google.auth._default - DEBUG - Checking None for explicit credentials as part of auth process...
2025-10-22 17:45:18,819 - google.auth._default - DEBUG - Checking Cloud SDK credentials as part of auth process...
2025-10-22 17:45:19,425 - gcsfs.credentials - DEBUG - connect -- Connected with method google_default
2025-10-22 17:45:19,425 - gcsfs.credentials - DEBUG - connect -- Connected with method google_default
2025-10-22 17:45:19,425 - gcsfs.credentials - DEBUG - Connected with method google_default
2025-10-22 17:45:19,427 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('gcsfs-hns-testing', 'benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_49/l2_194'), None
2025-10-22 17:45:19,427 - gcsfs - DEBUG - GET: b/{}/o/{}, ('gcsfs-hns-testing', 'benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_49/l2_194'), None
2025-10-22 17:45:19,427 - gcsfs.credentials - DEBUG - maybe_refresh -- GCS refresh
2025-10-22 17:45:19,427 - gcsfs.credentials - DEBUG - maybe_refresh -- GCS refresh
2025-10-22 17:45:19,427 - gcsfs.credentials - DEBUG - GCS refresh
2025-10-22 17:45:19,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): oauth2.googleapis.com:443
2025-10-22 17:45:19,482 - urllib3.connectionpool - DEBUG - https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
2025-10-22 17:45:19,695 - gcsfs - DEBUG - _call -- GET: b/{}/o, ('gcsfs-hns-testing',), None
2025-10-22 17:45:19,695 - gcsfs - DEBUG - GET: b/{}/o, ('gcsfs-hns-testing',), None

==================================================
RESULTS
==================================================
Path: gs://gcsfs-hns-testing/benchmark-exists-29103578-c3f1-46b9-aebd-cb905f72dcb3/l1_49/l2_194
Exists: True
Time: 1.067743 seconds
Time: 1067.74 milliseconds

Can you let us know about the dataset you are using and also the bucket type to help us debug more?

[Mahalaxmibejugam]

Identified an edge case where the performance of the exists() method has degraded between 2025.3.0 and 2025.9.0 versions. If the exists() method is called on a directory which only has sub-directories and no direct objects then the number of calls to GCS API are too high(equal to the number of the sub-directories under the provided path) which leads to performance degradation.

The exists() method uses info(), which was refactored in version 2025.9.0 to use list_objects() with max_results set to 1. This was intended to stop unnecessarily listing all objects. But with this change an unintended bug was introduced where we are making one call for each prefix under the provided path.

The root cause is in the while condition of the code where max_results is compared only to the number of items. If there are no immediate objects, the items count will always be 0, with the response only containing prefixes. Consequently, all subfolders under the path are fetched until next_page_token is null and in each call only 1 subfolder is fetched as the max_results parameter per call is set to 1.

page = await self._call(
            "GET",
            "b/{}/o",
            bucket,
            maxResults=num_items,
            ...)

        prefixes.extend(page.get("prefixes", []))
        items.extend(page.get("items", []))
        next_page_token = page.get("nextPageToken", None)

while len(items) < max_results and next_page_token is not None:
     num_items = min(items_per_call, max_results - len(items), 1000)
     page = await self._call(
          "GET",
          "b/{}/o",
          bucket,
          maxResults=num_items,
          pageToken=next_page_token,
    ...)

We will make changes to compare the max_results with both length of items and prefixes.

@APatrickJ Can you please confirm if this was the case for you as well? Are there any objects directly under the prefix(gs://earth-system-public-tiles-a86c/lima/tiles/10) you are trying to query?

[APatrickJ]

@Mahalaxmibejugam That is exactly the case! In the instances where we observe this degraded performance, we have directories that only contain subdirectories and no direct objects.

I will also note that we have hierarchical namespace enabled on these buckets, though I'm not sure if that's relevant.

In any case, I tried building a new gcsfs with #705 and re-running the benchmark script posted earlier. It's back to the same performance that we saw with 2025.3.0. Great debugging and nice work!

pip install "universal_pathlib==0.3.4" # To match the earlier environment
pip install git+https://github.com/fsspec/gcsfs.git@refs/pull/705/head
python benchmark.py gs://earth-system-public-tiles-a86c/lima/tiles/10

Output:

Benchmarking: gs://earth-system-public-tiles-a86c/lima/tiles/10
--------------------------------------------------

==================================================
RESULTS
==================================================
Path: gs://earth-system-public-tiles-a86c/lima/tiles/10
Exists: True
Time: 2.552677 seconds
Time: 2552.68 milliseconds

[Mahalaxmibejugam]

Thanks @APatrickJ for confirming, #705 is merged to main now.

[martindurant]

I anticipate a release in the next couple of days.