`dvc[gs]` that uses `gcsfs` as a backend is significantly slower since `2025.5.0.post1`

[Lhenzo]

Running operation like dvc pull runs in 38.959s with gcsfs==2025.5.0.post1 compared to 2.634s with gcsfs==2025.3.2. Running dvc pull --verbose does not give me more information, I'm sorry for the poor debug message. That's all I can give you for now. But I'm available if you need some informations/beta testing. Looking at the compare between these versions I can not imagine anything slowing down the process. Maybe it's on dvc's side?

[martindurant]

I have an idea. Can you please enable the loggers "gcsfs.credentials" and "gcsfs", e.g., with

fsspec.utils.setup_logging(logger_name="gcsfs")
fsspec.utils.setup_logging(logger_name="gcsfs.credentials")

What type of token are you using, does it have an expiry time?

[martindurant]

If my suspicion is right, the following may fix it

--- a/gcsfs/credentials.py
+++ b/gcsfs/credentials.py
@@ -178,8 +178,8 @@ class GoogleCredentials:
                 (
                     self.credentials.expiry
                     and (
-                        self.credentials.expiry.astimezone(timezone.utc)
-                        - datetime.now(timezone.utc)
+                        self.credentials.expiry
+                        - datetime.now(timezone.utc).replace(tzinfo=None)
                     ).total_seconds()
                     > refresh_buffer

[Lhenzo]

Thanks; looking into credentials.py, it seems that I enter _credentials_valid 258 times and that checking all of that takes some time (within a loop it seems). So, my guess is not related to timezone or credentials expiry but something related to the quantity of info we fetch/check before validating the credentials and allowing to continue the pull operation. I don't know if it could be related to max_results = kwargs.get("max_results")?

[Lhenzo]

Seems to be it, I go back to 2s if I put default_page_size which is default_page_size = 5000 compared to max_results that gives 1

[martindurant]

Sorry, what exactly do you need to pass to get the previous performance? I don't think these changed in the last release.

[Lhenzo]

In 2025.3.2...2025.5.0post1, I see that in gcsfs/core.py maxResults component now receives max_results instead of page_size that is default_page_size = 5000 so I went back to this variable and got back the performances

[martindurant]

Can you please check with the following?

--- a/gcsfs/core.py
+++ b/gcsfs/core.py
@@ -676,7 +676,7 @@ class GCSFileSystem(asyn.AsyncFileSystem):
                 delimiter=delimiter,
                 prefix=prefix,
                 versions=versions,
-                page_size=default_page_size,
+                page_size=max_results or default_page_size,
             )

[Lhenzo]

Nop, not doing the trick.
max_results returns 1 & default_page_size is 5000. It seems I go into the sequential because inventory_report_info is None. Putting the default_page_size instead of max_results for max_results in line 692 in the _sequential_list_objects_helper function then only then I get back the speed.
It's about this change

and this

[Lhenzo]

max_results to 25 is enough to get back the speed if you want to stick with this variable

[martindurant]

Sorry, I am well confused! Are you saying that changing max_results from 1 to 25 on line 1036 (in _info) makes things faster?

So then, dvc is calling info() a lot but not ls. Perviously, info would call ls(), which is more expensive for a single call, but does get cached for future calls. So the problem here is more subtle! If the caller wants the info for a single file, it makes sense to use the fastest method...

[Lhenzo]

Yes, exactly. Going from max_results=1 to max_results=25 give me the performance back.
Should I open a ticket on dvc? I'm not mastering enough the usage of this part of the code within dvc. I could have a look.

[martindurant]

It's a good idea. I still recommend that you use the "gcsfs" logger so that you can see exactly what calls dvc is making. From their point of view, doing an ls() (or maybe find() ) preemptively should cache all the file info and make things faster, I think.

[martindurant]

(also, please link this issue if you do make one with dvc)

[robince]

I'm also seeing this issue using dvc with a GCS remote with a high latency connection. According to the other ticket this should have been fixed in 2025.9.0 but I still have the problem.

time uv run --with "gcsfs==2025.3.2" -- dvc status -c
Executed in    2.63 secs      fish           external

 time uv run --with "gcsfs==2025.5.1" -- dvc status -c
Executed in   31.83 secs    fish           external

time uv run --with "gcsfs==2025.7.0" -- dvc status -c
Executed in   27.04 secs    fish           external

time uv run --with "gcsfs==2025.9.0" -- dvc status -c
Executed in   26.87 secs    fish           external