Clarify the type and number of Cloud Storage Operations performed by gcsfuse

[steve-marmalade]

Hi there,

I am trying debug a larger than expected number of Cloud Storage Class A operations, and I'm wondering if my usage of gcsfuse may explain it.

Concretely, if I mount a bucket via gcsfuse --implicit-dirs my-bucket /my-bucket and then attempt to read a file in a somewhat deeply nested directory /my-bucket/directory1/directory2/directory3/file, how many Class A Operations would you expect this to make?

I expected the answer to be 0 (just a single storage.objects.get, which is a Class B operation), but from re-reading the --implicit-dirs semantics, it looks like there will be at least one storage.objects.list.

Thanks for your help and for this useful utility.

[Tulsishah]

It is four in this case.
The number of calls is equal to the numbers of directories in the path + one for file name as mentioned here --implicit-dirs semantics

[steve-marmalade]

@Tulsishah , thank you very much for the response.

When I initially read the docs, I did not appreciate:

• That list requests would be made even when I am attempting to load a specific file (as opposed to listing the contents of a directory)
  • By extension, that there would be a list request for each directory in addition to the file.
• That storage.objects.list requests are billed differently from storage.objects.get requests (the former are more expensive)

To summarize, I did not expect there to be significant billing implications to consider when using gcsfuse, but there clearly are. This might be worth highlighting more prominently in the documentation (I certainly would have appreciated it).

[ahmedetefy]

I completely agree with @steve-marmalade 's sentiment on this. We also had significant billing implications with using gcsfuse because the list operations are billed as class A operations which are significantly more costly than class B operations.

This might be worth highlighting more prominently in the documentation (I certainly would have appreciated it).

+1

[vishal2-wiai]

This might be worth highlighting more prominently in the documentation (I certainly would have appreciated it).

+1. We have had cases of extremely high bill because someone ran an ML model training job on a gcsfuse mounted bucket. The documentation infact highlights ML use cases stating "cost effectiveness" of training a model on the data directly from the bucket.

Has anybody tried hierarchical namespace buckets? Does that reduce Class A ops?