High ListObjectsV2 cost when only accessing existing objects/files

[nguyenminhdzung]

Tell us more about this new feature.

I have 2 systems:

• one allows users to upload files and it saves files to a S3 bucket, with folder structure of yyyy/MM/dd (only added, no deleted or modified)
• one uses mountpoint-s3 to mount the bucket to a folder and uses nginx to serve uploaded files from the mounted folder. Path of url is like 2025/03/10/file1.dat

The problem is I use mountpoint-s3 to serve existing files (100% guarantee existing) but ListObjectsV2 charging me a huge bill because the traffics are very high.
Why don't you just simply use only Head and Get reqs for files reading? Currently List reqs cost me ten times of Head and Get reqs for a very simple thing: reading files.
I am very disappointed. The finished line is just 1 meters from me but now millions kilometers far far away.

See, LIST req costs about 10 times of HEAD or GET req. Why didn't you consider this money-impact when sending 2 reqs at the same time for just one purpose.

[dannycjones]

Hey @nguyenminhdzung, thank you for your feedback. I'll share some recommendations which I hope helps for your use case, and also provide some explanation on why Mountpoint leverages both ListObjectsV2 and HeadObject when accessing objects in S3.

Since you describe your use case as always accessing files which are never modified, I recommend using these configurations: --metadata-ttl indefinite --negative-metadata-ttl minimal. This leverages Mountpoint's metadata cache and tells it to retain information about an object indefinitely (the cache entry's "time-to-live"/TTL) and will minimize the ListObjectsV2 and HeadObject requests. The --negative-metadata-ttl argument also tells Mountpoint not to remember when a file or directory doesn't exist by minimising negative cache TTLs.

You may also benefit from the data cache, which can persist object data on a local file system to avoid going to S3. You can learn more about Mountpoint's caching configurations here: https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#caching-configuration

Regarding why the ListObjectsV2 occurs, this is due to the way the file system interface interacts with S3. Mountpoint is using the FUSE filesystem driver and it works by querying each path component ("a" and "b.txt" in "a/b.txt") to check if it exists: be it a directory or a file. We use ListObjectsV2 to check if the directory/prefix exists, and HeadObject to check if the file/object exists. Today we perform both concurrently to minimize latency when accessing files.

Please do let us know if the recommendations help for your use case, and reach out if you have any more questions!

[nguyenminhdzung]

Hi @dannycjones , thank you very much. I read all the issues related to this, I read the source code that send the 2 reqs concurrently, I understand your recommendation.
I already set metadata cache TTL to 6 months before I submitted this issue but you know, whenever a file is read, at least 2 reqs are sent to S3 (for the first time), List costs more than 10 times of Head.
My system uploads a huge new images everyday and the traffics to read them is extremely high. Even a long metadata cache TTL doesn't help the fact that the List req cost me a lot.
My system doesn't need to access folder, only files, surely know accessing url is a file.
I can give you an imaginable example, the Head reqs cost me $1 monthly and the List reqs cost me more than $10 monthly.
What if you provide an arg, such as read-mode, enum of only-read-file, all. With only-read-file mode, you assume the last part of a path is file, others are its parent folders and just send only Head req. It will reduce my bill a lot.

Thank you.

[dannycjones]

Thanks for the extra information. Unfortunately, I'm not super clear on the access pattern and why this results in a lot of traffic despite caching being enabled. Is it possible to share some Mountpoint debug logs to help explain the issue?

---

What if you provide an arg, such as read-mode, enum of only-read-file, all. With only-read-file mode, you assume the last part of a path is file, others are its parent folders and just send only Head req. It will reduce my bill a lot.

Thank you for sharing the idea. It is something we've considered however unfortunately the interaction between the kernel and our file system doesn't work this way - we only get requests for individual path components. As an example, for 2025/03/10/file1.dat we would get 2025 then 2025/03, 2025/03/10, and finally 2025/03/10/file1.dat. The problem is that Mountpoint can't know if there will be another path after that.

There is a feature request which may be of interest to you: #946. This request explores the idea of changing Mountpoint semantics to prioritize files over directories, and making a trade off on performance to allow reducing cost by first checking the response of HeadObject before performing a ListObjectsV2 which it would be possible to avoid if a "file"/object is found. If this feature request is of interest, I'd recommend reacting with a thumbs up (👍).

[nguyenminhdzung]

Hi @dannycjones , thank you very much.

There are a huge amount of new files uploaded directly to my S3 bucket, and all of those new file are accessed via mountpoint-S3 mounted after that. When a file, e.g. 2025/04/01/file_1.dat is accessed, a Head request to 2025/04/01/file_1.dat and a List req to 2025/04/01/file1.dat/. With that fact, it costs me a lot. A long metadata cache TTL only helps for already accessed files.

That --cheap option mentioned in #946 is somehow exactly what I need.