Mountpoint crashes when using cache

[the-lay]

Mountpoint for Amazon S3 version

mount-s3 1.12.0

AWS Region

eu-central-1

Describe the running environment

Running inside Ubuntu docker image (with --privileged flag). The host is an EC2 (c6i.32xlarge) on Amazon Linux 2. AWS credentials are provided with the instance profile credentials. Connecting to an S3 bucket in the same account and same region. /dev/nvme0n1p is an EBS gp3 drive.

Mountpoint options

mount-s3 --debug --log-directory logs BUCKET /mountpoint --prefix PREFIX --cache /tmp/s3_cache/

What happened?

Mount point fails while supporting a read-heavy, highly parallel scientific application with MPIs and threading. It is quite random, but it crashes quickly, within the first minute, the applications fails after not being able to read the files anymore. Accessing any file manually fails with "Transport endpoint is not connected". When I try to unmount the mountpoint, it fails with "target is busy".

My first guess was OOM, but the memory usage does not go above 30% (c6i.32xlarge with 256GB RAM). I found that the crash seems to happen only when --cache is specified. The disk is an EBS drive with more than enough free space. Please suggest how I can debug it further.

Here is the sanitized log file with --debug: initial mounting and then the last 5 seconds.

mountpoint-s3-2025-03-31T15-28-33Z.XqRodv.log

[passaro]

Hi @the-lay, thank you for opening the issue and providing your logs.

From an initial look:

• Mountpoint is reporting a few errors when reading from the disk cache. It is unlikely that they are causing your issue, since they should only result in the read to be served from a new S3 request rather than from the cache. But this is something we want to investigate further.

• I see that process.memory_usage: 1026052096. Since you are running Mountpoint in a container, is there any chance that it has memory limit set on it?

[the-lay]

Thank you!

Since you are running Mountpoint in a container, is there any chance that it has memory limit set on it?

Sorry, I forgot to mention that the containers are running on AWS Batch, and yes, they have a memory limit set. However, it is set to almost the maximum instance offers. In this case, it was set to 245GB (out of 256 total).

[dannycjones]

Hey @the-lay, sorry for the delay.

I think one of the best ways to try and understand what's happening would be to review the syslog and dmesg log. No logs in Mountpoint logs does suggest something external killed Mountpoint, and I hope that the system logs will reveal why - this is where we'd usually see OOMKiller.

On AL2, you can review the syslog using journalctl. You can review dmesg log using cat /var/log/dmesg.

[adpeace]

--cache /tmp/s3_cache/

Is it possible this directory is not on the EBS volume? Did you mount a separate EBS volume for the cache at that location, or is there any possibility that this a tmpfs and thus writing to the cache is consuming memory? I notice you didn't set a cache size limit on the command line so wondering if that could be where unexpected memory usage is coming from.

[the-lay]

@dannycjones thanks, I will review syslog and dmesg within the next couple of days and will add a comment with my findings.

@adpeace thanks, but we've just confirmed that /tmp is on EBS, both on the host and inside the docker container: