Skip to content

Using S3 list_objects_v2 call to check the existence of a bucket is too costly #960

@yima77

Description

@yima77

Here, I refer to the following code lines in s3fs/core.py:

1107             try:
1108                 await self._call_s3(
1109                     "list_objects_v2", MaxKeys=1, Bucket=bucket, **self.req_kw
1110                 )
1111                 return True
1112             except Exception:
1113                 pass

The purpose of these code lines is to check if the bucket exists. This can be accomplished by calling "head_bucket", instead. Listing objects, even with max-keys=1, can be costly on a bucket whose index is managed in multiple shards, like ceph does.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions