List Object Pagination by start a Postion

[hantsy]

I tried to use the following codes to get objects in a bucket.

client
    .listObjectsV2Paginated(
        ListObjectsV2Request
        {
            bucket = myBucketName
            maxKeys = limit
        }
    )
    .transform { response ->
        response.contents
            ?.mapNotNull { it.key }
            ?.forEach { emit(it) }
    }

I want to list the keys by an offset/limit parameter pair from an HTTP request, but ListObjectsV2Request does not accept an offset position like field to specify the start position. There is a startAfter field that requires a key instead.

[ianbotsf]

Hi @hantsy. ListObjectsV2Request does not contain a field for an offset position because the underlying service API does not provide a parameter for specifying a numeric offset.

Offset-based browsing is generally not supported in non-relational datastores (such as S3) because elements (such as S3 objects/keys) do not have intrinsic sequential numeric identifiers. Thus, attempting to list elements starting at say 1000 would require some service-side process to manually iterate through the elements, incrementing a counter each time until it got to 1000. This is a fundamentally unscalable pattern and, as such, is generally not offered.

Because S3 does not offer this feature, the SDK cannot provide the means to paginate by numeric offsets directly. You're welcome to open a support case with S3 to request this feature but they may have a similar take as mine above.

At a higher level, what are you trying to accomplish by using offset pagination?