Allow custom parameters in presigned URL generation

[drnextgis]

Describe the feature

This article outlines a method for monitoring the usage of presigned URLs. The provided code is specifically designed for this purpose, as there is currently no capability in boto3 to include custom parameters in presigned URLs. Therefore, it would be beneficial to introduce a feature enabling the inclusion of custom parameters in presigned URLs.

Use Case

Many software-as-a-service (SaaS) product offerings have a pay-as-you-go pricing model, charging customers only for the resources consumed. However, a pay-as-you-go pricing is only viable when you can accurately track each customer’s use of resources, such as compute capacity, storage, and networking bandwidth. Without this data, SaaS providers do not have visibility into resource consumption of each customer, and are unable to bill based on the their usage. This is just one example where the ability to accurately track consumption of similar resources can be a critical need.

Proposed Solution

To address this limitation, I made a modification to the content of the botocore/data/s3/2006-03-01/service-2.json file, incorporating the necessary custom parameter into the GetObjectRequest section.

"UserName":{
    "shape":"UserName",
    "documentation":"<p>username<p>",
    "location":"querystring",
    "locationName":"username"
}

Is this approach considered acceptable, or is there a more suitable method to achieve the same outcome?

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

1.26.10

Environment details (OS name and version, etc.)

Linux

[tim-finnigan]

Hi @drnextgis thanks for reaching out. I would caution against directly modifying service model JSON files, as doing so could cause unexpected behavior and possibly break workflows. Those files are also frequently updated, so it may be difficult to support custom behavior while staying up to date with recent versions.

That being said, I see the reasoning behind your use case and the approach you describe does appear to support it. But rather than modifying the service file directly, I recommend adding a custom file to ~/.aws/models if you need to override the existing file via the search path.

Have you also explored alternative approaches, for example using CloudTrail to identify S3 requests? One reason I ask about alternatives is because S3 server logs are delivered on a best effort basis, and the completeness/timeliness of server logging is not guaranteed.

[drnextgis]

Great! Thank you @tim-finnigan for sharing the correct approach for handling custom models. I wasn't aware of the 'best effort' basis for S3 logs, so I'll definitely take CloudTrail into consideration. Would it be acceptable if I keep this ticket open until the method for adding custom parameters to presigned URLs with boto3 is documented?

[tim-finnigan]

Thanks for following up — I think we can leave this open for more discussion and input from others. An approach that involves updating service models may not get documented because of the risks mentioned earlier. But if there were any documentation update it would probably happen on this page: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html.

Just to add some more context on what I observed:

import boto3

s3_client = boto3.client('s3')

# from botocore.config import Config
# s3_client = boto3.client('s3', config=Config(signature_version='s3v4'))

response = s3_client.generate_presigned_url('get_object',
                                            Params={
                                                    'Bucket': 'bucket-name',
                                                    'Key': 'test',
                                                    },
                                            ExpiresIn=3600)

with_custom_param = f"{response}&UserName=username"
print(with_custom_param)

Using the above does generate a working presigned URL using SigV2. But if you uncomment those two lines above it will use SigV4, which will results in a SignatureDoesNotMatch error due to the UserName parameter not being included. If you do include UserName in Params then you receive the error:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "UserName", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, ResponseCacheControl, ResponseContentDisposition, ResponseContentEncoding, ResponseContentLanguage, ResponseContentType, ResponseExpires, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber, ExpectedBucketOwner, ChecksumMode

That is why updating the models to support a custom parameter is required here. And any requests to add new supported parameters would need to get forwarded to the S3 team.

[drnextgis]

A more straightforward approach to do the same without touching the models: https://stackoverflow.com/a/59057975