Support for object level Tagging in boto3 upload_file method

[bhandaresagar]

I want to add tags to the files as I upload them to S3. Boto3 supports specifying tags with put_object method, however considering expected file size, I am using upload_file function which handles multipart uploads. But this function rejects 'Tagging' as keyword argument.

import boto3
client = boto3.client('s3', region_name='us-west-2')
client.upload_file('test.mp4', 'bucket_name', 'test.mp4',
                   ExtraArgs={'Tagging': 'type=test'})

ValueError: Invalid extra_args key 'Tagging', must be one of: ACL, CacheControl, ContentDisposition, ContentEncoding, ContentLanguage, ContentType, Expires, GrantFullControl, GrantRead, GrantReadACP, GrantWriteACP, Metadata, RequestPayer, ServerSideEncryption, StorageClass, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, SSEKMSKeyId, WebsiteRedirectLocation

I found a way to make this work by using S3 transfer manager directly and modifying allowed keyword list.

from s3transfer import S3Transfer
import boto3

client = boto3.client('s3', region_name='us-west-2')
transfer = S3Transfer(client)
transfer.ALLOWED_UPLOAD_ARGS.append('Tagging')
transfer.upload_file('test.mp4', 'bucket_name', 'test.mp4',
                     extra_args={'Tagging': 'type=test'})

Even though this works, I don't think this is the best way. It might create other side effects. Currently I am not able to find correct way to achieve this. Any advice would be great. Thanks.

[swetashre]

@bhandaresagar - Thank you for your post. As per our documentation Tagging is not supported as a valid argument for upload_file method that's why you are getting ValueError.

You can use put_object_tagging method to set tags after uploading a object to the bucket.

[bhandaresagar]

@swetashre I understand that the Tagging is not supported as as valid argument, that is the reason I am updating the ALLOWED_UPLOAD_ARGS in second example. My question is, is there any particular reason to not support in upload_file API, since the put_object already supports it. Using put_object_tagging is feasible but not desired way for me as it will double the current calls made to S3 API. The system currently makes about 1500 uploads per second. Individual file size varies from 200kb to 10 Mb.

[swetashre]

@bhandaresagar - Thanks for your reply. We just need to implement in s3transfer first and then it would be available in boto3.
It seems like there is already a request for adding Tagging to the ALLOWED_UPLOAD_ARGS. Let's track the progress of the issue under this one #94.

[bhandaresagar]

@swetashre Thanks a lot, if possible can you confirm if I can modify upload_args as shown above till this is supported in boto3.

[swetashre]

@bhandaresagar - Yeah you can modify upload_args for your use case till this is supported in boto3. I am closing this one as this issue is a duplicate of #94. Let's track the progress of the issue under this one #94.

[tomazberisa]

Since boto/s3transfer#94 is unresolved as of today and there are 2 open PRs (one of which is over 2 years old: boto/s3transfer#96 and boto/s3transfer#142), one possible interim solution is to monkey patch s3transfer.manager.TransferManager.

Working example for S3 object copy (in Python 3):

import boto3
import copy
from s3transfer.manager import TransferManager
import unittest.mock

def allowed_copy_args_fixed():
    args_copy = copy.deepcopy(TransferManager.ALLOWED_COPY_ARGS)
    if "Tagging" not in args_copy:
        args_copy.append("Tagging")
    if "TaggingDirective" not in args_copy:
        args_copy.append("TaggingDirective")
    return args_copy

s3 = boto3.resource("s3")

args_fixed = allowed_copy_args_fixed()
with unittest.mock.patch(
    "s3transfer.manager.TransferManager.ALLOWED_COPY_ARGS", args_fixed
):
    response = s3.Object("to_bucket", "to_key").copy(
        CopySource={"Bucket": "from_bucket", "Key": "from_key"},
        ExtraArgs={"Tagging": "key1=value2&key2=value2", "TaggingDirective": "REPLACE"}
    )

[WadePolo]

@swetashre - I'm also going to jump in here and say that this feature would be extremely useful for those of us using replication rules that are configured to pick up tagged objects that were uploaded programmatically. Currently, we are using the modified allowed keyword list that @bhandaresagar originally posted to bypass this limitation.

[smithericsmith]

This would be very helpful for me as well. I avoid using the upload_file() call because it does not support Tagging, so I am forced to read the contents into memory and use put_object() simply because I want to have the files Tagged when created.

[drake-adl]

While I don't see that issue 94 is resolved, the Tagging directive seems to be supported now: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig

ALLOWED_UPLOAD_ARGS = ['ACL', 'CacheControl', 'ContentDisposition', 'ContentEncoding', 'ContentLanguage', 'ContentType', 'Expires', 'GrantFullControl', 'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata', 'RequestPayer', 'ServerSideEncryption', 'StorageClass', 'SSECustomerAlgorithm', 'SSECustomerKey', 'SSECustomerKeyMD5', 'SSEKMSKeyId', 'Tagging', 'WebsiteRedirectLocation']¶

[vortexing]

I'm seeing Tagging as an option but still having trouble figuring out the actual formatting of the tag set to use. I can't seem to find any examples of the boto3 upload_file/ExtraArgs Tagging. @drake-adl did you manage to get an example of a tagset that works?

[drake-adl]

We utilize def convert_dict_to_string(tagging): return "&".join([k + "=" + v for k, v in tagging.items()]). The input param is a dictionary.

You can use: s3.put_object_tagging or s3.put_object with a Tagging arg.