Improve S3 key generation logic in the writer #711
Conversation
| var key string | ||
| if len(batch.Records) > 0 { | ||
| // Convert the key bytes to string | ||
| keyBytes := batch.Records[0].Key.Bytes() |
There was a problem hiding this comment.
Oof, that could go wrong. For instance, the key could be structured data, in which case you'd get a JSON string (which is not in line with the key naming guidelines). Also, the key is related to the first record, the rest of the batch might not be related to that. I'm not a fan of this change, tbh.
I'm thinking how else we could achieve what you're trying to do. We could introduce a metadata field (e.g. s3.key) and use that as the object key, if it's present. You could then use a processor to set that metadata field. The issue remains that the first record in the batch is the only record for which this field would be respected, unless we change the logic to group records based on the metadata field and create multiple objects.
There was a problem hiding this comment.
@lovromazgon this makes sense. I actually run into that exact issue of the key being structured data when the source was a Postgres database (I previously was testing with a Generator source). I can try the metadata approach to get by for the demo
3 participants
Description
Currently the S3 destination connector creates the object's name in the S3 bucket using a timestamp. This change uses the record's key as the object name if available. Otherwise it falls back to the previous method of using the timestamp
Fixes # (issue)
Quick checks: