SEAB-6452: add hour to bucket key for GitHub redelivery #166
Conversation
There was a problem hiding this comment.
This works fine, but gonna put this other thing out there:
Including parts of the date (previously the year/month/day, and here, the hour) in the keys helps us to retrieve the events that correspond to our target time range, without downloading lots of other irrelevant stuff. However, if we engineer the "resubmitter" processing correctly, we need not resubmit the entire set of downloaded events: we can examine the created_at date of each downloaded event, and only submit the subset that corresponds to the exact time range for which we want to resubmit.
| @@ -270,7 +270,8 @@ function logPayloadToS3(body, deliveryId) { | |||
| const uploadYear = date.getFullYear(); | |||
| const uploadMonth = (date.getMonth() + 1).toString().padStart(2, "0"); // ex. get 05 instead of 5 for May | |||
| const uploadDate = date.getDate().toString().padStart(2, "0"); // ex. get 05 instead of 5 for the 5th date | |||
| const bucketPath = `${uploadYear}-${uploadMonth}-${uploadDate}/${deliveryId}`; | |||
| const uploadHour = date.getHours().toString().padStart(2, "0"); // ex. get 05 instead of 5 for the 5th hour | |||
| const bucketPath = `${uploadYear}-${uploadMonth}-${uploadDate}/${uploadHour}/${deliveryId}`; | |||
There was a problem hiding this comment.
Apologies, I should have noted this in the previous PR, but why not separate each date component by a slash? That way they'd be more browseable.
Also, would recommend changing Date to Day, more specific.
Description
This PR changes the bucket key format to
YYYY-MM-DD/HH/deliveryidinstead ofYYYY-MM-DD/deliveryidsuch that we can resubmit events by the hourIssue
https://ucsc-cgl.atlassian.net/browse/SEAB-6452
Security
If there are any concerns that require extra attention from the security team, highlight them here.
Please make sure that you've checked the following before submitting your pull request. Thanks!