Skip to content

Archive Cloudwatch Log Groups by Tag and update Python to 3.9 #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Archive Cloudwatch Log Groups by Tag and update Python to 3.9 #11

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a7946b8
adding ability to subscribe log groups by AWS Tag as well as the exis…
Oct 11, 2022
6156aad
fixing reference to LogGroupTagName var
Oct 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions CloudWatch2S3.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ Metadata:
Parameters:
- SubscribeSchedule
- LogGroupNamePrefix
- LogGroupTagName
ParameterLabels:
AllowedAccounts:
default: Allowed Accounts
Expand All @@ -112,6 +113,8 @@ Metadata:
default: Export Format
LogGroupNamePrefix:
default: Required Log Group Name Prefix
LogGroupTagName:
default: Archive log groups by Tag
ProcessorBufferIntervalHint:
default: Processing Lambda Buffer Timeout
ProcessorBufferSizeHint:
Expand Down Expand Up @@ -199,6 +202,10 @@ Parameters:
Description: Prefix to match against log group that should be exported (leave
empty to export all log groups)
Type: String
LogGroupTagName:
Default: 'archive_log_group_to_s3'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the default should be empty to keep backwards compatibility.

Description: Adding this tag to a log group will force the scheduled LogSubscriberFunction to archive tagged log group (leave empty to not filter by tags)
Type: String
ProcessorBufferIntervalHint:
Default: '60'
Description: Processing Lambda buffer timeout (in seconds, only in raw format
Expand Down Expand Up @@ -582,7 +589,7 @@ Resources:
Fn::GetAtt:
- LogProcessorRole
- Arn
Runtime: python3.6
Runtime: python3.9
Timeout: 300
Type: AWS::Lambda::Function
LogProcessorRole:
Expand Down Expand Up @@ -663,7 +670,16 @@ Resources:

def subscribe_all():
for log_group_name in matched_log_groups("${LogGroupNamePrefix}"):
subscribe(log_group_name)
if not "${LogGroupTagName}":
subscribe(log_group_name)
else:
tags_in_group_response = logs_client.list_tags_log_group(
logGroupName = log_group_name
)
print("tags_in_group_response", tags_in_group_response)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this print?

for tag in tags_in_group_response["tags"]:
if tag == "${LogGroupTagName}":
subscribe(log_group_name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a break here just in case? Don't want for it to be possible to get double subscribe (which I think results in an exception).



def unsubscribe_all():
Expand Down Expand Up @@ -737,7 +753,7 @@ Resources:
Fn::GetAtt:
- LogSubscriberRole
- Arn
Runtime: python3.6
Runtime: python3.9
Timeout: 300
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's increase the timeout since we're doing much more.

Type: AWS::Lambda::Function
LogSubscriberPermission:
Expand Down Expand Up @@ -774,6 +790,7 @@ Resources:
- logs:DeleteSubscriptionFilter
- logs:DescribeLogGroups
- logs:PutSubscriptionFilter
- logs:ListTagsLogGroup
Effect: Allow
Resource: '*'
Sid: Logs
Expand Down Expand Up @@ -810,4 +827,4 @@ Resources:
- LogSubscriberFunction
- Arn
Type: Custom::Subscriber
Transform: AWS::Serverless-2016-10-31
Transform: AWS::Serverless-2016-10-31