Description
Problem
Hi, I'm using Fluentd agent to scrape streams from CloudWatch log groups. The issue I'm facing is that data is received only for specific log groups (those with names that start with /aws/lambda). These kinds of log groups have a specific template for log stream names, such as:
2024/07/04/[$LATEST]80cd86893da74d0783015c4173ce6eda
**We have permissions to fetch all log groups other than /aws/lambda using ROLE_ARN from AWS CLI
Attached is the Fluentd configuration:
system:
log_level: error
log_file: /var/log/fluent/fluentd.log
config:
- !include config.d/*.yaml
- match:
$type: forward
server:
name: HOST
host: HOST
port: 3000 - source:
$type: cloudwatch_logs
tag: cloudwatch.input1
log_group_name: LOG_GROUP_NAME
use_todays_log_stream: true
include_metadata: true
region: REGION
aws_use_sts: true
aws_sts_role_arn: ROLE_ARN
parse:
$type: none
** no errors in fluentd logs
Using Fluentd and CloudWatchLogs plugin versions
fluentd 1.16.3, within docker
OS:
NAME="Amazon Linux"
VERSION="2"
- Dependent gem versions
- paste result of
fluent-gem list,td-agent-gem listor your Gemfile.lock:
- paste result of
aws-eventstream (1.3.0)
aws-partitions (1.907.0, 1.867.0, 1.866.0)
aws-sdk-cloudwatch (1.84.0)
aws-sdk-cloudwatchlogs (1.76.0, 1.71.0)
aws-sdk-core (3.191.6, 3.190.0)
aws-sdk-kms (1.74.0)
aws-sdk-s3 (1.141.0)
aws-sigv4 (1.8.0)