Skip to content

S3 access log source fails transformation due to dependency on "name" config field #110

@timoguin

Description

@timoguin

Problem

The managed log source type AWS_S3ACCESS fails if the name property in config isn't set to aws_s3access.

Take this example config:

name: aws_s3_access

managed:
  type: AWS_S3ACCESS

ingest:
  s3_source:
    bucket_name: my-s3-access-logs-bucket

This fails with lots of Failed to find avro schema errors in the transformer.

It only works if the config is changed to name: aws_s3access.

Expected Behavior

The name field should not be depended on for determining data schema in the transformer, since it is a human-facing identifier. These messages make it through the data batcher as expected, and they are tagged appropriately with the aws_s3access source type.

I expect them to be transformed appropriately.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions