[exporter/awss3exporter] Encode and upload Parquet file

[pbootly]

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

awss3exporter supports JSON and OTLP-proto formats, which are functional for telemetry storage but inefficient for large-scale analytic tools. Folks who might want to analyze data with tools like Athena, Spark, or Glue are forced to manage this conversion out of band (separate process) or deal with less efficient query times.

Describe the solution you'd like

Add Parquet encoding support to awss3exporter, allowing data to be written in a columnar format with optional Hive-style partitioning.

Something like:

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    encoding: parquet

I'd imagine we'd define separate Parquet schemas per signal type.

Describe alternatives you've considered

Looked at other similar requests (resulting in parquet files in S3) such as #44124

Export to file as JSON and convert to Parquet with tools like duckdb/custom scripts and upload

• Requires separate service to manage data pipeline
• Requires something to manage truncation of otel written json file

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski @Erog38

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[pbootly]

I should add, this is an implementation that I'd be keen to undertake, but wanted to get thoughts on placement surrounding the suggestion 👍

[Erog38]

I can understand the idea behind this belonging to the s3 exporter, but it facilitates a fairly substantial set of requirements to ensure the format stays consistent and compatible with existing functionality.

I suspect that this may end up better suited as a new exporter to not over-complicate a single one.

I don't have the experience with Parquet to know what this requires long term, but I'm very open to the conversation either way.