k8s namespace as dimension in awsemf exporter

[crigertg]

I'm exporting metrics to cloudwatch and want the namespace of the containers/services/pods to be visible as a Dimension. But currently this is not working because the awsemf exporter forces me to configure a Namespace which seems to overwrite the Namespace which is available from the awscontainerinsightreceiver (see here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscontainerinsightreceiver )

In Cloudwatchlogs I can see that there is the correct Namespace but the metric is pushed with the namespace I have to configure for for the awsemf exporter.

This is my current configuration:

    extensions:
      health_check: 
      sigv4auth:
        region: eu-central-1
    receivers:
      awscontainerinsightreceiver:
        collection_interval: 
        container_orchestrator: 
        add_service_as_attribute: 
        prefer_full_pod_name: 
        add_full_pod_name_metric_label: 
    processors:
      batch/metrics:
        timeout: 60s
      attributes:
        actions:
          - key: foo
            action: insert
            value: bar
          - key: namespace
            action: insert
            from_attribute: Namespace
    exporters:
      awsemf:
        namespace: ContainerInsights
        log_group_name: '/aws/containerinsights/eks-dev/performance'
        log_stream_name: InputNodeName
        region: eu-central-1
        resource_to_telemetry_conversion:
          enabled: true
        dimension_rollup_option: NoDimensionRollup
        parse_json_encoded_attr_values:
          - Sources
          - kubernetes.namespace_name
        metric_declarations:
          - dimensions: [[Service, Namespace, ClusterName], [ClusterName]]
            metric_name_selectors:
              - service_number_of_running_pods
    service:
      pipelines:
        metrics:
          receivers:
          - awscontainerinsightreceiver
          processors:
          - batch/metrics
          - attributes
          exporters:
          - awsemf
      extensions:
      - health_check

As you might see I've tried to set a new namespace field which I would be able to use with a processor but this seems not to be working too. In Clloudwatch logs it always looks like this.

{
    "ClusterName": "eks-dev",
    "Namespace": "loki",
    "NodeName": "ip-10-1-79-141.eu-central-1.compute.internal",
    "Service": "loki",
    "Sources": [
        "apiserver"
    ],
    "Timestamp": "1670489791032",
    "Type": "ClusterService",
    "Version": "0",
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Namespace": "ContainerInsights",
                "Dimensions": [
                    [
                        "ClusterName",
                        "Namespace",
                        "Service"
                    ],
                    [
                        "ClusterName"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "service_number_of_running_pods",
                        "Unit": "Count"
                    }
                ]
            }
        ],
        "Timestamp": 1670489791032
    },
    "foo": "bar",
    "kubernetes": "{\"namespace_name\":\"loki\",\"service_name\":\"loki\"}",
    "service_number_of_running_pods": 1
}

For the moment I'm stuck and I would appreciate any help with this issue. I don't understand why there is a namespace parameter for the awsemf exporter. If not set it just writes default in the namespace field. So I need a possibility to use the real namespace.

[crigertg]

See: #17030