OpensearchExporter: Dynamic log index feature puts logs in wrong index

[Confushion]

Component(s)

exporter/opensearch

What happened?

Description

When using dynamic logs_index using a custom placeholder e.g. "{k8s.container.name}", the resulting index is not always the right one.

Steps to Reproduce

• Setup opentelemetry-collector (or operator) inside a Kubernetes cluster
• Setup an opensearch logs exporter with custom logs_index: "logs-otel-%{k8s.container.name}"

Expected Result

Each logentry should be placed in its proper index with the name of the container that it originates from.

Actual Result

A lot of logentries end up in arbitrary indices.

Looks like somewhere in the process of (bulk-)writing logentries, sets of logentries are written to opensearch with the dynamic logs_index feature only evaluated on the first entry.
Enabling or disabling batch-features do not seem to have any effect on this issue.

See screencapture of kibana output below: field "resource.k8s.container.name" should be incorporated in the "_index" but in a lot of cases it is different.

Collector version

v0.129.0 and higher

Environment information

Environment

Kubernetes: 1.33
Docker image: otel/opentelemetry-collector-contrib:0.129.0 (up to 0.136.0)

OpenTelemetry Collector configuration

exporters:
  opensearch/logs:
    http:
      endpoint: <my opensearch endpoint>
    logs_index: "logs-otel-%{k8s.container.name}"
    logs_index_fallback: "default"
    logs_index_time_format: "yyyy.MM.dd"
...

service:
  pipelines:
    logs:
      exporters:
        - opensearch/logs
...

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/opensearch: @ps48

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[constanca-m]

I removed the needs triage, but this issue requires more investigation. I quickly tried to reproduce it in a unit test and found no issues:

func TestIndexResolver_ResolveLogIndex_WithKubernetesContainerName(t *testing.T) {
	resolver := newIndexResolver()
	cfg := &Config{
		LogsIndex:           "logs-otel-%{k8s.container.name}",
		LogsIndexFallback:   "default",
		LogsIndexTimeFormat: "yyyy.MM.dd",
	}
	ts := time.Date(2025, 6, 7, 0, 0, 0, 0, time.UTC)

	tests := []struct {
		containerName string
		expected      string
	}{
		{
			containerName: "rundeck",
			expected:      "logs-otel-rundeck-2025.06.07",
		},
		{
			containerName: "kube2iam",
			expected:      "logs-otel-kube2iam-2025.06.07",
		},
		{
			containerName: "test-3",
			expected:      "logs-otel-test-3-2025.06.07",
		},
	}

	for _, test := range tests {
		t.Run(test.containerName, func(t *testing.T) {
			ld := plog.NewLogs()
			rl := ld.ResourceLogs().AppendEmpty()
			rl.Resource().Attributes().PutStr("k8s.container.name", test.containerName)
			index := resolver.ResolveLogIndex(cfg, ld, ts)
			expected := "logs-otel-" + test.containerName + "-2025.06.07"
			require.Equal(t, expected, index)
		})
	}
}

[VihasMakwana]

@Confushion There's a caveat to consider. k8s.container.name doesn't always resolve to resource.attributes["k8s.container.name"].
If there are any scope level or log level attributes with the same name, it will consider that instead of the resource attributes.

I suggest to use debugexporter to inspect individual log records and verify if there are no conflicting attributes at different levels.

[Confushion]

I finally had some time to do some debugging and I think I might have found something...

In function "pushLogData" in file "sso_log_exporter.go" a newLogBulkIndexer is created for an entire block "ld plog.Logs" and the indexName for te indexer is being resolved.

Inside this function the indexResolver will make a "flat-map" of all resource and log record attributes.
This flat-map is then searched for the right resource attribute to create the indexName.

However, an object "ld plog.Logs" can contain multiple resource records (and multiple log records).

In the debug example that I post below, the "ld plog.Logs" object contains 4 resource logs and 7 log records:

- ResourceLog #0
    - LogRecord #0
- ResourceLog #1
    - LogRecord #0
    - LogRecord #1
- ResourceLog #2
    - LogRecord #0
    - LogRecord #1
- ResourceLog #3
    - LogRecord #0
    - LogRecord #1

ResourceLog #0 contains attribute:
  - k8s.container.name: Str(kube-proxy)
ResourceLog #1, #2 and #3 all contain attribute:
  - k8s.container.name: Str(secret-watcher)

What I can tell so far is that only the attribute "k8s.container.name" of the last ResourceLog is being used for resolving the indexName. This would explain that "LogRecord #0" of "ResourceLog #0" is being written to the wrong indexName.

I think that instead of a LogBulkIndexer per "ld plog.Logs" object, we need a LogBulkIndexer per ResourceLog object.

Please correct me if I'm wrong....

Attached:

• Screendump of generated opensearch log records
• Debug Log of affected log records

debug_logs.txt