Otel Metrics is Empty but Index has documents

[harpreetcywarden]

Using Data Prepper with index_type: metric-analytics-plain, the metrics indices (e.g., metrics-otel-v1-000001) are:

Ingested successfully

Visible in Discover

Not showing up in Observability → Metrics UI dropdown, even though the required mappings like name, kind, and instrumentationScope.name are correctly defined.

Steps to Reproduce
Ingest OTel hostmetrics into AWS OpenSearch using Data Prepper with:

index_type: metric-analytics-plain

Check Otel Index dropdown → No index appears

Expected Behavior
The OTel metrics index (using the built-in metric-analytics-plain schema) should appear in the Otel Index dropdown and be available for visualization in the Metrics UI.

Environment Details
OpenSearch version: AWS-managed OpenSearch Service (e.g., v2.x)

OpenSearch Dashboards Observability plugin: 2.16 or latest

Data Prepper: version (used for ingestion)

Index mappings: Verified name, kind, and instrumentationScope.name are keyword type

Similar Reports
GitHub Issue #1927:

"There are no index to be selected at Otel Index selection box"

Request
Please investigate why Data Prepper’s OTel metrics ingestion — following the documented schema — does not result in visibility under the Observability → Metrics UI. This issue is currently blocking metric visualization out-of-the-box for OpenTelemetry users.

Otel Config:

receivers:
hostmetrics/fast:
collection_interval: 1s
scrapers:
cpu:
memory:

hostmetrics/slow:
collection_interval: 1s
scrapers:
disk:
filesystem:
load:

hostmetrics/ultraslow:
collection_interval: 3s
scrapers:
processes:

processors:

resourcedetection:
detectors: [env, system, ec2]

batch:
send_batch_size: 512
timeout: 5s

memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 256

exporters:
otlp/data-prepper/metrics:
endpoint: "localhost:21892"
tls:
insecure: true
timeout: 30s
sending_queue:
enabled: true
storage: file_storage

extensions:
health_check:
file_storage:
directory: /var/lib/otelcol/ingestion_queue
create_directory: true

service:
extensions: [health_check, file_storage]

pipelines:
metrics:
receivers: [hostmetrics/fast, hostmetrics/slow, hostmetrics/ultraslow]
processors: [memory_limiter, batch, resourcedetection]
exporters: [otlp/data-prepper/metrics]

Data Prepper Pipeline:

metric-processing-pipeline:
source:
otel_metrics_source:
port: 21892
ssl: false

buffer:
bounded_blocking:
buffer_size: 2048
batch_size: 512

processor:
- otel_metrics: {}
- copy_values:
entries:
- from_key: /time
to_key: /@timestamp

sink:
- opensearch:
hosts: ["https://search-opensearcxxxxxx.es.amazonaws.com:443"]
username: "xxxx"
password: "xxxxx"
index_type: metric-analytics-plain
index: otel-metrics

[ps48]

Hi @harpreetcywarden , Thanks for raising the issue. Can you please follow the steps mentioned here: #2172 (comment) to see if the metrics load up.

[harpreetcywarden]

Hi @ps48, thanks for the reply! 🙏

I tried using the Data Prepper pipeline as you suggested.

OTEL Collector Config:

receivers:
hostmetrics/fast:
collection_interval: 1s
scrapers:
cpu:
memory:

hostmetrics/slow:
collection_interval: 1s
scrapers:
disk:
filesystem:
load:

hostmetrics/ultraslow:
collection_interval: 3s
scrapers:
processes:

processors:
resourcedetection:
detectors: [env, system, ec2]

batch:
send_batch_size: 512
timeout: 5s

memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 256

exporters:
otlp/data-prepper/metrics:
endpoint: "localhost:21892"
tls:
insecure: true
timeout: 30s
sending_queue:
enabled: true
storage: file_storage

extensions:
health_check:
file_storage:
directory: /var/lib/otelcol/ingestion_queue
create_directory: true

service:
extensions: [health_check, file_storage]
pipelines:
metrics:
receivers: [hostmetrics/fast, hostmetrics/slow, hostmetrics/ultraslow]
processors: [memory_limiter, batch, resourcedetection]
exporters: [otlp/data-prepper/metrics]

Data Prepper Config:

entry-pipeline:
source:
otel_metrics_source:
ssl: false
path: "/v1/metrics"
port: 21892
processor:
- otel_metrics:
sink:
- pipeline:
name: "metrics-pipeline"

metrics-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- otel_metrics:
sink:
- opensearch:
hosts: ["https://xxxxx.us-east-1.es.amazonaws.com:443"]
username: "xxxx"
password: "xxxxx"
index_type: custom
index: "ss4o_metrics-otel-%{yyyy.MM.dd}"

Issue:

When the path: "/v1/metrics" is included in the pipeline, I get the following OTEL Collector error:

Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found)

If I remove the path, the error goes away, and the metrics are ingested into OpenSearch. However, they still don't show up in the Metrics plugin.

Index Mapping:

{
"properties": {
"aggregationTemporality": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"description": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"flags": {
"type": "long"
},
"instrumentationScope": {
"type": "object",
"properties": {
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"isMonotonic": {
"type": "boolean"
},
"kind": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"metric": {
"type": "object",
"properties": {
"attributes": {
"type": "object",
"properties": {
"cpu": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"device": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"direction": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"mode": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"mountpoint": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"state": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"status": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"type": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
}
}
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"resource": {
"type": "object",
"properties": {
"attributes": {
"type": "object",
"properties": {
"cloud@account@id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cloud@availability_zone": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cloud@platform": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cloud@provider": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cloud@region": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host@id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host@image@id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host@name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host@type": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"os@type": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
}
}
},
"schemaUrl": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"startTime": {
"type": "date"
},
"time": {
"type": "date"
},
"unit": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"value": {
"type": "float"
}
}
}

I’ve followed the steps from #2172 (comment), but still facing the same issue with visualization. Let me know if there are additional mappings or field formats expected by the Metrics plugin. Appreciate your help!

Additional Notes:

• They are not using the default otel-v1-metrics index-type; instead, I'm using a custom index (ss4o_metrics-otel-*).

• Because of this, the ingested metrics do not include the @timestamp field, which might be required by the Metrics plugin to work properly.

Sample Document from Index:

{
"_index": "ss4o_metrics-otel-2025.07.14",
"_id": "noX_CZgBA2uBbsBtH08w",
"_version": 1,
"_score": null,
"_source": {
"kind": "SUM",
"flags": 0,
"description": "Bytes of memory in use.",
"serviceName": null,
"schemaUrl": "https://opentelemetry.io/schemas/1.9.0",
"isMonotonic": false,
"unit": "By",
"aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE",
"exemplars": [],
"name": "system.memory.usage",
"startTime": "2025-07-11T11:21:44Z",
"time": "2025-07-14T17:31:32.375511629Z",
"value": 533696512,
"resource.attributes.cloud@account@id": "1xxxxxx12",
"resource.attributes.host@name": "ip-172-31-21-104.ec2.internal",
"resource.attributes.host@id": "i-060xxxx53205305",
"resource.attributes.cloud@region": "us-east-1",
"resource.attributes.host@image@id": "ami-07xxxx32f281e",
"instrumentationScope.version": "0.128.0",
"instrumentationScope.name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/memoryscraper",
"resource.attributes.cloud@platform": "aws_ec2",
"resource.attributes.os@type": "linux",
"resource.attributes.cloud@availability_zone": "us-east-1c",
"resource.attributes.host@type": "t3.medium",
"metric.attributes.state": "used",
"resource.attributes.cloud@provider": "aws"
},
"fields": {
"startTime": [
"2025-07-11T11:21:44.000Z"
],
"time": [
"2025-07-14T17:31:32.375Z"
]
},
"sort": [
1752232904000
]
}