[BUG] - k8s integration ingestion method is not clear enough #201
Description
What is the bug?
We've been able to install the integration dashboards as well as the sample data. However, it is unclear how one is expected to ingest the relevant data.
The repo mentions using OpenTelemetry collectors to collect the data. However, the documentation in the OpenTelemetry website is insufficient for this - I'm getting errors from the opentelemetry pod.
2024-12-12T13:35:10.976Z error internal/queue_sender.go:92 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue_sender.go:92
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43
I've also tried the fluentbit configuration though unfortunately it seems to be irrelevant. The fluentbit kubernetes filter only collects partial data from Kubernetes which does not match the data required by the integration's dashboards (such as the deployment details).
There is a fluent bit issue describing this:
The filebeat versions supported by Opensearch have a similar issue, so it is not an option. The other options mentioned in that issue require developing support for the additional fields.
How can one reproduce the bug?
Use attached data prepper and otel configurations in accordance with the documentation.
What is the expected behavior?
Kubernetes integration data being loaded into the cluster.
I would appreciate templates for data prepper and otel (assuming that is the solution) that correctly collect the integration data for k8s.
What is your host/environment?
- OS: Linux (EKS)
- Opensearch 2.14
data-prepper-k8s.yaml.txt
values.yaml.txt
Thanks.