Data prepper fails when sending traces from different EKS (multiple data prepper singletons) to ES #479
Description
Using data prepper to send traces from EKS clusters (3) to the same elastic search. This works when sending traces from just one EKS but with multiple EKS (therefore multiple data-prepper) , this fails with :
{"error":{"root_cause":[{"type":"invalid_alias_name_exception","reason":"Invalid alias name [otel-v1-apm-span], an index exists with the same name as the alias","index_uuid":"0-ZhKKdyRHeJjw4QKjF-Yg","index":"otel-v1-apm-span"}],"type":"invalid_alias_name_exception","reason":"Invalid alias name [otel-v1-apm-span], an index exists with the same name as the alias","index_uuid":"0-ZhKKdyRHeJjw4QKjF-Yg","index":"otel-v1-apm-span"},"status":400}
To Reproduce
Steps to reproduce the behavior:
- Run data prepper as a deployment of 1 from 2 different EKS clusters sending traces to same ES
- data prepper : amazon/opendistro-for-elasticsearch-data-prepper:0.7.1-alpha
- Observe error in data-prepper as it tries to send traces
Expected behavior
Traces should be sent from both EKS clusters to ES . Alias should be configurable within data prepper configmap if need be.
Screenshots
Am I missing something in the config ? I dont see a way to set the alias in the data prepper configmap.