Open
Description
Overview
This issue outlines the comprehensive roadmap for enhancing OpenSearch's observability capabilities, focusing on OpenTelemetry integration, improved querying languages, and unified observability experiences.
1. OTel-Aligned Recommended Architecture
2. Telemetry ingestion with Data prepper
- New OTel format output for logs, traces and metrics: OTEL sources should create events without any transformations data-prepper#5259
- Unified OTLP endpoint for logs, traces and metrics: [RFC] OTel telemetry unified source data-prepper#5596
- Support for Prometheus as a sink for OTel metrics: Support Prometheus as a Sink storage (remote-write model) data-prepper#3028
3. PPL as a Querying language for telemetry signals
- New engine based on Apache Calcite: [RFC] Integrate Apache Calcite as a new query engine for SQL and PPL sql#3229
4. Explore and investigate telemetry with curated experiences:
- Discover Logs: [RFC] New Discover experience for Observability Workspaces OpenSearch-Dashboards#9826
- PPL based explorations with auto-complete
- Natural language support
- Query based Logs to metrics visualizations
- Discover Traces: [RFC] Introduce React Flow Library in OpenSearch Dashboards OpenSearch-Dashboards#9898
- PPL based trace explorations
- Curated exploration UI for traces
- Trace details
- Service Map
- Easy correlations with associated logs
- Facets
- Discover Metrics: [RFC] Prometheus as a first class datasource OpenSearch-Dashboards#9535
- PromQL based explorations with Auto-complete
- Support creation on visualizations with PromQL
5. Prometheus as first class datasource
- Prometheus as a datasource with Resource and Query APIs: Add direct-query-core module for prometheus integration sql#3440
6. Single pane of glass for all telemetry
- Dashboards variables
- PPL based filtering on visualizations in Dashboards
7. Monitoring
- PPL based alert and trigger setup: [RFC] PPL Alerting Support in OpenSearch alerting#1819
- Prometheus alerts in OpenSearch Dashboards: [RFC] Prometheus as a first class datasource OpenSearch-Dashboards#9535
8. Trace & Services enhancements
- Performance & Query Optimization: [WIP] Trace Query & Page Load Optimization #2334
- Span Kind based filtering: Update custom traces table with filters #2178
- Enrich traces table with attributes: Traces - Custom source switch to data grid #2390
- Updated service map with reset and drill down on dependencies & Gantt-chart updates: Gantt Chart / Service Map followup #2294
- Cross Cluster support: [FEATURE] Trace overview page and enable CCS for tracing data #2052
- Easy correlations between logs, traces and services: Support custom logs correlation #2375
9. Support for OpenSearch exporter in OTel Collector contrib
- Support for Dynamic indexes
- Time format suffix to index name
- Service map processor
- OTel format ingestion