diff --git a/docs/images/about-page-license.png b/docs/images/about-page-license.png new file mode 100644 index 00000000..23697bf7 Binary files /dev/null and b/docs/images/about-page-license.png differ diff --git a/docs/images/action-tab.png b/docs/images/action-tab.png new file mode 100644 index 00000000..1e161ecf Binary files /dev/null and b/docs/images/action-tab.png differ diff --git a/docs/images/add-new-license-key.png b/docs/images/add-new-license-key.png new file mode 100644 index 00000000..f4c14854 Binary files /dev/null and b/docs/images/add-new-license-key.png differ diff --git a/docs/images/after-flattening.png b/docs/images/after-flattening.png new file mode 100644 index 00000000..130ff499 Binary files /dev/null and b/docs/images/after-flattening.png differ diff --git a/docs/images/alert-details-drawer.png b/docs/images/alert-details-drawer.png new file mode 100644 index 00000000..cdcc0094 Binary files /dev/null and b/docs/images/alert-details-drawer.png differ diff --git a/docs/images/associate-query.png b/docs/images/associate-query.png new file mode 100644 index 00000000..053ff5f2 Binary files /dev/null and b/docs/images/associate-query.png differ diff --git a/docs/images/condition-node.png b/docs/images/condition-node.png new file mode 100644 index 00000000..18dd714e Binary files /dev/null and b/docs/images/condition-node.png differ diff --git a/docs/images/configure-scheduled-pipelines.png b/docs/images/configure-scheduled-pipelines.png new file mode 100644 index 00000000..a11c6151 Binary files /dev/null and b/docs/images/configure-scheduled-pipelines.png differ diff --git a/docs/images/connect-nodes.png b/docs/images/connect-nodes.png new file mode 100644 index 00000000..c2065278 Binary files /dev/null and b/docs/images/connect-nodes.png differ diff --git a/docs/images/create-new-function.png b/docs/images/create-new-function.png new file mode 100644 index 00000000..13294ff6 Binary files /dev/null and b/docs/images/create-new-function.png differ diff --git a/docs/images/create-new-stream.png b/docs/images/create-new-stream.png new file mode 100644 index 00000000..d86fd92e Binary files /dev/null and b/docs/images/create-new-stream.png differ diff --git a/docs/images/download-pipeline-json.png b/docs/images/download-pipeline-json.png new file mode 100644 index 00000000..aaa83ce5 Binary files /dev/null and b/docs/images/download-pipeline-json.png differ diff --git a/docs/images/enterprise-branding.png b/docs/images/enterprise-branding.png new file mode 100644 index 00000000..c0fb8790 Binary files /dev/null and b/docs/images/enterprise-branding.png differ diff --git a/docs/images/external-destination.png b/docs/images/external-destination.png new file mode 100644 index 00000000..fba8812b Binary files /dev/null and b/docs/images/external-destination.png differ diff --git a/docs/images/filter-pipeline.png b/docs/images/filter-pipeline.png new file mode 100644 index 00000000..326302df Binary files /dev/null and b/docs/images/filter-pipeline.png differ diff --git a/docs/images/import-pipelines.png b/docs/images/import-pipelines.png new file mode 100644 index 00000000..2f469d64 Binary files /dev/null and b/docs/images/import-pipelines.png differ diff --git a/docs/images/json-file-pipeline-export.png b/docs/images/json-file-pipeline-export.png new file mode 100644 index 00000000..e4bd7998 Binary files /dev/null and b/docs/images/json-file-pipeline-export.png differ diff --git a/docs/images/license-management.png b/docs/images/license-management.png new file mode 100644 index 00000000..50d68b8a Binary files /dev/null and b/docs/images/license-management.png differ diff --git a/docs/images/license-server.png b/docs/images/license-server.png new file mode 100644 index 00000000..65bbdf57 Binary files /dev/null and b/docs/images/license-server.png differ diff --git a/docs/images/manage_pipelines.png b/docs/images/manage_pipelines.png new file mode 100644 index 00000000..6cf3efa0 Binary files /dev/null and b/docs/images/manage_pipelines.png differ diff --git a/docs/images/pipeline-editor.png b/docs/images/pipeline-editor.png new file mode 100644 index 00000000..808929e9 Binary files /dev/null and b/docs/images/pipeline-editor.png differ diff --git a/docs/images/pipeline-error-view.png b/docs/images/pipeline-error-view.png new file mode 100644 index 00000000..7332b1d3 Binary files /dev/null and b/docs/images/pipeline-error-view.png differ diff --git a/docs/images/pipeline-error.png b/docs/images/pipeline-error.png new file mode 100644 index 00000000..308830ee Binary files /dev/null and b/docs/images/pipeline-error.png differ diff --git a/docs/images/pipeline-import-json.png b/docs/images/pipeline-import-json.png new file mode 100644 index 00000000..bfe943a0 Binary files /dev/null and b/docs/images/pipeline-import-json.png differ diff --git a/docs/images/pipeline-import.png.png b/docs/images/pipeline-import.png.png new file mode 100644 index 00000000..325b38b2 Binary files /dev/null and b/docs/images/pipeline-import.png.png differ diff --git a/docs/images/pipeline-list-view.png b/docs/images/pipeline-list-view.png new file mode 100644 index 00000000..c724f0d4 Binary files /dev/null and b/docs/images/pipeline-list-view.png differ diff --git a/docs/images/pipeline-name.png b/docs/images/pipeline-name.png new file mode 100644 index 00000000..c00ad1eb Binary files /dev/null and b/docs/images/pipeline-name.png differ diff --git a/docs/images/pipeline-new-editor.png b/docs/images/pipeline-new-editor.png index e3690e20..45e06cc1 100644 Binary files a/docs/images/pipeline-new-editor.png and b/docs/images/pipeline-new-editor.png differ diff --git a/docs/images/pipeline-new-realtime-destination.png b/docs/images/pipeline-new-realtime-destination.png index 10619882..7ce55f6e 100644 Binary files a/docs/images/pipeline-new-realtime-destination.png and b/docs/images/pipeline-new-realtime-destination.png differ diff --git a/docs/images/pipeline-new-realtime-transform-condition.png b/docs/images/pipeline-new-realtime-transform-condition.png index dd8d3060..d449ddb8 100644 Binary files a/docs/images/pipeline-new-realtime-transform-condition.png and b/docs/images/pipeline-new-realtime-transform-condition.png differ diff --git a/docs/images/pipeline-new-scheduled-condition.png b/docs/images/pipeline-new-scheduled-condition.png index 79e1dc9d..5e3437c3 100644 Binary files a/docs/images/pipeline-new-scheduled-condition.png and b/docs/images/pipeline-new-scheduled-condition.png differ diff --git a/docs/images/pipeline-permission.png b/docs/images/pipeline-permission.png new file mode 100644 index 00000000..f2c7fd39 Binary files /dev/null and b/docs/images/pipeline-permission.png differ diff --git a/docs/images/pipelines-new-realtime.png b/docs/images/pipelines-new-realtime.png index de04a4db..ff06da04 100644 Binary files a/docs/images/pipelines-new-realtime.png and b/docs/images/pipelines-new-realtime.png differ diff --git a/docs/images/pipelines-tab.png b/docs/images/pipelines-tab.png new file mode 100644 index 00000000..391131c0 Binary files /dev/null and b/docs/images/pipelines-tab.png differ diff --git a/docs/images/query-output.png b/docs/images/query-output.png new file mode 100644 index 00000000..6936a497 Binary files /dev/null and b/docs/images/query-output.png differ diff --git a/docs/images/save-pipeline.png b/docs/images/save-pipeline.png new file mode 100644 index 00000000..38c99eed Binary files /dev/null and b/docs/images/save-pipeline.png differ diff --git a/docs/images/schedule-condition-node.png b/docs/images/schedule-condition-node.png new file mode 100644 index 00000000..710bcf38 Binary files /dev/null and b/docs/images/schedule-condition-node.png differ diff --git a/docs/images/schedule-connect-nodes.png b/docs/images/schedule-connect-nodes.png new file mode 100644 index 00000000..a3d2dd7f Binary files /dev/null and b/docs/images/schedule-connect-nodes.png differ diff --git a/docs/images/schedule-stream-destination.png b/docs/images/schedule-stream-destination.png new file mode 100644 index 00000000..9ccd3945 Binary files /dev/null and b/docs/images/schedule-stream-destination.png differ diff --git a/docs/images/scheduled-pipeline-config-delay.png b/docs/images/scheduled-pipeline-config-delay.png index ee78c051..ea6b8124 100644 Binary files a/docs/images/scheduled-pipeline-config-delay.png and b/docs/images/scheduled-pipeline-config-delay.png differ diff --git a/docs/images/scheduled-pipeline-list-view.png b/docs/images/scheduled-pipeline-list-view.png new file mode 100644 index 00000000..edbb09ea Binary files /dev/null and b/docs/images/scheduled-pipeline-list-view.png differ diff --git a/docs/images/scheduled-pipeline-name.png b/docs/images/scheduled-pipeline-name.png new file mode 100644 index 00000000..c534fad6 Binary files /dev/null and b/docs/images/scheduled-pipeline-name.png differ diff --git a/docs/images/scheduled-variables.png b/docs/images/scheduled-variables.png new file mode 100644 index 00000000..25d5c009 Binary files /dev/null and b/docs/images/scheduled-variables.png differ diff --git a/docs/images/search-pipeline.png b/docs/images/search-pipeline.png new file mode 100644 index 00000000..0e469c9c Binary files /dev/null and b/docs/images/search-pipeline.png differ diff --git a/docs/images/select-existing-stream.png b/docs/images/select-existing-stream.png new file mode 100644 index 00000000..d55403de Binary files /dev/null and b/docs/images/select-existing-stream.png differ diff --git a/docs/images/sort-columns.png b/docs/images/sort-columns.png new file mode 100644 index 00000000..81fc59f4 Binary files /dev/null and b/docs/images/sort-columns.png differ diff --git a/docs/images/stream-destination.png b/docs/images/stream-destination.png new file mode 100644 index 00000000..7359b8b6 Binary files /dev/null and b/docs/images/stream-destination.png differ diff --git a/docs/images/theme-config.png b/docs/images/theme-config.png new file mode 100644 index 00000000..5fd86860 Binary files /dev/null and b/docs/images/theme-config.png differ diff --git a/docs/images/view-result.png b/docs/images/view-result.png new file mode 100644 index 00000000..0655e338 Binary files /dev/null and b/docs/images/view-result.png differ diff --git a/docs/images/view-scheduled-result.png b/docs/images/view-scheduled-result.png new file mode 100644 index 00000000..c5e0450e Binary files /dev/null and b/docs/images/view-scheduled-result.png differ diff --git a/docs/performance.md b/docs/performance.md index 9a43d4ff..803d67f8 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -246,37 +246,23 @@ ZO_USE_MULTIPLE_RESULT_CACHE: "true" # Enable to use mulple result caches for qu ``` ## Query partitioning +OpenObserve improves query responsiveness by processing large result sets in smaller units called **partitions**. A partition represents a segment of the overall query based on time range or data volume. -Query performance UX is not always about delivering query results faster. Imagine if you woudn't have to wait for the results of the query but could keep getting the results of the query incrementally as they are processed. This would be similar (but slighly better) to knowing the status of where your uber driver is and how long s/he is going to take to reach you even if it takes the same time without knowing it. +- When you run a query on the Logs page or in a dashboard panel. OpenObserve divides the query into multiple partitions. +- Each partition is processed sequentially, and partial results are returned as soon as each partition completes. This reduces waiting time and improves the time to first result during long-range or high-volume queries. -In the case of a query result on log search page or getting the results on dashboard panel, OpenObserve can partition the query to get results incrementally. +Partitioning introduces a tradeoff. Smaller partitions return early results and improve responsiveness, but they also increase the number of operations the system must perform. This can extend the total time required to complete the query. OpenObserve addresses this by combining partitioning with a streaming delivery model based on **HTTP2**. -e.g. A query for 1 day may be broken into 4 queries of 6 hours each (UI would automatically do this for you) and you would see the results of first 6 hours and then incrementally get all the results. All the requests are made incrementally by the UI. By default UI uses AJAX requests for each qyery partition. +To learn more, visit the [Steaming Aggregation](https://openobserve.ai/docs/user-guide/management/aggregation-cache/) page. -While query partitioning can improve user experience greatly, it can also reduce the overall speed of getting the result. e.g. One day query was broken into 48 individual queries. Now this query without partition may have gitten completed in 6 seconds. Howver making 48 separate HTTP requests sequentially may take 24 seconds to get the results (HTTP requests have overhead). In order to tackle this you can enable websockets. You can enable websockets using: +## Mini-partition +OpenObserve uses a mini-partition to return the first set of results faster. The mini-partition is a smaller slice of the first partition and is controlled by the environment variable `ZO_MINI_SEARCH_PARTITION_DURATION_SECS`, which defines the mini-partition duration in seconds. The default value is sixty seconds. -```shell -ZO_WEBSOCKET_ENABLED: "true" -``` -Enabling websockets would also require you to setup more things if you are using a reverse proxy like nginx. -Official helm chart has all of this setup for you so you don't have to worry about it. However if you are setting it up yourself or using another environment make sure that these (or it's equivalents) are configured: -Add nginx annotations: - -```yaml - nginx.ingress.kubernetes.io/proxy-http-version: "1.1" # Enable HTTP/1.1 for WebSockets - nginx.ingress.kubernetes.io/enable-websocket: "true" - # nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive # disable keep alive to use websockets - nginx.ingress.kubernetes.io/proxy-set-headers: | - Upgrade $http_upgrade; - Connection "Upgrade"; -``` -Websockets as of 0.14.1 is an experimental feature and you must enable it from the UI as well. `Settings > General settings > Enable Websocket Search` -Result caching + Query partition + Websockets = Huge performance gains and great UX. ## Large Number of Fields diff --git a/docs/user-guide/.pages b/docs/user-guide/.pages index 88829c92..7a0a425f 100644 --- a/docs/user-guide/.pages +++ b/docs/user-guide/.pages @@ -1,11 +1,10 @@ nav: - - Concepts: concepts.md - Logs: logs + - Traces: traces - Metrics: metrics - Streams: streams - Ingestion: ingestion - Pipelines: pipelines - - Traces: traces - Alerts: alerts - Dashboards: dashboards - Actions: actions diff --git a/docs/user-guide/alerts/alert-history.md b/docs/user-guide/alerts/alert-history.md index e6cc5a2d..15ceb132 100644 --- a/docs/user-guide/alerts/alert-history.md +++ b/docs/user-guide/alerts/alert-history.md @@ -2,13 +2,16 @@ This guide provides information about how the Alert History feature in OpenObserve works, where the data originates from, who can access it, how to interpret the Alert History table, and how to debug failed or skipped alerts. ## Overview -All alert trigger data is stored in the triggers stream inside the `_meta` organization. Access to `_meta` is restricted and managed through IAM, which means only users with elevated privileges can view it directly. +OpenObserve records alert evaluation events in a dedicated stream called `triggers`. Each organization has its own `triggers` stream. When an alert is evaluated, the evaluation result is written to the triggers stream inside that organization. OpenObserve also writes a copy of the same event to the `triggers` stream in the `_meta` organization for organization level monitoring. -The Alert History page brings this information into the user’s organization. It provides visibility into alert evaluations, including when each alert ran, its evaluation duration, and its final status. This design allows alert owners to monitor alert performance and troubleshoot issues without requiring access to the `_meta` organization. +> An evaluation is the system checking whether the alert’s condition is true. For scheduled alerts, this check happens at the set frequency. For real time alerts, the check happens whenever new data arrives. The condition defines what should trigger the alert. +A trigger happens when the evaluation finds the condition to be true. This creates a firing event and can send a notification if one is set. !!! note "Who can access it" - Any user who has permission to view, update, or delete alerts can also access Alert History. This ensures that alert managers and operators can analyze their alerts’ execution history without depending on users with higher administrative access. + Any user who has permission to view, update, or delete alerts can also access Alert History. Users do not need access to the `_meta` organization to view alert history for their own organization. Access to the `_meta` organization is only required when administrators need to review alert evaluation events across all organizations. +!!! note "Environment variable" + `ZO_USAGE_REPORT_TO_OWN_ORG`: Controls where alert evaluation events are stored. When it is enabled, OpenObserve writes each evaluation event to the organization’s own `triggers` stream and also keeps a copy in the `_meta` organization. This allows users to view their alert history within their own organization without requiring access to `_meta`, while still supporting organization level debugging from the `_meta` organization. ## How to interpret the Alert History table ![alert-history](../../images/alert-history.png) @@ -22,8 +25,9 @@ Each row represents one alert evaluation. - **Start Time** and **End Time**: The time range of data evaluated. - **Duration**: How long the alert condition remained true. - **Status**: The result of the alert evaluation. -- **Retries**: Number of times the system retried alert delivery when the destination did not acknowledge it. The system retries up to three times.
-**Note**: The environment variable `ZO_SCHEDULER_MAX_RETRIES` defines how many times the scheduler retries a failed execution. +- **Retries**: Number of times the system retried alert delivery when the destination did not acknowledge it. The system retries up to three times.
**Note**: The environment variable `ZO_SCHEDULER_MAX_RETRIES` defines how many times the scheduler retries a failed execution. +- **Total Evaluations**: Shows how many times the alert rule has been evaluated over the selected time range. Each evaluation corresponds to one run of the alert’s query and condition. +- **Firing Count**: Shows how many of those evaluations resulted in a firing event, that is, how many times the alert condition was satisfied and the alert was triggered. - **Actions**: Opens a detailed view that includes: - **Evaluation Time**: The time taken to complete the alert’s search query. @@ -37,6 +41,9 @@ Each row represents one alert evaluation. - **condition_not_met**: The configured alert condition was not satisfied for that time range. - **skipped**: The scheduled evaluation window was missed due to a delay, and the system evaluated the next aligned window. +- **Alert Details** drawer: Opens when the user clicks an alert in the Alerts list. The drawer displays the alert condition, description, and evaluation history. + +![Alert details drawer](../../images/alert-details-drawer.png) ## How to debug a failed alert This process applies only to users who have access to the `_meta` organization. ![debug-alert-history](../../images/debug-alert-history.png) @@ -62,5 +69,6 @@ This process applies only to users who have access to the `_meta` organization. ## Why you might see a skipped status A **skipped** status appears when a scheduled alert runs later than its expected window.
-For example, an alert configured with a 5-minute period and 5-minute frequency is scheduled to run at 12:00 PM. It should normally evaluate data from 11:55 to 12:00. -If the alert manager experiences a delay and runs the job at 12:05 PM, it evaluates the current aligned window (12:00 to 12:05) instead of the earlier one. The earlier window (11:55 to 12:00) is marked as skipped to indicate that evaluation for that range did not occur because of delay in job pickup or data availability. \ No newline at end of file +For example, an alert configured with a 5-minute period and 5-minute frequency is scheduled to run at 12:00 PM.
It should normally evaluate data from 11:55 to 12:00. +If the alert manager experiences a delay and runs the job at 12:05 PM, it evaluates the current aligned window (12:00 to 12:05) instead of the earlier one.
The earlier window (11:55 to 12:00) is marked as skipped to indicate that evaluation for that range did not occur because of delay in job pickup or data availability. + diff --git a/docs/user-guide/dashboards/custom-charts/.pages b/docs/user-guide/dashboards/custom-charts/.pages index def797af..dd17077c 100644 --- a/docs/user-guide/dashboards/custom-charts/.pages +++ b/docs/user-guide/dashboards/custom-charts/.pages @@ -4,4 +4,5 @@ nav: - Custom Charts with Flat Data: custom-charts-flat-data.md - Custom Charts with Nested Data: custom-charts-nested-data.md - Event Handlers and Custom Functions: custom-charts-event-handlers-and-custom-functions.md - - Custom charts for metrics using PromQL: custom-charts-for-metrics-using-promql.md \ No newline at end of file + - Custom Charts for Metrics Using PromQL: custom-charts-for-metrics-using-promql.md + - Custom Charts for Metrics Using Multiple PromQL Queries: custom-charts-for-metrics-using-multiple-promql-queries.md diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md b/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md index 05142f68..b0c8c0c7 100644 --- a/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md +++ b/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md @@ -5,8 +5,7 @@ description: >- --- This guide shows how to make your [custom charts](what-are-custom-charts.md) interactive using event handlers and reusable custom functions (customFn). -## What Are Event Handlers - +## What are event handlers? Event handlers let you define what happens when a user interacts with the chart, such as clicking or hovering over a data point. Use event handlers in the custom chart logic to display messages, log actions, or apply filters based on user input. Common event handlers: @@ -21,7 +20,7 @@ Before you begin, note the following: - Use the `o2_events` block to specify the event type, such as `click`. - Associate the event with a function that will run when the event occurs. -## How to Create Event Handlers +## How to create event handlers ### Step 1: Create a basic chart with labels and values @@ -92,7 +91,7 @@ o2_events: { } ``` -## What Are Custom Functions +## What are custom functions? Custom functions (customFn) are special sections inside the chart’s `option` object where you can define reusable functions. @@ -102,7 +101,7 @@ These functions can be used: - To apply logic such as filters - To keep your event handlers simple -## How to Create Custom Functions +## How to create custom functions **Example**: You have a chart showing bars for A, B, and C. When the user clicks on a bar, you want to format the output like: ``` diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md b/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md index 9eb1f935..0736803d 100644 --- a/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md +++ b/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md @@ -5,12 +5,12 @@ description: >- --- The following step-by-step instructions can help you build a [custom chart that expects flat data](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). -## Use Case +## Use case Build a custom **heatmap chart** to understand which organization and search type combinations generate the most query load. -## Before You Begin +## Before you begin To build a custom chart, you need to bridge two things: @@ -19,7 +19,7 @@ To build a custom chart, you need to bridge two things: > **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart. -## Step 1: Understand the Ingested Dataset +## Step 1: Understand the ingested dataset In OpenObserve, the data ingested into a stream is typically in a flat structure. **Example:** In the following dataset, each row represents a single event or query log with its own timestamp, organization ID, search type, and query duration. @@ -39,7 +39,7 @@ In OpenObserve, the data ingested into a stream is typically in a flat structure **Note**: Use the **Logs** page to view the data ingested to the stream. -## Step 2: Identify the Expected Data Structure +## Step 2: Identify the expected data structure Before moving ahead, [identify what structure the chart expects](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). The heatmap chart expects flat data. @@ -51,7 +51,7 @@ In this example, each row in [data[0]](what-are-custom-charts.md#the-data-object **Note**: For charts that expect flat data, [reshaping is not needed](what-are-custom-charts.md#build-the-chart). SQL alone is enough to prepare the data in required format. -## Step 3: Prepare the Data (via SQL) +## Step 3: Prepare the data (via SQL) In the [Add Panel](what-are-custom-charts.md#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. ![custom-chart-flat-data-add-panel](../../../images/custom-chart-flat-data-add-panel.png) @@ -85,7 +85,7 @@ Select a time range to fetch the relevant dataset for your chart. ![custom-chart-flat-data-time-range-selection](../../../images/custom-chart-flat-data-time-range.png) -**Expected Query Result** +**Expected query result** ```linenums="1" data=[[ @@ -99,7 +99,7 @@ data=[[ **Note**: OpenObserve stores the result of the query in [the `data` object](what-are-custom-charts.md#the-data-object) as an **array of an array**. -## Step 4: Inspect the Queried Dataset +## Step 4: Inspect the queried dataset Inspect the queried dataset: @@ -108,7 +108,7 @@ console.log(data); console.log(data[0]); ``` -## Step 5: JavaScript Code to Render the Heatmap +## Step 5: JavaScript code to render the heatmap In the JavaScript editor, you must construct an [object named `option`](what-are-custom-charts.md#the-option-object). This `option` object defines how the chart looks and behaves. To feed data into the chart, use the query result stored in `data[0]` @@ -165,13 +165,13 @@ option = { }; ``` -## Step 6: View Result +## Step 6: View result Click **Apply** to generate the chart. ![custom-chart-flat-data-result](../../../images/custom-chart-flat-data-result.png) -### Understand the Chart +### Understand the chart In the chart, @@ -208,7 +208,7 @@ Use the following guidance to identify and fix common issues when working with c - Open your browser's developer console to locate the error. - Use `console.log()` to test your script step by step. -**4. Chart Not Rendering:** +**4. Chart not rendering:** **Cause**: The query returned data, but the chart did not render. **Fix**: diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-multiple-promql-queries.md b/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-multiple-promql-queries.md new file mode 100644 index 00000000..0d0abe2c --- /dev/null +++ b/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-multiple-promql-queries.md @@ -0,0 +1,278 @@ +This guide extends the previous page on [building custom charts using a single PromQL query](https://openobserve.ai/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-promql/). +You will now learn how to add multiple PromQL queries in the same panel, inspect their combined results, and build a chart that displays the output of several queries together. + +!!! note "Note" + This feature is useful when you need to compare related metrics across queries. Examples include comparing CPU time with CPU usage, or comparing memory used with memory cached. + +!!! note "Before you begin" + Adding more than one PromQL query does not change the structure of the PromQL result. + Instead of one entry in the data array, you will receive one entry per PromQL query. +
+ **For example**: + + - `data[0]` contains results of the first PromQL query. + - `data[1]` contains results of the second PromQL query. + - `data[2]` contains results of the third PromQL query. + + Each entry keeps the same matrix format that you saw in the single-query guide. +## How to build the custom chart for metrics using multiple PromQL queries +??? "Step 1: Add multiple PromQL queries" + ### Step 1: Add multiple PromQL queries + + 1. Open or create a dashboard. + 2. Add a panel and select **Custom Chart** mode. + 3. In the **Fields** section, set **Stream Type** to **metrics** and choose the correct metrics stream. + 4. At the bottom right corner of the editor, click **PromQL**. + 5. Enter your first PromQL query. Example: `container_cpu_time{}` + 6. Click the plus icon to add another query. + 7. Enter your second PromQL query. Example: `container_cpu_usage{}` + +??? "Step 2: View and understand the raw PromQL result" + ### Step 2: View and understand the raw PromQL result + + 1. Paste the following JavaScript in the editor. This code does not create a chart yet. It shows you how OpenObserve returns the data for each query. + ```javascript linenums="1" + console.clear(); + + console.log("=== RAW DATA ==="); + console.log(JSON.stringify(data, null, 2)); + + console.log("Query Count:", data.length); + + data.forEach((query, index) => { + console.log("--- QUERY " + index + " ---"); + console.log("resultType:", query.resultType); + console.log("Series Count:", query.result.length); + }); + + // Minimal option to avoid rendering errors + option = { + xAxis: { type: "time" }, + yAxis: { type: "value" }, + series: [] + }; + ``` + 2. Select a time range in the time range selector. + 3. Keep the JavaScript section empty for now. + 4. Open your browser developer tools and go to the Console tab. + 5. Go back to the OpenObserve UI and click **Apply** in the panel editor. + + !!! note "What you will see" + Each query appears as a separate entry in the data array. Each entry has the same structure. + ```json + { + resultType: "matrix", + result: [ + { + metric: { labels... }, + values: [ + [timestamp, value], + ... + ] + }, + ... + ] + } + ``` + + This confirms that the structure you learned in the single-query guide remains the same.The only difference is the number of top-level items in the data array. + + +??? "Step 3: Build a chart that uses multiple PromQL queries" + ### Step 3: Build a chart that uses multiple PromQL queries + + You will now extend the same logic used in the single-query guide. + The chart follows the same flow: `data → transform → series → option → chart` + The only addition is a loop that processes each PromQL query separately. + + Paste the the following code block in the JavaScript editor. + ```js linenums="1" + // Step 1: prepare the final series array + const series = []; + + // Step 2: loop through all PromQL queries + for (let queryIndex = 0; queryIndex < data.length; queryIndex++) { + const query = data[queryIndex]; + + if (!query || !Array.isArray(query.result)) { + continue; + } + + // Step 3: loop through all metric series inside this query + for (const item of query.result) { + if (!Array.isArray(item.values)) { + continue; + } + + // Step 4: convert raw datapoints into chart-friendly values + const points = item.values.map(([timestamp, value]) => [ + new Date(timestamp * 1000).toISOString(), + Number(value) + ]); + + // Step 5: generate a readable name for the legend + const label = + item.metric.k8s_pod_name || + item.metric.container_id || + "series"; + + const name = label + " Q" + (queryIndex + 1); + + // Step 6: add this dataset to the final series + series.push({ + name: name, + type: "line", + data: points, + smooth: true, + showSymbol: false + }); + } + } + + // Step 7: final chart configuration + option = { + tooltip: { trigger: "axis" }, + legend: { type: "scroll" }, + xAxis: { type: "time" }, + yAxis: { type: "value" }, + series + }; + ``` + + This creates a single line chart with one line for each metric series from each query. + Each legend item ends with a suffix such as Q1 or Q2 so that users can identify which query produced the series. + +??? "Step 4: View the result" + ### Step 4: View the result + Click **Apply** to view the result. + ![view-promql-chart-result](view-promql-chart-result.png) + Each metric appears as a separate line in the chart. + You can toggle lines using the legend. + + +??? "How to extend this code for advanced visualizations" + ### How to extend this code for advanced visualizations + The code in Step 3 already prepares a reusable `series` array: + + - Each entry in `series` represents one metric time series + - Each entry has a `name`, `type`, and `data: [time, value]` points + You can reuse this same `series` array to build more advanced charts such as dual-axis visualizations, bar charts, scatter plots, and simple heatmaps. + +
+ **Dual-axis line chart** +
+ A dual-axis chart is helpful when you want to compare two related metrics that have different scales. + For example, you may want to plot `container_cpu_time{}` on the left axis and `container_cpu_usage{}` on the right axis. + The following example assumes that: + + - You have at least two PromQL queries in the panel + - The code from Step 3 has already run and populated the `series` array + + Replace the `option = { ... }` block from Step 3 with the following code: + ```js linenums="1" + // If available, map the first two series to separate Y axes + if (series.length >= 2) { + // First query on primary axis + series[0].yAxisIndex = 0; + // Second query on secondary axis + series[1].yAxisIndex = 1; + } + // Dual axis configuration + option = { + tooltip: { trigger: "axis" }, + legend: { type: "scroll" }, + xAxis: { type: "time" }, + // Two vertical axes: one for each metric group + yAxis: [ + { type: "value", name: "Metric from Q1" }, + { type: "value", name: "Metric from Q2" } + ], + series: series + }; + ``` + The data transformation remains the same. Only the axis configuration changes and two series are assigned to different `yAxisIndex` values. + +
**Bar and scatter charts**
+ For bar or scatter charts, the structure of `series` can stay the same. + Only the `type` field needs to change. + In the Step 3 loop where the series objects are pushed, change this: + ```js + type: "line", + ``` + to one of the following: + ```js + type: "bar", + ``` + or + ```js + type: "scatter", + ``` + The rest of the code, including the `points` transformation, remains identical because the chart still expects `[time, value]` points. + +
+ **Simple heatmap example**
+ Heatmaps require a different data structure. + Instead of `[time, value]` pairs, ECharts expects each point as `[xIndex, yIndex, value]`. + The following example converts the first series from Step 3 into a one-row heatmap. This is useful when you want to show intensity over time for a single metric. + ```js linenums="1" + // Use the existing series from Step 3 + if (!Array.isArray(series) || series.length === 0) { + option = { + xAxis: { type: "category", data: [] }, + yAxis: { type: "category", data: [] }, + series: [] + }; + } else { + const baseSeries = series[0]; + // Extract time labels and values from the existing points + const timeLabels = baseSeries.data.map(([time]) => time); + // Convert into [xIndex, yIndex, value] for heatmap + const heatmapData = baseSeries.data.map(([time, value], index) => [ + index, // xIndex over time + 0, // yIndex single row + value // numeric intensity + ]); + option = { + tooltip: { position: "top" }, + xAxis: { + type: "category", + data: timeLabels, + name: "Time", + axisLabel: { show: false } + }, + yAxis: { + type: "category", + data: ["Series 1"], + name: "Metric" + }, + visualMap: { + min: Math.min(...heatmapData.map(p => p[2])), + max: Math.max(...heatmapData.map(p => p[2])), + calculable: true, + orient: "horizontal", + left: "center", + bottom: "5%" + }, + series: [ + { + name: baseSeries.name, + type: "heatmap", + data: heatmapData + } + ] + }; + } + ``` + This example: + + - Reuses the same transformed `series[0].data` created in Step 3 + - Maps each time point to an `xIndex` + - Uses a single row on the Y axis + - Uses `visualMap` to control the color scale + + You can later extend this pattern to: + + - Use one row per query or per label + - Group timestamps into buckets for coarser visualizations + The important point is that the initial transformation from PromQL remains the same. + You only reshape the transformed data to match what each chart type expects. \ No newline at end of file diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-promql.md b/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-promql.md index 4845249c..53eb9afa 100644 --- a/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-promql.md +++ b/docs/user-guide/dashboards/custom-charts/custom-charts-for-metrics-using-promql.md @@ -1,7 +1,7 @@ This guide explains how to build custom charts for metrics in OpenObserve using PromQL. The goal is to help new and advanced users understand how raw metrics data transforms into a fully rendered chart through predictable and repeatable steps. ## How metric data flows into a chart -Metrics data in OpenObserve follows a fixed transformation flow: +Custom charts use a clear and repeatable flow to convert PromQL results into a visual chart. When you build a chart with the Custom Chart panel, OpenObserve processes the data in the following order: `Metrics data in OpenObserve > PromQL query > Matrix JSON > Transform into timestamp-value pairs > Render chart` diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md b/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md index 74f18f7b..cd259d83 100644 --- a/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md +++ b/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md @@ -7,7 +7,7 @@ The following step-by-step instructions show how to build a [custom chart that e This example starts with flat data from the `default` stream, fetched and prepared using SQL, and reshaped and rendered using JavaScript. [Learn more about data preparation and reshaping for custom charts](what-are-custom-charts.md#build-the-chart). -## Use Case +## Use case Build a custom **Sunburst chart** to visualize how **query execution time** is distributed across different **organizations** and **search types**. The goal is to understand: @@ -16,7 +16,7 @@ The goal is to understand: - Whether certain usage patterns need optimization or resource allocation -## Step 1: Choose the Chart +## Step 1: Choose the chart To build a custom chart, you need to bridge two things: @@ -26,7 +26,7 @@ To build a custom chart, you need to bridge two things: > **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart. -## Step 1: Understand the Ingested Data +## Step 1: Understand the ingested data OpenObserve stores ingested data in a flat structure. **Example:** In the following dataset, each row represents a single event or query log, with its own timestamp, organization ID, search type, and query duration. @@ -46,13 +46,13 @@ OpenObserve stores ingested data in a flat structure. **Note**: To view the ingested data, go to the **Logs** page and run the query against the stream for a selected time range. -## Step 2: Identify the Expected Data Structure +## Step 2: Identify the expected data structure Before starting, [identify what structure the chart expects](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). Sunburst chart expects data to be in nested format or parent-child hierarchy. Each parent (`organization_id`) should contain its children (`search_types`), each with a value. -## Step 3: Fetch and Prepare the Data +## Step 3: Fetch and prepare the data In the [**Add Panel**](what-are-custom-charts.md#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. ![custom-charts-add-panel](../../../images/custom-chart-nested-data-add-panel.png) @@ -90,7 +90,7 @@ Select a time range to fetch the relevant dataset for your chart. ![custom-chart-select-time-range](../../../images/custom-chart-nested-time-range-selection.png) -**Expected Query Result** +**Expected query result** ```linenums="1" data = [[ @@ -124,7 +124,7 @@ data = [[ **Note**: OpenObserve stores the result of the query in the [`data` object](what-are-custom-charts.md#the-data-object) as an array of an array. -## Step 4: Inspect the Queried Data +## Step 4: Inspect the queried data Inspect the queried dataset before reshaping: @@ -139,7 +139,7 @@ This helps confirm: - If there is any inconsistencies or formatting issues - If the data requires any preparation before reshaping -## Step 5: Reshape Data Into Nested Structure +## Step 5: Reshape data into nested structure In the JavaScript editor, write a script to convert the queried result into a nested array suitable for a Sunburst chart: @@ -165,7 +165,7 @@ const treeData = Object.values(grouped); Note: In the `option` object, use the reshaped `treeData` array as the data field in your chart configuration. -## Step 6: Render the Chart +## Step 6: Render the chart Construct the [`option` object](what-are-custom-charts.md#the-option-object) in the JavaScript code to define the reshaped dataset and configure how the chart should appear. @@ -189,7 +189,7 @@ option = { **Note**: Further, you can enhance the chart with [event handlers and reusable functions](custom-charts-event-handlers-and-custom-functions.md). -## Step 7: View Results +## Step 7: View results Click **Apply** to generate the chart. @@ -198,7 +198,7 @@ If the query and JavaScript logic are correct, the chart appears in the preview ![custom-charts-result](../../../images/custom-chart-nested-data-result.png)
-### Understand the Chart +### Understand the chart The above Sunburst chart shows a breakdown of total query time across organizations and their search types. diff --git a/docs/user-guide/dashboards/custom-charts/image.png b/docs/user-guide/dashboards/custom-charts/image.png new file mode 100644 index 00000000..dc4e091f Binary files /dev/null and b/docs/user-guide/dashboards/custom-charts/image.png differ diff --git a/docs/user-guide/dashboards/custom-charts/view-promql-chart-result.png b/docs/user-guide/dashboards/custom-charts/view-promql-chart-result.png new file mode 100644 index 00000000..7aab7082 Binary files /dev/null and b/docs/user-guide/dashboards/custom-charts/view-promql-chart-result.png differ diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 25d116d1..07a69ba8 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -1,21 +1,20 @@ # User Guide -1. [Concepts](concepts.md) -2. [Log Search](logs) -3. [Metrics](metrics) -4. [Real User Monitoring (RUM)](rum.md) -5. [Organizations](identity-and-access-management/organizations.md) -6. [Users](users.md) -7. [Streams](streams.md) -8. [Ingestion](ingestion) -9. [Pipelines](pipelines) -10. [Alerts](alerts) -11. [Dashboards](dashboards) -12. [Actions](actions) -13. [Functions](functions) -14. [Migration](migration) -15. [Identity and Access Management (IAM)](identity-and-access-management) -16. [Management](management) -17. [Profile](profile) -18. [Performance](performance) -19. [Best Practices](best-practices) +1. [Log Search](logs) +2. [Metrics](metrics) +3. [Real User Monitoring (RUM)](rum.md) +4. [Organizations](identity-and-access-management/organizations.md) +5. [Users](users.md) +6. [Streams](streams.md) +7. [Ingestion](ingestion) +8. [Pipelines](pipelines) +9. [Alerts](alerts) +10. [Dashboards](dashboards) +11. [Actions](actions) +12. [Functions](functions) +13. [Migration](migration) +14. [Identity and Access Management (IAM)](identity-and-access-management) +15. [Management](management) +16. [Profile](profile) +17. [Performance](performance) +18. [Best Practices](best-practices) diff --git a/docs/user-guide/management/.pages b/docs/user-guide/management/.pages index 28307121..edaace2f 100644 --- a/docs/user-guide/management/.pages +++ b/docs/user-guide/management/.pages @@ -1,7 +1,8 @@ nav: + - General Settings: general-settings.md - Management Overview: index.md - Streaming Search: streaming-search.md - - Aggregation Cache: aggregation-cache.md + - Streaming Aggregation: aggregation-cache.md - Query Management: query-management.md - Alert Destinations: alert-destinations.md - Templates: templates.md @@ -9,4 +10,5 @@ nav: - Cipher Keys: cipher-keys.md - Nodes in OpenObserve: nodes.md - SSO Domain Restrictions: sso-domain-restrictions.md - - Sensitive Data Redaction: sensitive-data-redaction.md \ No newline at end of file + - Sensitive Data Redaction: sensitive-data-redaction.md + - License Management: lincense.md diff --git a/docs/user-guide/management/general-settings.md b/docs/user-guide/management/general-settings.md new file mode 100644 index 00000000..47473411 --- /dev/null +++ b/docs/user-guide/management/general-settings.md @@ -0,0 +1,55 @@ +## General settings +This page explains how to use the settings available under General Settings in OpenObserve. These settings allow you to configure scrape interval, theme colors, and Enterprise branding. + +## What you can do in general settings +General settings provides the following configuration options: + +- Scrape interval +- Theme configuration for light and dark modes +- Enterprise branding features such as custom logo text and custom logo + +!!! note "Note" + `_meta` organizations can access all settings. Regular organizations see only the options allowed for their specific license and permissions. + +### Scrape interval +> This setting is available for all organizations. + +![theme-config](../../images/theme-config.png) +The scrape interval defines how often the monitoring system collects metrics. Enter the number of seconds and select Save. + +### Theme configuration +Theme configuration allows you to manage the default colors for light mode and dark mode for your organization. When you save these values, they are applied across the UI for all users in the organization. +![theme-config](../../images/theme-config.png) + +You can set the following fields: + +- Light mode color +- Dark mode color + +Select **Save** to apply the changes. + +### Theme configuration using environment variables +You can also configure the default theme colors at the system level. These values become the initial defaults for all organizations. + +- `ZO_DEFAULT_THEME_LIGHT_MODE_COLOR`: Sets the default color used in light mode. Default:"#3F7994" +- `ZO_DEFAULT_THEME_DARK_MODE_COLOR`: Sets the default color used in dark mode. Default: "#5B9FBE" + +If these variables are set, the UI loads these colors automatically unless overridden by an organization through the General Settings page. + +### Enterprise branding +Enterprise branding options appear only in the `_meta` org when Enterprise features are enabled. +![enterprise-branding](../../images/enterprise-branding.png) +
+ +**Custom logo text** +Use this field to replace the default OpenObserve text shown in the header. Enter the required text and save the changes. +
+ +**Custom logo** +Upload a custom branding logo to replace the default OpenObserve logo in the header. + +- Supported file formats: PNG, JPG, JPEG, GIF, BMP +- Maximum file size: 20 KB +- Maximum dimensions: 150 by 30 pixels + +After uploading the file, select the save icon to apply the logo. \ No newline at end of file diff --git a/docs/user-guide/management/lincense.md b/docs/user-guide/management/lincense.md new file mode 100644 index 00000000..d2053bdf --- /dev/null +++ b/docs/user-guide/management/lincense.md @@ -0,0 +1,60 @@ +This page explains how OpenObserve Enterprise licenses work and how users can view, request, add, and update licenses. The guide is role based so that each user clearly understands what they can do and how to do it. + +## Overview +The license defines the daily ingestion capacity for your OpenObserve installation. It ensures that the installation operates within the allowed limits and that enterprise features remain active. The system monitors ingestion volume throughout the day and blocks ingestion and search when the daily limit is crossed. + +!!! note "Who can access" + + OpenObserve provides access to license information based on the following roles. Your role decides what you can see and what actions you can perform. + + | **Role** | **What you can do** | **What you cannot do** | + | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | + | **If you have access to an organization** | - View license information in the About page of the organization.
- Check daily ingestion limits.
- Check current usage. | - Use the Manage License button in the About page of the organization. Selecting it shows a `403` unauthorized message.
- Access the License page under Management.
- Add or update a license key. | + | **If you have access to the meta organization but do not have the update permission for the License module under IAM > Roles > Permission** | - Open the License page under Management in the meta org and view detailed license information.
- Request a new license.
- Use the Manage License button in the About page of any organization. | - Add or update a license key in the License page under Management. | + | **If you have access to the meta organization and have create and update permission for the License module under IAM > Roles > Permission** | - Add a new license key in the License page under Management.
- Update an existing license key. | - | + + +## How to manage licenses +To perform the following tasks, you must have appropriate permission. Review the **Who can access** section to know more. + +??? "View license information" + ### View license information + ![about-page](../../images/about-page-license.png) + + 1. Click the Help (?) icon in the top-right corner. + 2. Click **About**. + 2. In the **Enterprise License Details** section, review license status, ingestion limit, and daily usage. + +??? "Request a new license" + ### Request a new license + + 1. In the **Enterprise License Details** section of the **About** page, click **Manage License**. + ![about-page](../../images/about-page-license.png) + 2. It takes you to the License page under **Management** in the `_meta` organization. + ![license-management](../../images/license-management.png) + 3. Click **Request New License**. + 4. Complete the form by following the guidelines mentioned below and click **Submit Request**. + + !!! note "Guideline for completing the license request form" + Use the information below to complete each field accurately when requesting a new license through the License Server page. + ![license-server](../../images/license-server.png) + + - **Installation ID**: The Installation ID field is usually prefilled by the system. If it is not prefilled, copy it from the License page under Management in the meta organization. + - **Contact Name and Email Address**: Enter the name and email address of the primary contact person for your organization. This person will receive the license key and future communication related to license renewal or verification. + - **Address**: Provide the official address of your organization. This must be a valid, complete postal address because it is used for verification and record keeping. + - **Use case**: Describe how you plan to use OpenObserve. Provide clear details about your expected data volume or operational requirements. + - **Ingestion limit**: Select a limit based on the actual ingestion volume of your installation. To determine the correct value, review the usage stream in OpenObserve to identify your peak daily ingestion. Choose a limit that is higher than your highest observed ingestion volume to avoid data loss or ingestion failures. + - **Additional email addresses**: List additional recipients who should receive notifications related to license issuance or updates. + - **Base URLs**: Enter the base URLs used by your OpenObserve installation. The License Server accepts multiple base URLs because some installations may operate from more than one domain. Add each URL that users use to access your OpenObserve instance. + + +??? "Add a new license key" + ### Add a new license key + ![add-new-license-key](../../images/add-new-license-key.png) + + 1. In the `_meta` organization, navigate to **Management** from the top-right corner. + 2. Select **License**. + 3. Select **Add New License Key**. + 4. Paste the license key in the **Update License Key** field. + 5. Select **Update License**. + diff --git a/docs/user-guide/pipelines/.pages b/docs/user-guide/pipelines/.pages index 46684ef9..05e0f3b9 100644 --- a/docs/user-guide/pipelines/.pages +++ b/docs/user-guide/pipelines/.pages @@ -1,7 +1,8 @@ nav: - Pipelines Overview: index.md - Pipelines in OpenObserve: pipelines.md - - Create and Use Pipelines: use-pipelines.md + - Create and Use Real-time Pipeline: create-and-use-real-time-pipeline.md + - Create and Use Scheduled Pipeline: create-and-use-scheduled-pipeline.md - Remote Destination: remote-destination.md - Import and Export Pipelines: import-and-export-pipelines.md - Manage Pipelines: manage-pipelines.md diff --git a/docs/user-guide/pipelines/configurable-delay-in-scheduled-pipelines.md b/docs/user-guide/pipelines/configurable-delay-in-scheduled-pipelines.md index 38746db6..73ded4d5 100644 --- a/docs/user-guide/pipelines/configurable-delay-in-scheduled-pipelines.md +++ b/docs/user-guide/pipelines/configurable-delay-in-scheduled-pipelines.md @@ -3,41 +3,40 @@ description: >- Delay scheduled pipeline execution in OpenObserve to ensure data completeness by waiting minutes after the trigger time before processing data. --- - -This guide explains how to use the configurable delay feature for [scheduled pipelines](../use-pipelines/). +This guide explains how to use the configurable delay feature for [scheduled pipelines](../pipelines/create-and-use-scheduled-pipeline/). This feature allows you to delay the execution of a scheduled pipeline by a set number of minutes after its scheduled start time. This is useful to ensure data completeness and accommodate late-arriving events before the pipeline runs. -## How to Configure Delay +## How to configure delay When editing the **Associate Query** page: -1. In the **Set Variables** section, configure: - - - **Frequency**: How often the pipeline should run. - - **Period**: The data window the pipeline should process. - - **Delay**: How many minutes to wait after the scheduled time before running the pipeline. +1. In the **Set Variables** section, add **Delay**. Delay defines how many minutes the system waits after the scheduled trigger time. The pipeline runs only after this waiting period completes. ![configurable-delay](../../images/scheduled-pipeline-config-delay.png) 2. Select **Validate and Close**. -**Note**: Since Delay is a required field when setting up a scheduled pipeline, enter 0 minutes if you want the pipeline to run exactly at the scheduled time without any delay. +!!! note "Note" + Since Delay is a required field when setting up a scheduled pipeline, enter 0 minutes if you want the pipeline to run exactly at the scheduled time without any delay. -**Example** +!!! note "Example" -If the pipeline is configured with the following settings: + If the pipeline is configured with the following settings: -- Frequency: 5 minutes -- Period: 5 minutes -- Delay: 2 minutes + - Frequency: 5 minutes + - Period: 5 minutes + - Delay: 2 minutes -**Behavior:** The pipeline is scheduled every 5 minutes and runs 2 minutes after each scheduled time, processing data from the previous 5-minute window. -At 10:00 AM (scheduled time), the pipeline executes at 10:02 AM, processing data from 9:55 AM to 10:00 AM. + **Behavior:** The pipeline is scheduled every 5 minutes and runs 2 minutes after each scheduled time, processing data from the previous 5-minute window. + At 10:00 AM (scheduled time), the pipeline executes at 10:02 AM, processing data from 9:55 AM to 10:00 AM. -**Note** + **Note** -- **Works with Cron**: Delay also applies when using cron-based schedules. -- **Data Window Unchanged**: Delay only affects when execution starts. It does not change the data range. + - **Works with Cron**: Delay also applies when using cron-based schedules. + - **Data Window Unchanged**: Delay only affects when execution starts. It does not change the data range. -**Related Links**
-For steps on creating scheduled pipelines, see [Scheduled Pipelines](../use-pipelines/). +## Related links +- [Pipelines in OpenObserve](../pipelines/pipelines/) +- [Create and Use Real-time Pipeline](../pipelines/create-and-use-real-time-pipeline/) +- [Create and Use Scheduled Pipeline](../pipelines/create-and-use-scheduled-pipeline/) +- [Manage Pipelines](../pipelines/manage-pipelines/) diff --git a/docs/user-guide/pipelines/create-and-use-real-time-pipeline.md b/docs/user-guide/pipelines/create-and-use-real-time-pipeline.md new file mode 100644 index 00000000..24025f3b --- /dev/null +++ b/docs/user-guide/pipelines/create-and-use-real-time-pipeline.md @@ -0,0 +1,159 @@ +Use this guide to create and run real-time pipelines. Real-time pipelines process data immediately after it is ingested into a source stream. They allow you to apply conditions and functions and then route the processed data to one or more destinations. + +> Visit the [Pipelines in OpenObserve](https://openobserve.ai/docs/user-guide/pipelines/pipelines/) page before starting. It explains pipeline concepts, types, and components used in this guide. + +## Prerequisites +- Your OpenObserve Cloud or self-hosted instance is running. +- You have permissions to work with pipelines. Access is controlled from IAM > Roles > Permissions. + +## Steps to create a real-time pipeline + +??? "Step 1: Open the pipeline editor" + ### Step 1: Open the pipeline editor + 1. Log in to OpenObserve. + 2. In the left navigation panel, select **Pipelines**. By default, the system opens the **Pipelines** tab. + ![Pipelines tab](../../images/pipelines-tab.png) + 3. Click **Add Pipeline** at the top-right corner. + ![Open the pipeline editor](../../images/pipeline-editor.png) +??? "Step 2: Enter a pipeline name" + ### Step 2: Enter a pipeline name + Provide a unique name for the pipeline. + ![Pipeline name](../../images/pipeline-name.png) + +??? "Step 3: Configure the Source node" + ### Step 3: Configure the Source node + + 1. From the **Source** section, drag the **Stream** node into the editor. + 2. Edit the source node: + + - Select the **Stream Type**. + - Choose one of the following options: + + - **Option A**: Select an existing stream. Choose a stream from the **Stream Name** dropdown. + ![Select an existing stream](../../images/select-existing-stream.png) + - **Option B:** Create a new stream. Enable the **Create new Stream** toggle and create a new stream if necessary. + ![alt text](../../images/create-new-stream.png) + 3. Click **Save** to confirm the source node. + + !!! note "Important behavior" + When you select a source stream for a real-time pipeline, OpenObserve automatically adds a destination that points back to the same stream. This preserves the original data. + If you remove this default destination and add only filtered routes, events that do not match the filters will be dropped. Add a destination pointing to the source stream if you want to retain all data. + +??? "Step 4: Configure the Transform node" + ### Step 4: Configure the Transform node + + 1. From the **Transform** section, drag either a **Condition** or **Function** node into the editor. + 2. If you add a Condition node: + ![condition-node](../../images/condition-node.png) + + 1. In the **Associate Conditions** screen, select the field name in the **Column** drop-down list. This list displays the fields detected in the source stream. + 2. Select an operator from the **Operator** drop-down list. + 3. Enter a value in the **Value** input box. + 4. Add more conditions if needed. + + !!! note "Important" + **Condition nodes require a stream schema.**
+ If you create a new stream in **Step 3** and do not define any fields, the **Condition** editor does not display field names. + To use conditions, add at least one field while creating the source stream or ingest sample data so that the schema is inferred. + !!! note "Guidelines" + - Use an empty string to check for empty values. For example, `app_name != ""` + - Use null to check for null values. For example, `app_name != null` + - If the condition does not match, the record is dropped. + - If the record does not contain the specified field, it is also dropped. + + 3. If you add a Function node: + + Use a Function node to transform events using a VRL function. + > A Function does not require predefined fields. You can use it even if the source stream has no schema.
+ !!! note "To create a new function:" + ![create-new-function](../../images/create-new-function.png) + + 1. Enable **Create new function** toggle. + 2. In the **Associate Function** tab, enter the function name. + 3. Open the **Query** tab. + 4. Select the stream type, stream name, and duration for which you want to query the data. + 5. Click the **Run Query** option at the top-right corner of the **Query** tab. + 6. In the **Function** tab, write a VRL function. + 7. Click the **Test Function** button at the top-right corner of the screen. + 8. Click **Save** to save the function. + + 4. The **After Flattening** toggle is enabled by default. It ensures the function processes normalized data. Disable this toggle only if you need the original structure. + + !!! note "Guidelines" + - **RBF (Run Before Flattening)**: Function executes before data structure is flattened. + - **RAF (Run After Flattening)**: Function executes after data structure is flattened. + ![After flattening](../../images/after-flattening.png) + + 5. Click **Save** to confirm the transform node. + +??? "Step 5: Configure the Destination node" + ### Step 5: Configure the Destination node + A destination defines where the processed events are written. You can forward data to another stream or to an external system. + + **To add a stream destination:** + ![external-destination](../../images/external-destination.png) + + 1. Drag a **Stream** node into the editor. + 2. In the **Associate Stream** panel, configure the destination stream. + 3. Select an existing stream or create a new one. Stream creation follows the same configuration as shown in Step 3. + + !!! note "Note" + - For real-time pipelines, you can select **logs**, **metrics**, or **traces** as the destination stream type. + - **Enrichment_tables** as a destination stream is only available for scheduled pipelines. It cannot be used as a destination in real-time pipelines. + + !!! note "Dynamic stream names" + You can configure the stream name dynamically with curly braces. + Example: `static_text_{fieldname}_postfix` + Static text before and after the braces is optional. + + !!! warning "Default destination and data loss" + When you select a source stream for a real-time pipeline, OpenObserve automatically adds a destination that points back to the same stream. This default destination ensures that all ingested events continue to be stored in the source stream. + If you remove this default destination and only keep filtered routes, events that do not match any condition are dropped and are not written to the source stream. + Add at least one catch-all route back to the original stream if you want to preserve all events. + + + **To add an external destination:** + ![stream-destination](../../images/stream-destination.png) + + 1. Drag a **Remote** node into the editor. + 2. In the **External Destination** panel, either select an existing destination or enable **Create new Destination** and fill in the required details such as name, URL, method, output format, and headers. + 4. Click **Save**. + + > For detailed steps on configuring external destinations, see the [Remote Destination](https://openobserve.ai/docs/user-guide/pipelines/remote-destination/) guide. + + +??? "Step 6: Connect the nodes" + ### Step 6: Connect the nodes + Use the connection icon to link Source > Transform > Destination. + To remove a link, select the link and press delete on your keyboard. + ![connect-nodes](../../images/connect-nodes.png) + +??? "Step 7: Save the pipeline" + ### Step 7: Save the pipeline + Click **Save**. The pipeline becomes active immediately. + ![Save pipeline](../../images/save-pipeline.png) + + +## Use the real-time pipeline + +??? "Step 1: Ingest data" + ### Step 1: Ingest data + Use curl or any supported ingestion method. + To find the sample curl command, go to **Data Sources** > **Custom** > **Logs** > **Curl**. + Each new event ingested into the source stream is processed instantly. + +??? "Step 2: Verify output" + ### Step 2: Verify output + ![view-result](../../images/view-result.png) + + 1. Go to **Logs**. + 2. From the stream selection dropdown, select the stream you added as destination. + 3. Select the time range from the time range selector. + 3. Click **Run query** to view the result. + +## Related links +- [Pipelines in OpenObserve](../pipelines/pipelines/) +- [Create and Use Scheduled Pipeline](../pipelines/create-and-use-scheduled-pipeline/) +- [Import and Export Pipelines](../pipelines/import-and-export-pipelines/) +- [Manage Pipelines](../pipelines/manage-pipelines/) +- [Configurable Delay in Scheduled Pipelines](../pipelines/configurable-delay-in-scheduled-pipelines/) \ No newline at end of file diff --git a/docs/user-guide/pipelines/create-and-use-scheduled-pipeline.md b/docs/user-guide/pipelines/create-and-use-scheduled-pipeline.md new file mode 100644 index 00000000..9d2a1cf0 --- /dev/null +++ b/docs/user-guide/pipelines/create-and-use-scheduled-pipeline.md @@ -0,0 +1,154 @@ +Use this guide to create and run scheduled pipelines. Scheduled pipelines run at a fixed interval. They query existing data from a stream using SQL or PromQL, apply optional transformations, and write the results to a destination stream or external system. + +> Visit the [Pipelines overview](https://openobserve.ai/docs/user-guide/pipelines/pipelines/) page before starting. + +## Prerequisites +- Your OpenObserve Cloud or self-hosted instance is running. +- You have permissions to work with pipelines. Access is controlled from IAM > Roles > Permissions. + +## Steps to create a scheduled pipeline + +??? "Step 1: Open the pipeline editor" + ### Step 1: Open the pipeline editor + 1. Log in to OpenObserve. + 2. In the left navigation panel, select **Pipelines**. By default, the system opens the **Pipelines** tab. + ![Pipelines tab](../../images/pipelines-tab.png) + 3. Click **Add Pipeline** at the top-right corner. + ![Open the pipeline editor](../../images/pipeline-editor.png) +??? "Step 2: Enter a pipeline name" + ### Step 2: Enter a pipeline name + Enter a unique name in the input box at the top of the editor. + ![scheduled-pipeline-name](../../images/scheduled-pipeline-name.png) +??? "Step 3: Configure the Source node (Query)" + ### Step 3: Configure the Source node (Query) + Scheduled pipelines do not use a **Stream** node as the source. They always begin with a **Query** node. + + 1. From the **Source** section, drag the **Query** node into the editor. + 2. The **Associate Query** panel opens. The panel contains the following sections, **Build Query**, **SQL Query** editor, **Output**, **Set Variables**. + ![Associate Query](../../images/associate-query.png) + 3. In the **Build Query** section: + + 1. Select the **Stream Type**. + 2. Select the **Stream Name**. + 4. Edit or write your SQL query in the **SQL query** editor. + 5. Select the time range from the time range selector at the top-right corner. + 5. Click **Run query** to preview data in the **Output** panel. + ![output-preview](../../images/query-output.png) + 6. In the **Set Variables** section configures how the pipeline runs using the following settings: + + - **Enable Cron**: Switch from interval-based scheduling to cron-based scheduling. + - **Frequency**: How often the pipeline runs. + - **Period**: The data window to process. Period must equal Frequency. + - **Delay**: How many minutes the pipeline waits after the scheduled time before execution. + + > See the [Pipelines overview](https://openobserve.ai/docs/user-guide/pipelines/pipelines/) page to learn more about frequency, period, and cron. + > See the [Configurable Delay in Scheduled Pipelines](https://openobserve.ai/docs/user-guide/pipelines/configurable-delay-in-scheduled-pipelines/) page to learn how to configure delay in scheduled-pipelines. + + ![variables](../../images/scheduled-variables.png) + + 7. After configuring the query and variables, click **Validate and Close**. This adds the Query node to the editor. + +??? "Step 4: Configure the Transform node" + ### Step 4: Configure the Transform node + + 1. From the **Transform** section, drag either a **Condition** or **Function** node into the editor. + 2. If you add a Condition node: + ![condition-node](../../images/schedule-condition-node.png) + + 1. In the **Associate Conditions** screen, select the field name in the **Column** drop-down list. This list displays the fields detected in the source stream. + 2. Select an operator from the **Operator** drop-down list. + 3. Enter a value in the **Value** input box. + 4. Add more conditions if needed. + + !!! note "Guidelines" + - Use an empty string to check for empty values. For example, `app_name != ""` + - Use null to check for null values. For example, `app_name != null` + - If the condition does not match, the record is dropped. + - If the record does not contain the specified field, it is also dropped. + + 3. If you add a Function node: + + Use a Function node to transform events using a VRL function. + > A Function does not require predefined fields. You can use it even if the source stream has no schema.
+ !!! note "To create a new function:" + + 1. Enable **Create new function** toggle. + 2. In the **Associate Function** tab, enter the function name. + 3. Open the **Query** tab. + 4. Select the stream type, stream name, and duration for which you want to query the data. + 5. Click the **Run Query** option at the top-right corner of the **Query** tab. + 6. In the **Function** tab, write a VRL function. + 7. Click the **Test Function** button at the top-right corner of the screen. + 8. Click **Save** to save the function. + + 4. The **After Flattening** toggle is enabled by default. It ensures the function processes normalized data. Disable this toggle only if you need the original structure. + + !!! note "Guidelines" + - **RBF (Run Before Flattening)**: Function executes before data structure is flattened. + - **RAF (Run After Flattening)**: Function executes after data structure is flattened. + + 5. Click **Save** to confirm the transform node. + +??? "Step 5: Configure the Destination node" + ### Step 5: Configure the Destination node + A destination defines where the processed events are written. You can forward data to another stream or to an external system. + + **To add a stream destination:** + ![stream-destination](../../images/schedule-stream-destination.png) + + 1. Drag a **Stream** node into the editor. + 2. In the **Associate Stream** panel, configure the destination stream. + 3. Select an existing stream or create a new one. Stream creation follows the same configuration as shown in Step 3. + + !!! note "Note" + You can add **Enrichment_tables** as a destination stream for scheduled pipelines. + + !!! note "Dynamic stream names" + You can configure the stream name dynamically with curly braces. + Example: `static_text_{fieldname}_postfix` + Static text before and after the braces is optional. + + + **To add an external destination:** + + 1. Drag a **Remote** node into the editor. + 2. In the **External Destination** panel, either select an existing destination or enable **Create new Destination** and fill in the required details such as name, URL, method, output format, and headers. + 4. Click **Save**. + + > For detailed steps on configuring external destinations, see the [Remote Destination](https://openobserve.ai/docs/user-guide/pipelines/remote-destination/) guide. + + +??? "Step 6: Connect the nodes" + ### Step 6: Connect the nodes + Use the connection icon to link Source > Transform > Destination. + To remove a link, select the link and press delete on your keyboard. + ![connect-nodes](../../images/schedule-connect-nodes.png) + +??? "Step 7: Save the pipeline" + ### Step 7: Save the pipeline + Click **Save**. The pipeline becomes active immediately. + ![scheduled-pipeline-list-view](../../images/scheduled-pipeline-list-view.png) + +## Use the scheduled pipeline + +??? "Step 1: Ingest data" + ### Step 1: Ingest data + Use curl or any supported ingestion method. + To find the sample curl command, go to **Data Sources** > **Custom** > **Logs** > **Curl**. + The scheduled pipeline does not run immediately when data is ingested. + It processes data only when the schedule triggers, using the configured frequency, period, and delay. +??? "Step 2: Verify output" + ### Step 2: Verify output + ![view-result](../../images/view-scheduled-result.png) + + 1. Go to **Logs**. + 2. From the stream selection dropdown, select the stream you added as destination. + 3. Select the time range from the time range selector. + 3. Click **Run query** to view the result. + +## Related links +- [Pipelines in OpenObserve](../pipelines/pipelines/) +- [Create and Use Real-time Pipeline](../pipelines/create-and-use-real-time-pipeline/) +- [Import and Export Pipelines](../pipelines/import-and-export-pipelines/) +- [Manage Pipelines](../pipelines/manage-pipelines/) +- [Configurable Delay in Scheduled Pipelines](../pipelines/configurable-delay-in-scheduled-pipelines/) diff --git a/docs/user-guide/pipelines/import-and-export-pipelines.md b/docs/user-guide/pipelines/import-and-export-pipelines.md index 72e46f8b..fb0a8049 100644 --- a/docs/user-guide/pipelines/import-and-export-pipelines.md +++ b/docs/user-guide/pipelines/import-and-export-pipelines.md @@ -6,49 +6,36 @@ description: >- This guide explains how to import and export pipelines. Use this feature to replicate existing pipelines across environments without recreating them from scratch. -## Export a Pipeline +## Export a pipeline To export an existing pipeline configuration: 1. From the left navigation menu, go to **Pipelines**. 2. In the **Pipelines** tab, locate the pipeline you want to export. -3. In the **Actions** column of the pipeline you want to export, click the download icon. - +3. In the **Actions** column of the pipeline, click the download icon. +![download-pipeline](../../images/download-pipeline-json.png) The downloaded `.json` file contains the pipeline configuration and can be used to import the pipeline into another environment.
-![export pipeline](../../images/pipeline-import-export-1.png) - -For example, exporting the pipeline `encrypt_test_pipeline` generates a JSON file similar to the following: -```json -{ - "name": "encrypt_test_pipeline", - "stream_name": "k8s_events", - "stream_type": "logs", - "type": "realtime", - "nodes": [...], - "edges": [...], - "enabled": false -} -``` - -## Import a Pipeline +Exporting the pipeline `scheduled-pipeline-demo` generates a JSON file similar to the following: +![json-file-pipeline-export](../../images/json-file-pipeline-export.png) +## Import a pipeline To import a previously exported pipeline: +![alt text](../../images/import-pipelines.png) 1. Go to **Pipelines**. 2. Click **Import Pipeline** in the top-right corner.
- ![import pipeline](../../images/pipeline-import-export-2.jpg) 3. Choose one of the supported import methods: - **Upload JSON Files**: Select one or more JSON files containing pipeline configurations from your local system. - **Enter URL**: Provide a URL to fetch the pipeline configuration. - **Paste JSON Object**: Copy and paste the JSON pipeline definition in the JSON editor. -![import methods](../../images/pipeline-import-export-3.png)
-To import pipelines in bulk, choose multiple JSON files, as shown below:
-![bulk import](../../images/pipeline-import-export-4.png) -4. Click **Import**. +4. To import pipelines in bulk, choose multiple JSON files. +5. Click **Import**. +The imported pipeline appears in the **Pipelines** tab. -## Handle Validation Errors + +## Handle validation errors while importing pipelines If any validation errors occur during import, refer to the following resolutions: @@ -62,9 +49,10 @@ If any validation errors occur during import, refer to the following resolutions - **Function name does not exist:** Enter the function name - **Remote Destination does not exist:** Ensure the remote destination is accurate. -![validation error](../../images/pipeline-import-export-5.png) - -The imported pipeline appears in the **Pipelines** tab. -## Related Links -- [Manage Pipelines](../manage-pipelines/) +## Related links +- [Pipelines in OpenObserve](../pipelines/pipelines/) +- [Create and Use Real-time Pipeline](../pipelines/create-and-use-real-time-pipeline/) +- [Create and Use Scheduled Pipeline](../pipelines/create-and-use-scheduled-pipeline/) +- [Manage Pipelines](../pipelines/manage-pipelines/) +- [Configurable Delay in Scheduled Pipelines](../pipelines/configurable-delay-in-scheduled-pipelines/) diff --git a/docs/user-guide/pipelines/index.md b/docs/user-guide/pipelines/index.md index 9b9c4cec..23b67736 100644 --- a/docs/user-guide/pipelines/index.md +++ b/docs/user-guide/pipelines/index.md @@ -3,7 +3,8 @@ In OpenObserve, pipelines are data processing workflows used to transform logs, **Learn more:** - [Pipelines in OpenObserve](../pipelines/pipelines/) -- [Create and Use Pipelines](../pipelines/use-pipelines/) +- [Create and Use Real-time Pipeline](../pipelines/create-and-use-real-time-pipeline/) +- [Create and Use Scheduled Pipeline](../pipelines/create-and-use-scheduled-pipeline/) - [Import and Export Pipelines](../pipelines/import-and-export-pipelines/) - [Manage Pipelines](../pipelines/manage-pipelines/) - [Configurable Delay in Scheduled Pipelines](../pipelines/configurable-delay-in-scheduled-pipelines/) \ No newline at end of file diff --git a/docs/user-guide/pipelines/manage-pipelines.md b/docs/user-guide/pipelines/manage-pipelines.md index 0a75c3a3..51a78b81 100644 --- a/docs/user-guide/pipelines/manage-pipelines.md +++ b/docs/user-guide/pipelines/manage-pipelines.md @@ -3,56 +3,69 @@ description: >- Manage, filter, search, edit, pause, or delete real-time and scheduled pipelines in OpenObserve using the intuitive pipeline list and actions panel. --- -This page describes the features designed to help you manage your pipelines in OpenObserve. +This page describes how to manage real-time and scheduled pipelines in OpenObserve. -![pipeline management](../../images/pipeline12_manage_pipelines.png) +## Pipeline management options +You can filter, search, view, edit, pause, resume, or delete pipelines from a single interface. +![pipeline management](../../images/manage_pipelines.png) +??? "Filter pipelines" + ## Filter pipelines + Use the filter options **All**, **Real-Time**, and **Scheduled** to view only the type of pipelines you want to work with. + ![filter-pipeline](../../images/filter-pipeline.png) -## Filter Options +??? "Search pipelines" + ## Search pipelines + Use the search bar to locate pipelines by name or attributes. + ![search-pipeline](../../images/search-pipeline.png) -Toggle between **All**, **Real-Time**, and **Scheduled** pipelines to focus on the type of pipelines you want to view. +??? "Pipeline list view" + ## Pipeline list view + The pipeline list displays key information for each pipeline. + ![pipeline-list-view](../../images/pipeline-list-view.png) + - **Pipeline Name**: Identifies the pipeline. + - **Type**: Indicates whether the pipeline is real-time or scheduled. + - **Stream Name**: Shows the associated source stream. + - **Stream Type**: Indicates the type of data in the stream such as logs, metrics, traces, or enrichment tables. + - **Frequency**: Displays how often the scheduled pipeline runs. + - **Period**: Shows the time duration used for each execution window of a scheduled pipeline. + - **Cron**: Indicates whether the pipeline is configured using a cron expression. -## Search Pipelines +??? "Actions tab" + ## Actions tab + Each pipeline row includes the following actions. + ![action-tab](../../images/action-tab.png) -Use the search bar to locate pipelines by name or attributes. + - Start or pause pipeline + - Edit pipeline + - Export pipeline configuration + - Delete pipeline + - View pipeline details + ### Pause and resume pipelines -## Pipeline List View + You can pause both real-time and scheduled pipelines.
-View all pipelines in a tabular format, including: - -- **Pipeline Name**: Identify the pipeline. -- **Type**: Displays whether it is real-time or scheduled. -- **Stream Name**: Indicates the associated source stream. -- **Stream Type**: Specifies the type of data in the stream- logs, metrics, traces, or enrichment tables. -- **Frequency**: Shows how often the scheduled pipeline executes. -- **Period**: Indicates the interval duration for scheduled pipeline execution. -- **Silence**: Shows the configured silence period for scheduled pipelines. -- **Cron**: Specifies whether the scheduled pipeline uses a cron schedule (True/False). -- **SQL Query**: Displays the SQL query used in the scheduled pipeline configuration. - - -## Actions Tab - -- **Edit Pipeline**: Modify the configuration of an existing pipeline. -- **Delete Pipeline**: Remove a pipeline permanently from your system. -- **Pause/Start Pipelines**: Temporarily stop or restart pipelines as needed. - -!!! Info "Pause and Resume a Scheduled Pipeline" - **Pausing a Scheduled Pipeline:** + **Pausing a pipeline:** - When paused, the pipeline stops executing on its scheduled intervals. - The system preserves the exact timestamp when the pause occurred. - Pipeline configuration and state are maintained during the pause. - **Unpausing a Scheduled Pipeline:** + **Resuming a real-time pipeline**: + + - Real-time pipelines resume instantly with one click. + - No dialog or additional options are shown. + - Processing continues from the current moment. + + + **Resuming a scheduled pipeline**: - When resuming a paused scheduled pipeline, OpenObserve presents a **Resume Pipeline Ingestion** dialog with two options: -
+ Scheduled pipelines display a Resume Pipeline Ingestion dialog with two options. ![Resume pipeline](../../images/resume-scheduled-pipeline.png) - - **Continue from where it paused:** + - **Continue from where it paused:** - Processes all data from the pause timestamp to the current time. - Maintains complete data continuity with no gaps. @@ -60,11 +73,40 @@ View all pipelines in a tabular format, including: > **Note:** Use the **Continue from where it paused** option, when data completeness is critical. - - **Start from now:** + - **Start from now:** - Begins processing from the current timestamp. - Creates a data gap between the pause and resume timestamps. - Provides immediate resumption with minimal resource usage. > **Note:** Use the **Start from now** option, when data gaps are acceptable. - + + ### Pipeline error + If a pipeline encounters an error, an error icon appears in the **Actions** column. + ![pipeline-error](../../images/pipeline-error.png) + Selecting this icon opens a dialog that displays the full error summary. + ![pipeline-error-view](../../images/pipeline-error-view.png) + +??? "Pipeline history" + ## Pipeline history + You can view past executions, errors, and performance statistics in the Pipeline History page. + + The [Pipeline History](https://openobserve.ai/docs/user-guide/pipelines/pipeline-history/) page provides a complete timeline of runs with detailed execution information. + +??? "Import and export pipelines" + ## Import and export pipelines + You can export any pipeline and import pipeline configurations through the Import and **Export** options. For more details, see [Import and export pipelines](https://openobserve.ai/docs/user-guide/pipelines/import-and-export-pipelines/). + +??? "Sort columns" + ## Sort columns + ![sort-columns](../../images/sort-columns.png) + + You can sort the Pipelines table by selecting any column header. The table supports ascending, descending, and alphabetical sorting depending on the data in that column. + + +## Related links +- [Pipelines in OpenObserve](../pipelines/pipelines/) +- [Create and Use Real-time Pipeline](../pipelines/create-and-use-real-time-pipeline/) +- [Create and Use Scheduled Pipeline](../pipelines/create-and-use-scheduled-pipeline/) +- [Import and Export Pipelines](../pipelines/import-and-export-pipelines/) +- [Configurable Delay in Scheduled Pipelines](../pipelines/configurable-delay-in-scheduled-pipelines/) \ No newline at end of file diff --git a/docs/user-guide/pipelines/pipeline-history.md b/docs/user-guide/pipelines/pipeline-history.md index 7160bfe1..e3d79dfc 100644 --- a/docs/user-guide/pipelines/pipeline-history.md +++ b/docs/user-guide/pipelines/pipeline-history.md @@ -1,14 +1,13 @@ This guide provides information about how the Pipeline History feature in OpenObserve works, where the data originates from, who can access it, and how to interpret the pipeline execution records. +> For information on other pipeline management options, see the [Manage Pipelines](../pipelines/manage-pipelines/) page. ## Overview Pipeline History provides visibility into every pipeline run, including its execution time, status, and duration. Each record represents one instance of a scheduled or manually triggered pipeline execution. -!!! "Who can access" - +!!! note "Who can access" Any user who has permission to view, update, or delete pipelines can also view pipeline history. This ensures that users responsible for managing or maintaining pipelines can monitor their performance and investigate failures without requiring elevated access. - ## How to interpret the Pipeline History table ![pipeline-history](../../images/pipeline-history.png) The table lists each pipeline run with key execution details. @@ -31,7 +30,7 @@ The table lists each pipeline run with key execution details. ![actions-pipeline-history](../../images/actions-pipeline-history.png) - - **Query Time**: Time taken by the SQL query within the pipeline to execute. This helps measure query performance. - - **Source Node**: Node responsible for executing the run. This helps identify where the execution occurred for debugging and performance monitoring.
- Example:
- **Source Node**: `o2-openobserve-alertmanager-0` \ No newline at end of file +- **Query Time**: Time taken by the SQL query within the pipeline to execute. This helps measure query performance. +- **Source Node**: Node responsible for executing the run. This helps identify where the execution occurred for debugging and performance monitoring.
+Example:
+**Source Node**: `o2-openobserve-alertmanager-0` \ No newline at end of file diff --git a/docs/user-guide/pipelines/pipelines.md b/docs/user-guide/pipelines/pipelines.md index afbe8ddd..ff744d54 100644 --- a/docs/user-guide/pipelines/pipelines.md +++ b/docs/user-guide/pipelines/pipelines.md @@ -1,69 +1,113 @@ This page provides an overview of pipelines, their types, and how they work.
-## What are Pipelines? -Pipelines enable seamless data ingestion and transformation using an intuitive drag-and-drop interface. - +## Overview +Pipelines let you control what happens to your data after it is ingested into OpenObserve. They provide a clear way to read data from a source, apply changes, and send the processed output to a destination. ![Pipelines in OpenObserve](../../images/pipeline-new-editor.png) -## Types of Pipelines -OpenObserve supports two types of pipelines to cater to different data processing needs: +!!! note "Who can access pipelines" + Access to pipelines is controlled through role permissions. + Administrators can grant or restrict access from IAM > Roles > Permissions. + To allow a user to work with pipelines, the role must have permissions for the **Pipelines** module. These permissions control whether the user can list, create, update, or delete pipelines. + ![pipeline-permission](../../images/pipeline-permission.png) + +!!! note "Where to find pipelines" + You can access pipelines from the main navigation panel. Select **Pipelines** to view, create, and manage real-time and scheduled pipelines. + + +## Types of pipelines +OpenObserve supports two types of pipelines: - **Real-time pipelines** - **Scheduled pipelines** -### Real-Time Pipelines -A real-time Pipeline processes incoming raw data instantly, enabling immediate transformations and routing without delays. +!!! note "Both types use the same building blocks:" + + - **Source** defines where the data is read from. + - **Transform** defines how the data is changed or filtered. + - **Destination** defines where the processed data is written. + + Supported source stream types are logs, metrics, and traces. + Supported destination stream types are logs, metrics, traces, and enrichment tables. Enrichment tables are valid as destinations only for scheduled pipelines. -#### How they work +## Real-time pipelines +A real-time pipeline processes incoming data as soon as it is ingested into the source stream. It is used when you need immediate transformations, filtering, or routing. -1. **Source**: As soon as raw data is ingested into the source stream, the pipeline begins processing it. - - The supported source stream types are Logs, Metrics, or Traces.
**Note**: Each source stream can be associated with only one real-time pipeline. +> For more details, refer to the [Create and Use Real-time Pipelines](../create-and-use-real-time-pipeline/) guide. + +### How real-time pipelines work + +??? "Source" + The pipeline starts from a source stream. As soon as new data is ingested into this stream, the pipeline begins processing it.
+ **Each stream can be associated with only one real-time pipeline as its source.** This ensures a single, predictable processing flow for all new events in that stream. ![Pipelines in OpenObserve](../../images/pipelines-new-realtime.png) -2. **Transform**: The pipeline applies conditions or functions to filter and transform the data in real-time. -![Pipelines in OpenObserve](../../images/pipeline-new-realtime-transform-condition.png) -3. **Destination**: The transformed data is sent to the following destination(s) for further use or storage: + +??? "Transform" + The pipeline applies conditions or functions to filter and transform the data in real-time. + ![Pipelines in OpenObserve](../../images/pipeline-new-realtime-transform-condition.png) + +??? "Destination" + The transformed data is written to one or more destinations: + - **Stream**: The supported destination stream types are Logs, Metrics, Traces, or Enrichment tables.
**Note**: Enrichment Tables can only be used as destination streams in scheduled pipelines. ![Pipelines in OpenObserve](../../images/pipeline-new-realtime-destination.png) - - **Remote**: Select **Remote** if you wish to send data from the pipeline to [external destinations](#external-pipeline-destinations). + - **Remote**: Select **Remote** if you wish to send data from the pipeline to [external destinations](https://openobserve.ai/docs/user-guide/pipelines/remote-destination/). +
+ **Default destination added automatically**
+ When you select a source stream for a real-time pipeline, OpenObserve automatically adds a destination that points back to the same stream. This ensures that your original data continues to be stored in the source stream even when you add filters or functions in the pipeline. +
+ If this default destination is removed and you add only filtered or routed destinations, events that do not match any condition will be dropped. They will not be written to the source stream unless you explicitly add a destination for it. -#### When to use + **Key points**: + + - The automatic destination prevents data loss. + - It guarantees that every ingested event has at least one route. + - If you change the default routing and want to keep the original data, add a destination node that points to the source stream. + +### When to use real-time pipelines Use real-time pipelines when you need immediate processing, such as monitoring live data and cleaning logs in real-time. -### Scheduled Pipelines +## Scheduled pipelines +A scheduled pipeline processes historical data from an existing stream at user-defined intervals. This is useful when you need to extract, transform, and load (ETL) data at regular intervals without manual intervention. -A scheduled pipeline automates the processing of historical data from an existing stream at user-defined intervals. This is useful when you need to extract, transform, and load (ETL) data at regular intervals without manual intervention. -![Scheduled Pipelines in OpenObserve](../../images/pipelines-new-%20scheduled.png) -!!! note "Performance" - OpenObserve maintains a cache for scheduled pipelines to prevent the alert manager from making unnecessary database calls. This cache becomes particularly beneficial when the number of scheduled pipelines is high. For example, with 500 scheduled pipelines, the cache eliminates 500 separate database queries each time the pipelines are triggered, significantly improving performance. +> For more details, refer to the [Create and Use Scheduled Pipelines](../create-and-use-scheduled-pipeline/) guide. -#### How they work - -1. **Source**: To create a scheduled pipeline, you need an existing stream, which serves as the source stream. - - The supported source stream types are Logs, Metrics, or Traces. -2. **Query**: You write a SQL query to fetch historical data from the source stream. When defining the query, you also set the [**Frequency** and **Period**](#the-scheduled-pipeline-runs-based-on-the-user-defined-frequency-and-period) to control the scheduling. -![Scheduled Pipelines Query in OpenObserve](../../images/create-pipeline-sch-query.png) -3. **Transform**: The fetched data is transformed by applying functions or conditions. -![Scheduled Pipelines Transform in OpenObserve](../../images/pipeline-new-scheduled-condition.png) -4. **Destination**: The transformed data is sent to the following destination(s) for storage or further processing: - - **Stream**: The supported destination stream types are Logs, Metrics, Traces, or Enrichment tables.
**Note**: Enrichment Tables can only be used as destination streams in scheduled pipelines. - - **Remote**: Select **Remote** if you wish to send data to [external destination](https://openobserve.ai/docs/user-guide/pipelines/remote-destination/). +### How scheduled pipelines work + +??? "Source" + To create a scheduled pipeline, you need an existing stream, which serves as the source stream. The supported source stream types are Logs, Metrics, or Traces. + +??? "Query" + You define a SQL query that fetches historical data from the source stream. For metrics, you can also use PromQL. The query is controlled by **Frequency**, **Period**, and optional **Cron** settings. + ![Scheduled Pipelines in OpenObserve](../../images/configure-scheduled-pipelines.png) -#### Frequency and Period -The scheduled pipeline runs based on the user-defined **Frequency** and **Period**. + !!! note "Frequency, period, and cron" + Scheduled pipelines run according to the **Frequency** and **Period** you configure: -- **Frequency**: Defines how often the query should be executed.
**Example**: **Frequency**: 5 minutes
It ensures the query runs every 5 minutes. -- **Period**: Defines the period for which the query fetches the data.
- **Note**: The period should be the same as frequency, and both must be greater than 4.
- **Example**: **Period**: 5 minutes
It ensures the query fetches the data that was ingested in the last 5 minutes. -- **Frequency with Cron**: Cron allows you to define custom execution schedules based on specific expressions and timezones. It is ideal for scenarios requiring tasks to run at predefined times.
**Example**: **Cron Expression**: `0 0 1 * *` -
**Timezone**: Asia/Kolkata -
It ensures the query runs at 12:00 AM IST (00:00 in 24-hour format) on the first day of the month. + - **Frequency** defines how often the query runs. For example, a frequency of five minutes runs the query every five minutes. + - **Period** defines the time range the query reads in each run. usually be the same as the frequency. For example, a period of five minutes means the query fetches the data ingested in the last five minutes. + - **Cron** allows you to define custom execution schedules in a specific timezone. + For example, the cron expression `0 0 1 * *` with timezone `Asia/Kolkata` runs the query at midnight on the first day of each month. -#### When to use +??? "Transform" + After the query runs, the retrieved data passes through transform nodes. It allows you to filter or modify data before it reaches the destination. + ![Scheduled Pipelines Transform in OpenObserve](../../images/pipeline-new-scheduled-condition.png) + +??? "Destination" + The transformed data is written to one or more destinations. + + - **Stream** destinations can be Logs, Metrics, Traces, or Enrichment tables.
**Note**: Enrichment Tables can only be used as destination streams in scheduled pipelines. + - **Remote** destinations send data to external systems. To learn more about remote destinations, click [here](https://openobserve.ai/docs/user-guide/pipelines/remote-destination/). + +### When to use scheduled pipelines Use scheduled pipelines for tasks that require processing at fixed intervals instead of continuously, such as generating periodic reports and processing historical data in batches. +!!! note "Cache for scheduled pipelines" + OpenObserve maintains a cache for scheduled pipelines to prevent the alert manager from making unnecessary database calls. This cache becomes particularly beneficial when the number of scheduled pipelines is high. For example, with 500 scheduled pipelines, the cache eliminates 500 separate database queries each time the pipelines are triggered, significantly improving performance. + +## Next steps -## Next Steps -- [Create and Use Pipelines](../use-pipelines/) -- [Manage Pipelines](../manage-pipelines/) \ No newline at end of file +- [Create and Use Real-time Pipelines](../create-and-use-real-time-pipeline/) +- [Create and Use Scheduled Pipeline](../create-and-use-scheduled-pipeline/) +- [Manage Pipelines](../manage-pipelines/) +- [Environment Variables to Configure Pipelines](https://openobserve.ai/docs/environment-variables/#pipeline) \ No newline at end of file diff --git a/docs/user-guide/traces/.pages b/docs/user-guide/traces/.pages index 1f9899fb..1214dff1 100644 --- a/docs/user-guide/traces/.pages +++ b/docs/user-guide/traces/.pages @@ -2,4 +2,4 @@ nav: - Traces Overview: index.md - Traces in OpenObserve: traces.md - +- Service Graph: service-graph.md diff --git a/docs/user-guide/traces/service-graph.md b/docs/user-guide/traces/service-graph.md new file mode 100644 index 00000000..3687b5e7 --- /dev/null +++ b/docs/user-guide/traces/service-graph.md @@ -0,0 +1,109 @@ +=== "Overview" + Service graph provides a real time visual overview of how your services communicate with each other. It reads distributed traces and identifies which services call downstream services, how many requests flow between them, and how healthy those interactions are. + + !!! note "Note" + Service Graph is an Enterprise-only feature. + + **Key points**: + + - It helps you understand system behaviour at a glance. + - It highlights unusual behaviour so that you can quickly decide where to investigate next using Logs, Metrics, or Traces. + - It is not intended for detailed debugging. + - It helps you see the overall topology and identify the services that need closer attention. + + Service graph focuses on recent activity. When services are inactive for a period of time, they are removed from the graph to prevent unbounded growth and to keep the view relevant. + + + !!! note "Where to find this" + + 1. Sign in to OpenObserve. + 2. Select **Traces** in the left navigation panel. + 3. The **Service graph** icon that appears at the top-left corner of the page. + + The topology loads automatically when recent trace activity is available. + If there is no trace activity, the section displays a message indicating that no service graph data is available. + + + ## What service graph displays + Service graph displays the services in your system and the communication observed between them. + + ??? "Services" + ### Services + Each service represents an application component discovered from distributed traces. The view displays: + + - The service name + - A summary of recent requests + - A colour that reflects the recent error behaviour + + ??? "Colours" + ### Colours + Colours indicate the health of each service. + + - Green shows healthy behaviour + - Yellow and orange show increased errors + - Red shows repeated failures + + ??? "Edges" + ### Edges + Edges represent calls from one service to another. They indicate downstream communication and help you identify where issues may originate. + + ??? "Topology behaviour" + ### Topology behaviour + Service graph displays only recent activity. When a service produces no trace data for a set duration, it is removed from the topology. This design focuses attention on the active state of your system. + + + ## Data source + Service graph uses distributed traces as the single data source. + OpenObserve matches client spans and server spans belonging to the same request. This allows the system to identify service relationships, request patterns, and error behaviour. + + Service graph does not use logs or metrics directly. + + ## Main controls + + ??? "Stream selector" + ### Stream selector + Select a trace stream to build the topology from. You can choose a single stream or combine all streams. + + ??? "Search services" + ### Search services + Enter a service name to filter the view. The graph focuses on the selected service and the services connected to it. + + ??? "Refresh controls" + ### Refresh controls + Use manual refresh or enable automatic refresh. Auto refresh supports intervals between five seconds and three hundred seconds. + + ??? "View mode" + ### View mode + Choose between Tree view and Graph view. Both views show the same information using different layouts. + + ??? "Layout options" + ### Layout options + Tree view supports horizontal, vertical, and radial layouts. + Graph view supports force directed and circular layouts. + + ## Tree view + Tree view arranges services in a structured hierarchy. It is useful when you want to follow traffic from entrypoint services to downstream dependencies. + + ## Graph view + Graph view arranges services as a network. It uses a physics based simulation to maintain stable spacing between services. Force directed layouts group related services together. Circular layouts arrange services around a circle. + + ## Interaction + You can drag services to reposition them. You can zoom and pan to explore specific areas. Hovering over a service displays a summary of request and error behaviour. Filters and layouts can be combined to focus on specific sections of the topology. + +=== "How-to" + ## Filter the graph by service + + 1. Enter a service name in the **Search services** field. + 4. Review the focused view of the selected service. + + ## Change the view type + + 1. Select Tree view or Graph view. + 4. Select a layout. + + ## Investigate an issue + + 1. In the **Service graph**, identify any services with warning or error colours. + 3. Follow the edges to determine which downstream services may be affected. + 4. Note the service name. + 5. Switch to Logs, Metrics, or Traces and filter by that service for further analysis. \ No newline at end of file diff --git a/docs/user-guide/traces/traces.md b/docs/user-guide/traces/traces.md index 62402a52..9c7b2b66 100644 --- a/docs/user-guide/traces/traces.md +++ b/docs/user-guide/traces/traces.md @@ -76,7 +76,7 @@ This document explains how to use OpenObserve to collect, view, and analyze dist ![Service map](../../images/service-map.png) **Span details:** Provide metadata such as file path, code line, service version, thread ID, and additional attributes. Events and error messages appear when available. ![Span details](../../images/span-details.png) -=== "Configure" +=== "How-to"
The following configuration steps show how to set up OpenObserve for self-monitoring and for collecting traces from external applications. @@ -246,7 +246,6 @@ This document explains how to use OpenObserve to collect, view, and analyze dist **Note**: If no results appear after selecting **View Trace**, confirm that in your log stream the trace ID and span ID are available in the `trace_id` and `span_id` respectively. If not, ensure the custom trace ID and span ID fields are configured in the Organization parameter section under Management. ![config-traces-in-org-parameter](../../images/config-traces-in-org-parameter.png) -=== "How to" ## View a trace 1. Go to **Traces**. diff --git a/docs/work_group.md b/docs/work_group.md index c8fa0412..999bb4ef 100644 --- a/docs/work_group.md +++ b/docs/work_group.md @@ -1,13 +1,39 @@ --- -description: >- - Manage query performance in enterprise with Work Group settings—optimize CPU, - memory, and concurrency for long and short queries in OpenObserve streams. +description: Manage query performance in enterprise with Work Group settings—optimize CPU, memory, and concurrency for long and short queries in OpenObserve streams. --- -> `Applicable to enterprise version` -## All the ENV +Work groups control how OpenObserve allocates CPU, memory, and concurrency for different types of search tasks. They help maintain consistent performance when multiple users and system processes run searches at the same time. -``` +!!! note "Note" + This feature is available in the Enterprise Edition. + + +## Overview +OpenObserve evaluates each search task and assigns it to a work group. Each group receives its own limits for CPU, memory, and concurrency. When many tasks run at the same time, work groups prevent heavy tasks from slowing down interactive user queries. + +OpenObserve uses three work groups: + +- Short +- Long +- Background + +!!! note "Note" + Short and long groups manage user queries. The background group handles system tasks. + +## Background work group +The background work group handles system tasks that run independently of user activity. These tasks include: + +- Alert evaluations +- Report generation +- Derived stream processing + +The background group uses its own queue and resource limits. This ensures that system tasks do not interfere with user query performance. + + +## Environment variables +Work groups rely on the following environment variables for resource management and concurrency limits. + +```ini ZO_FEATURE_QUERY_QUEUE_ENABLED = false O2_SEARCH_GROUP_BASE_SPEED = 1024 // MB @@ -27,43 +53,70 @@ O2_SEARCH_GROUP_SHORT_MAX_CONCURRENCY = 4 O2_SEARCH_GROUP_USER_LONG_MAX_CONCURRENCY = 1 O2_SEARCH_GROUP_USER_SHORT_MAX_CONCURRENCY = 2 ``` +These variables define how much CPU and memory each group can use and how many queries each group can run at the same time. + + +### How OpenObserve assigns queries to work groups +OpenObserve estimates the execution time for each query and assigns it to the short or long work group based on the expected cost. -### How it works +The estimation uses the following values: + +- Allocate resources based on whether the query is short or long. +- Define two resource groups with 3 environments to limit the resource on each querier node: -1. Allocate resources based on whether it's a short or long query. -2. Define two resource groups with 3 environments to limit the resource on each querier node: - `O2_SEARCH_GROUP_x_MAX_CPU`, should be a percentage of the total CPU cores. - `O2_SEARCH_GROUP_x_MAX_MEMORY`, should be a percentage of the total `Datafusion` memory. - `O2_SEARCH_GROUP_x_MAX_CONCURRENCY`, should be a fixed number, minimal `1`. - 1. Long query group + + - Long query group: + - `O2_SEARCH_GROUP_LONG_MAX_CPU = 80%` - `O2_SEARCH_GROUP_LONG_MAX_MEMORY = 80%` - `O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY = 2` - 2. Short query group + + - Short query group: + - `O2_SEARCH_GROUP_SHORT_MAX_CPU = 20%` - `O2_SEARCH_GROUP_SHORT_MAX_MEMORY = 20%` - `O2_SEARCH_GROUP_SHORT_MAX_CONCURRENCY = 4` -3. The amount of available memory for per query request equals to `O2_SEARCH_GROUP_x_MAX_MEMORY / O2_SEARCH_GROUP_x_MAX_CONCURRENCY`. For example, if total system memory is `10GB` then Datafusion allow to use `50%`, which amounts to `5GB`; therefore, long-query groups have access to `80%` equating to `4GB` and supporting `2` concurrent processes means each search request can use up to `2GB` of RAM. -4. The search request will always use all of the CPU cores in its group. -5. Search requests exceeding concurrency limits will be queued and executed in FIFO order. -### User Quota-Based Resource Management +- The amount of memory available to a single search request in a group equals: `O2_SEARCH_GROUP_x_MAX_MEMORY / O2_SEARCH_GROUP_x_MAX_CONCURRENCY` + +!!! note "Example" + + - If total system memory is `10GB` then Datafusion can use `50%`, DataFusion has `5GB`. + - If the long group has `O2_SEARCH_GROUP_LONG_MAX_MEMORY = 80%`, it can use `4GB`. + - With `O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY = 2`, each long query can use up to `2GB` of memory. + +- A search request uses all CPU cores assigned to its work group as defined by `O2_SEARCH_GROUP_x_MAX_CPU`. +- When a search request exceeds the concurrency defined by `O2_SEARCH_GROUP_x_MAX_CONCURRENCY`, it is placed in a queue and executed later in first-in, first-out order. + -On top of global resource management settings, we also have user quota-based design elements. For example: + +### User quota-based resource management + +OpenObserve applies per-user quotas inside each work group. These limits prevent a single user from using all group concurrency. - `O2_SEARCH_GROUP_USER_LONG_MAX_CONCURRENCY = 1` - `O2_SEARCH_GROUP_USER_SHORT_MAX_CONCURRENCY = 2` -Even we allow to run `4` short queries in concurrent, but for same user only allow to run `2` short queries, if the user has over `2` request in concurrent then the exceeding request need wait into queue. +**Example**: + +- Short group allows four concurrent short queries through `O2_SEARCH_GROUP_SHORT_MAX_CONCURRENCY = 4`. +- One user can run only two short queries at the same time. +- If the same user sends more than two short queries, the extra queries wait in the queue even if global capacity is available. + +### How to calculate whether a search request is a long query or a short query +OpenObserve uses base speed and base seconds to estimate the expected execution time. -### How to calculate whether the search request is a long query or a short query? +- `O2_SEARCH_GROUP_BASE_SPEED = 1024` defines the assumed scan speed in megabytes per second, which is about one gigabyte per second. +- `O2_SEARCH_GROUP_BASE_SECS = 10` defines the time threshold for classification. Queries that exceed this threshold are long queries. +- The system knows the total CPU cores across querier nodes. +- The system knows the `scan_size` of the search request. -- We assume the search speed is `1GB`, `O2_SEARCH_GROUP_BASE_SPEED=1024`, this is configurable. -- We assume greater than `10s` is a long query, `O2_SEARCH_GROUP_BASE_SECS=10`, this is configurable. -- We know the total CPU cores of the queries in the cluster. -- We also know the `scan_size` of a search request, and then we can calculate the predicted seconds: +OpenObserve uses the following logic: -```rust +```rust linenums="1" let cpu_cores = max(2, CLUSTER_TOTAL_CPU_CORES * O2_SEARCH_GROUP_SHORT_MAX_CPU); let predict_secs = scan_size / O2_SEARCH_GROUP_BASE_SPEED / cpu_cores; if predict_secs > O2_SEARCH_GROUP_BASE_SECS { @@ -73,40 +126,82 @@ if predict_secs > O2_SEARCH_GROUP_BASE_SECS { } ``` -## How to decide Long or Short query +- The predicted time is the scan size divided by `O2_SEARCH_GROUP_BASE_SPEED` and then divided by the number of CPU cores used for the calculation. +- If the predicted time is greater than the value of `O2_SEARCH_GROUP_BASE_SECS`, the request is a long query. +- Otherwise, it is a short query. + + +## How to decide long or short query: example + +Cluster example: + +- Ten querier nodes +- Sixteen CPU cores and sixty-four gigabytes of memory per node +- Total CPU cores for queries: one hundred sixty + +From the environment variables: + +- Long queries use eighty percent of available CPU cores, which is one hundred twenty-eight cores. +- Short queries use twenty percent of available CPU cores, which is thirty-two cores. + + +**Short query example**: -- we have 10 querier, each node have 16 CPU cores and 64GB. then we have 160 CPU cores in the cluster for query. -- then we can use 128 (80% set using O2_SEARCH_GROUP_LONG_MAX_CPU ) CPU cores for long term query. -- then we can use 32 (20% set using O2_SEARCH_GROUP_SHORT_MAX_CPU ) CPU cores for short term query. +- Scan size: one hundred gigabytes +- Base speed: one gigabyte per second +- CPU for short queries: thirty-two cores -### Short query -We fire a query and we know that the scan_size is 100GB, then 100GB / 1GB (base_speed set using O2_SEARCH_GROUP_BASE_SPEED) / 32 CPU cores = `3s`, and the base_secs (set using O2_SEARCH_GROUP_BASE_SECS) is `10s`, so we decide this is a **short query**. +**Estimated time**: `100GB / 1GB per second / 32 cores = 3 seconds` +
+**Threshold**: `O2_SEARCH_GROUP_BASE_SECS = 10` +
+**Result**: This is a short query. -### Long query +
+**Long query example** -We fire a query and we know that the scan_size is 1TB, then 1024GB / 1GB (base_speed) / 32 CPU cores = `31s`, and the base_secs is `10s`, so we decide this is a **long query**. +- Scan size: one terabyte +- Base speed: one gigabyte per second +- CPU for short queries: thirty-two cores -## How to decide the MAX_CONCURRENCY +**Estimated time**: `1024GB / 1GB per second / 32 cores = 31 seconds` +
+**Result**: This is a long query. -This is based on your resource and the response time that you expect. -For example. We have 160 CPU cores and we assume the search speed is 1GB/core/secs, then we know if we want to search 1TB data, it need 1024GB / 1GB / 160 CPU = `6.4s`, but this is for single request. +## How to decide the maximum concurrency +Concurrency determines how many queries are allowed to run at the same time in each group. Increasing concurrency increases parallelism but also increases the response time for each task. -If you want to support two requests processing in parallel, then each request can use 50% resource it means only 80 CPU cores, then concurrent 2 requests and both search for 1TB data, each request will response in 1024GB / 1GB / 80 CPU = `12.8s`. -If you set the `O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY = 4`, then it will be: +Cluster example: -- when there is only one request, actually it can use all the CPU cores, the response time is `6.4s` -- when there are 2 concurrent requests, each request can use 50% CPU cores, the response time is `12.8s` -- when there are 4 concurrent requests, each request can use 25% CPU cores, the response time is `25.6s` -- when there are 5 concurrent requests, each request can use 25% CPU cores, the response time is `25.6s`, and the 5 request need wait in long term queue. +- Total CPU cores: one hundred sixty +- Scan size: one terabyte +- Base speed: one gigabyte per second -If you set the `O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY = 10`, then it will be: -- when there is only one request, actually it can use all the CPU cores, the response time is `6.4s` -- when there are 2 concurrent requests, each request can use 50% CPU cores, the response time is `12.8s` -- when there are 4 concurrent requests, each request can use 25% CPU cores, the response time is `25.6s` -- when there are 10 concurrent requests, each request can use 10% CPU cores, the response time is `64s` -- when there are 11 concurrent requests, each request can use 10% CPU cores, the response time is `64s`, and the 11 request need wait in long term queue. +**Single request**: +
+`1024GB / 1GB per second / 160 cores = 6.4 seconds` +

+**Two parallel requests** +
+Each request receives eighty cores: +
+`1024GB / 1GB per second / 80 cores = 12.8 seconds` +

+**Four parallel requests** +
+Each request receives forty cores: +
+`1024GB / 1GB per second / 40 cores = 25.6 seconds` +

+**Ten parallel requests** +
+Each request receives sixteen cores: +
+`1024GB / 1GB per second / 16 cores = 64 seconds` +
+If the number of requests exceeds the value of `O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY`, the extra requests wait in the queue. \ No newline at end of file diff --git a/overrides/partials/index.html b/overrides/partials/index.html index 6ec017a7..1cde255b 100644 --- a/overrides/partials/index.html +++ b/overrides/partials/index.html @@ -457,7 +457,7 @@

WINDOWS

Community & Support