diff --git a/.cursor/rules/word-list.mdc b/.cursor/rules/word-list.mdc new file mode 100644 index 000000000..a7b8c0a55 --- /dev/null +++ b/.cursor/rules/word-list.mdc @@ -0,0 +1,10 @@ +--- +description: Project word list and preferred spelling +alwaysApply: true +--- + +# Word list + +Use these spellings in this project (docs and code): + +- **drop-down** (not "dropdown") — e.g. "the filter drop-down", "drop-downs and filters" diff --git a/README.md b/README.md index ad1e593aa..cc2438339 100644 --- a/README.md +++ b/README.md @@ -1,315 +1,6 @@ -# Official ClickHouse data source for Grafana +### ClickHouse Support -The ClickHouse data source plugin allows you to query and visualize ClickHouse data in Grafana. +Grafana supports ClickHouse through a plugin. You can perform a variety of simple or complex ClickHouse queries to visualize logs or metrics stored in ClickHouse. You can also annotate your graphs with events stored in ClickHouse. -Grafana Dashboard Screenshot - Query Analysis - -Grafana Dashboard Screenshot - Data Analysis - -## Version compatibility - -Users on Grafana `v9.x` and higher of Grafana can use `v4`. -Users on Grafana `v8.x` are encouraged to continue using `v2.2.0` of the plugin. - -\* _As of 2.0 this plugin will only support ad hoc filters when using ClickHouse 22.7+_ - -## Installation - -For detailed instructions on how to install the plugin on Grafana Cloud or locally, -please checkout the [Plugin installation docs](https://grafana.com/docs/grafana/latest/plugins/installation/). - -## Configuration - -### ClickHouse user for the data source - -Set up an ClickHouse user account with [readonly](https://clickhouse.com/docs/en/operations/settings/permissions-for-queries#settings_readonly) permission and access to -databases and tables you want to query. -Please note that Grafana does not validate that queries are safe. Queries can contain any SQL statement. -For example, statements like `ALTER TABLE system.users DELETE WHERE name='sadUser'` -and `DROP TABLE sadTable;` would be executed. - -To configure a readonly user, follow these steps: - -1. Create a `readonly` user profile following the [Creating Users and Roles in ClickHouse](https://clickhouse.com/docs/en/operations/access-rights) guide. -2. Ensure the `readonly` user has enough permission to modify the `max_execution_time` setting required by the underlying [clickhouse-go client](https://github.com/ClickHouse/clickhouse-go/). -3. If you're using a public ClickHouse instance, it's not recommended to set `readonly=2` in the `readonly` profile. Instead, leave `readonly=1` and set the constraint type of `max_execution_time` to [changeable_in_readonly](https://clickhouse.com/docs/en/operations/settings/constraints-on-settings) to allow modification of this setting. - -### ClickHouse protocol support - -The plugin supports both `Native` (default) and `HTTP` transport protocols. -This can be enabled in the configuration via the `protocol` configuration parameter. -Both protocols exchange data with ClickHouse using optimized native format. - -Note that the default ports for `HTTP/S` and `Native` differ: - -- HTTP - 8123 -- HTTPS - 8443 -- Native - 9000 -- Native with TLS - 9440 - -### Manual configuration via UI - -Once the plugin is installed on your Grafana instance, follow -[these instructions](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/) -to add a new ClickHouse data source, and enter configuration options. - -### With a configuration file - -It is possible to configure data sources using configuration files with Grafana’s provisioning system. -To read about how it works, refer to -[Provisioning Grafana data sources](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources). - -Here are some provisioning examples for this data source using basic authentication: - -```yaml -apiVersion: 1 -datasources: - - name: ClickHouse - type: grafana-clickhouse-datasource - jsonData: - defaultDatabase: database - port: 9000 - host: localhost - username: username - tlsSkipVerify: false - # tlsAuth: - # tlsAuthWithCACert: - # secure: - # dialTimeout: - # queryTimeout: - # protocol: - # defaultTable: - # httpHeaders: - # - name: X-Example-Header - # secure: false - # value: - # - name: Authorization - # secure: true - # logs: - # defaultDatabase: - # defaultTable: - # otelEnabled: - # otelVersion: - # timeColumn: - # ...Column: - # traces: - # defaultDatabase: - # defaultTable: - # otelEnabled: - # otelVersion: - # durationUnit: - # traceIdColumn: - # ...Column: - secureJsonData: - password: password - # tlsCACert: - # tlsClientCert: - # tlsClientKey: - # secureHttpHeaders.Authorization: -``` - -## Building queries - -Queries can be built using the raw SQL editor or the query builder. -Queries can contain macros which simplify syntax and allow for -dynamic SQL generation. - -### Time series - -Time series visualization options are selectable after adding a `datetime` -field type to your query. This field will be used as the timestamp. You can -select time series visualizations using the visualization options. Grafana -interprets timestamp rows without explicit time zone as UTC. Any column except -`time` is treated as a value column. - -#### Multi-line time series - -To create multi-line time series, the query must return at least 3 fields in -the following order: - -- field 1: `datetime` field with an alias of `time` -- field 2: value to group by -- field 3+: the metric values - -For example: - -```sql -SELECT log_time AS time, machine_group, avg(disk_free) AS avg_disk_free -FROM mgbench.logs1 -GROUP BY machine_group, log_time -ORDER BY log_time -``` - -### Tables - -Table visualizations will always be available for any valid ClickHouse query. - -### Visualizing logs with the Logs Panel - -To use the Logs panel your query must return a timestamp and string values. To default to the logs visualization in Explore mode, set the timestamp alias to _log_time_. - -For example: - -```sql -SELECT log_time AS log_time, machine_group, toString(avg(disk_free)) AS avg_disk_free -FROM logs1 -GROUP BY machine_group, log_time -ORDER BY log_time -``` - -To force rendering as logs, in absence of a `log_time` column, set the Format to `Logs` (available from 2.2.0). - -### Visualizing traces with the Traces Panel - -Ensure your data meets the [requirements of the traces panel](https://grafana.com/docs/grafana/latest/explore/trace-integration/#data-api). This applies if using the visualization or Explore view. - -Set the Format to `Trace` when constructing the query (available from 2.2.0). - -If using the [Open Telemetry Collector and ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter), the following query produces the required column names (these are case sensitive): - -```sql -SELECT - TraceId AS traceID, - SpanId AS spanID, - SpanName AS operationName, - ParentSpanId AS parentSpanID, - ServiceName AS serviceName, - Duration / 1000000 AS duration, - Timestamp AS startTime, - arrayMap(key -> map('key', key, 'value', SpanAttributes[key]), mapKeys(SpanAttributes)) AS tags, - arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags, - if(StatusCode IN ('Error', 'STATUS_CODE_ERROR'), 2, 0) AS statusCode -FROM otel.otel_traces -WHERE TraceId = '61d489320c01243966700e172ab37081' -ORDER BY startTime ASC -``` - -### Macros - -To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros. - -Here is an example of a query with a macro that will use Grafana's time filter: - -```sql -SELECT date_time, data_stuff -FROM test_data -WHERE $__timeFilter(date_time) -``` - -| Macro | Description | Output example | -| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| _$\_\_dateFilter(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the date range of the panel | `date >= toDate('2022-10-21') AND date <= toDate('2022-10-23')` | -| _$\_\_timeFilter(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in seconds | `time >= toDateTime(1415792726) AND time <= toDateTime(1447328726)` | -| _$\_\_timeFilter_ms(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in milliseconds | `time >= fromUnixTimestamp64Milli(1415792726123) AND time <= fromUnixTimestamp64Milli(1447328726456)` | -| _$\_\_dateTimeFilter(dateColumn, timeColumn)_ | Shorthand that combines $**dateFilter() AND $**timeFilter() using separate Date and DateTime columns. | `$__dateFilter(dateColumn) AND $__timeFilter(timeColumn)` | -| _$\_\_fromTime_ | Replaced by the starting time of the range of the panel casted to `DateTime` | `toDateTime(1415792726)` | -| _$\_\_toTime_ | Replaced by the ending time of the range of the panel casted to `DateTime` | `toDateTime(1447328726)` | -| _$\_\_fromTime_ms_ | Replaced by the starting time of the range of the panel casted to `DateTime64(3)` | `fromUnixTimestamp64Milli(1415792726123)` | -| _$\_\_toTime_ms_ | Replaced by the ending time of the range of the panel casted to `DateTime64(3)` | `fromUnixTimestamp64Milli(1447328726456)` | -| _$\_\_interval_s_ | Replaced by the interval in seconds | `20` | -| _$\_\_timeInterval(columnName)_ | Replaced by a function calculating the interval based on window size in seconds, useful when grouping | `toStartOfInterval(toDateTime(column), INTERVAL 20 second)` | -| _$\_\_timeInterval_ms(columnName)_ | Replaced by a function calculating the interval based on window size in milliseconds, useful when grouping | `toStartOfInterval(toDateTime64(column, 3), INTERVAL 20 millisecond)` | -| _$\_\_conditionalAll(condition, $templateVar)_ | Replaced by the first parameter when the template variable in the second parameter does not select every value. Replaced by the 1=1 when the template variable selects every value. | `condition` or `1=1` | - -The plugin also supports notation using braces {}. Use this notation when queries are needed inside parameters. - -### Templates and variables - -To add a new ClickHouse query variable, refer to [Add a query -variable](https://grafana.com/docs/grafana/latest/variables/variable-types/add-query-variable/). - -After creating a variable, you can use it in your ClickHouse queries by using -[Variable syntax](https://grafana.com/docs/grafana/latest/variables/syntax/). -For more information about variables, refer to [Templates and -variables](https://grafana.com/docs/grafana/latest/variables/). - -### Importing dashboards for ClickHouse - -Follow these -[instructions](https://grafana.com/docs/grafana/latest/dashboards/export-import/#import-dashboard) -to import a dashboard. - -You can also find available, pre-made dashboards by navigating to the data -sources configuration page, selecting the ClickHouse data source and clicking -on the Dashboards tab. - -We distribute the following dashboards with the plugin. These are aimed at assisting with support analysis of a ClickHouse cluster and do not rely on external datasets. The querying user requires access to the `system` database. - -1. Cluster Analysis - an overview of configured clusters, merges, mutations and data replication. -2. Data Analysis - an overview of current databases and tables, including their respective sizes, partitions and parts. -3. Query Analysis - an analysis of queries by type, performance and resource consumption. - -### Ad Hoc Filters - -Ad hoc filters are only supported with version 22.7+ of ClickHouse. - -Ad hoc filters allow you to add key/value filters that are automatically added -to all metric queries that use the specified data source, without being -explicitly used in queries. - -By default, Ad Hoc filters will be populated with all Tables and Columns. If -you have a default database defined in the Datasource settings, all Tables from -that database will be used to populate the filters. As this could be -slow/expensive, you can introduce a second variable to allow limiting the -Ad Hoc filters. It should be a `constant` type named `clickhouse_adhoc_query` -and can contain: a comma delimited list of databases, just one database, or a -database.table combination to show only columns for a single table. - -Ad Hoc filters also work with the Map and JSON types for OTel data. -Map is the default, and will automatically convert the merged labels output into a usable filter. -To have the filter logic use JSON syntax, add a dashboard variable with a `constant` type called `clickhouse_adhoc_use_json` (the variable's `value` is ignored, it just has to be present). - -For more information on Ad Hoc filters, check the [Grafana -docs](https://grafana.com/docs/grafana/latest/variables/variable-types/add-ad-hoc-filters/) - -#### Using a query for Ad Hoc filters - -The second `clickhouse_adhoc_query` also allows any valid ClickHouse query. The -query results will be used to populate your ad-hoc filter's selectable filters. -You may choose to hide this variable from view as it serves no further purpose. - -For example, if `clickhouse_adhoc_query` is set to `SELECT DISTINCT -machine_name FROM mgbench.logs1` you would be able to select which machine -names are filtered for in the dashboard. - -#### Manual Ad Hoc Filter Placement with `$__adHocFilters` - -By default, ad-hoc filters are automatically applied to queries by detecting the -target table using SQL parsing. However, for queries that use CTEs or ClickHouse-specific -syntax like `INTERVAL` or aggregate functions with parameters, the automatic -detection may fail. In these cases, you can manually specify where to apply -ad-hoc filters using the `$__adHocFilters('table_name')` macro. - -This macro expands to the ClickHouse `additional_table_filters` setting with the -currently active ad-hoc filters. It should be placed in the `SETTINGS` clause of -your query. - -Example: - -```sql -SELECT * -FROM ( - SELECT * FROM my_complex_table - WHERE complicated_condition -) AS result -SETTINGS $__adHocFilters('my_complex_table') -``` - -When ad-hoc filters are active (e.g., `status = 'active'` and `region = 'us-west'`), -this expands to: - -```sql -SELECT * -FROM ( - SELECT * FROM my_complex_table - WHERE complicated_condition -) AS result -SETTINGS additional_table_filters={'my_complex_table': 'status = \'active\' AND region = \'us-west\''} -``` - -## Learn more - -- Add [Annotations](https://grafana.com/docs/grafana/latest/dashboards/annotations/). -- Configure and use [Templates and variables](https://grafana.com/docs/grafana/latest/variables/). -- Add [Transformations](https://grafana.com/docs/grafana/latest/panels/transformations/). -- Set up alerting; refer to [Alerts overview](https://grafana.com/docs/grafana/latest/alerting/). +Read more about it here: +[Grafana ClickHouse Documentation](https://grafana.com/docs/grafana/latest/datasources/clickhouse/) diff --git a/cspell.config.json b/cspell.config.json index 039ca6ad6..69bd81390 100644 --- a/cspell.config.json +++ b/cspell.config.json @@ -8,7 +8,10 @@ "yarn.lock", "package-lock.json", "go.sum", - "mage_output_file.go" + "mage_output_file.go", + "test-results/**", + "test_summary.json", + "playwright-report/**" ], "words": [ "aggregatable", @@ -46,6 +49,7 @@ "mgbench", "Milli", "millis", + "mydb", "networkidle", "nofile", "nolint", @@ -68,6 +72,7 @@ "stagename", "stretchr", "subquery", + "subqueries", "subresource", "sugg", "supress", @@ -89,6 +94,7 @@ "Ulimits", "uuidv", "vectorator", - "WorkDir" + "WorkDir", + "varname" ] } diff --git a/docs/sources/_index.md b/docs/sources/_index.md new file mode 100644 index 000000000..079da9759 --- /dev/null +++ b/docs/sources/_index.md @@ -0,0 +1,89 @@ +--- +description: This document introduces the ClickHouse data source +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source +menuTitle: ClickHouse data source +title: ClickHouse data source +weight: 10 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# ClickHouse data source + +The ClickHouse data source allows you to query and visualize ClickHouse data in Grafana. Use the raw SQL editor or query builder to create dashboards, time series, tables, logs, and traces. + +## Supported features + +| Feature | Supported | +|---------|-----------| +| Metrics | Yes | +| Logs | Yes | +| Traces | Yes | +| Alerting | Yes | +| Annotations | Yes | + +## Requirements + +| Grafana version | Plugin version | +|-----------------|----------------| +| 9.x and later | v4 | +| 8.x | v2.2.0 | + +{{< admonition type="note" >}} +Ad hoc filters require ClickHouse 22.7 or later (from plugin v2.0 onward). +{{< /admonition >}} + +## Get started + +The following documents help you get started: + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/) +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) +- [ClickHouse template variables](/docs/plugins/grafana-clickhouse-datasource//template-variables/) +- [ClickHouse annotations](/docs/plugins/grafana-clickhouse-datasource//annotations/) +- [ClickHouse alerting](/docs/plugins/grafana-clickhouse-datasource//alerting/) +- [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/) + +## Additional features + +After configuring the data source, you can: + +- Use [Explore](https://grafana.com/docs/grafana/latest/explore/) to query data without building a dashboard +- Add [Transformations](https://grafana.com/docs/grafana/latest/panels/transformations/) to manipulate query results +- Set up [Alerting](https://grafana.com/docs/grafana/latest/alerting/) rules + +## Pre-built dashboards + +The plugin includes the following pre-built dashboards: + +- **ClickHouse - Query Analysis** — Query performance, time distribution, top users, and memory usage. +- **ClickHouse - Data Analysis** — Disk usage, table and database summary, parts over time, and dictionaries. +- **ClickHouse - Cluster Analysis** — Cluster overview, merges, mutations, and replicated table delay. +- **Simple ClickHouse OTel Dashboard** — Traces, logs, and service performance for OpenTelemetry data in ClickHouse. +- **Advanced ClickHouse Monitoring Dashboard** — System metrics (CPU, queries/sec, IO, memory) similar to ClickHouse built-in monitoring. + +To import a pre-built dashboard: + +1. Go to **Connections** > **Data sources**. +2. Select your ClickHouse data source. +3. Click the **Dashboards** tab. +4. Click **Import** next to the dashboard you want to use. + +## Plugin updates + +Always ensure that your plugin version is up-to-date so you have access to all current features and improvements. Navigate to **Plugins and data** > **Plugins** to check for updates. Grafana recommends upgrading to the latest Grafana version, and this applies to plugins as well. + +{{< admonition type="note" >}} +Plugins are automatically updated in Grafana Cloud. +{{< /admonition >}} + +## Related resources + +- [ClickHouse documentation](https://clickhouse.com/docs) +- [Grafana community forum](https://community.grafana.com/) diff --git a/docs/sources/alerting.md b/docs/sources/alerting.md new file mode 100644 index 000000000..28d53d686 --- /dev/null +++ b/docs/sources/alerting.md @@ -0,0 +1,123 @@ +--- +description: Create alert rules from ClickHouse queries +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source + - alerting +menuTitle: ClickHouse alerting +title: ClickHouse alerting +weight: 55 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# ClickHouse alerting + +The ClickHouse data source supports [Grafana Alerting](https://grafana.com/docs/grafana/latest/alerting/) and [Grafana-managed recording rules](https://grafana.com/docs/grafana/latest/alerting/configure-alert-rules/create-recording-rules/). You can create alert rules that run ClickHouse SQL queries and fire when the result meets a condition (for example, a value is above a threshold or no data is returned). + +Alert rules run as background processes. Grafana executes your ClickHouse query on a schedule, then evaluates the result using expressions such as **Reduce** and **Threshold**. Your query must return numeric data that Grafana can evaluate. + +For an overview of alerting in Grafana, see [Alert rules](https://grafana.com/docs/grafana/latest/alerting/configure-alert-rules/) and [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/latest/alerting/configure-alert-rules/create-grafana-managed-rule/). + +## Before you begin + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/). +- Ensure your ClickHouse user has read access to the tables used in your alert query. +- Familiarize yourself with [Grafana Alerting concepts](https://grafana.com/docs/grafana/latest/alerting/). + +## Query requirements for alerting + +Alert rules need numeric values to evaluate. Your ClickHouse query should return data that Grafana can use in a **Reduce** expression and then compare in a **Threshold** expression. + +| Query result | Use case | +|--------------|----------| +| **Time series** (time column + numeric column) | Threshold alerts on a metric (e.g. average CPU, error count per interval). Use **Reduce** (Last, Max, Mean, etc.) to get a single value from the series. | +| **Single row, numeric value** | Threshold alerts on an aggregate (e.g. `SELECT count() FROM errors WHERE ...`). Use **Reduce** > **Last** to use the value. | + +Queries that return only text or non-numeric data cannot be used directly for threshold evaluation. Use `count()`, `avg()`, `sum()`, or similar in your SQL so the result is numeric. + +Use the **$__timeFilter(column)** macro in your WHERE clause so the query respects the alert rule’s evaluation interval and time range. See the [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) Macros section. + +## Create an alert rule + +To create an alert rule using ClickHouse data: + +1. Go to **Alerting** > **Alert rules**. +2. Click **New alert rule**. +3. Enter a **Name** for the rule. +4. In **Define query and alert condition**: + - Select your **ClickHouse** data source. + - In the **Query** tab, write a ClickHouse SQL query that returns a time column and a numeric column (or a single numeric value). Use **Format** > **Time series** if your query returns time + value; use **Table** if it returns a single row. + - Add a **Reduce** expression to aggregate the query result (e.g. **Last** to use the latest value, **Max** for the highest, **Mean** for the average). + - Add a **Threshold** expression to define when the alert fires (e.g. **Is above** 80, **Is below** 3). +5. Configure **Set evaluation behavior**: choose a folder and evaluation group, set the evaluation interval, and set the pending period. +6. Add **Labels** and **Annotations** for notifications. +7. Click **Save rule**. + +For detailed steps, see [Create a Grafana-managed alert rule](https://grafana.com/docs/grafana/latest/alerting/configure-alert-rules/create-grafana-managed-rule/). + +## Example: Metric threshold alert + +This example fires when the average value of a metric exceeds 80. Replace the table and column names with your own. + +**Query (Time series format):** + +```sql +SELECT + $__timeInterval(timestamp) AS time, + avg(value) AS value +FROM my_app.metrics +WHERE $__timeFilter(timestamp) + AND metric_name = 'cpu_usage' +GROUP BY time +ORDER BY time +``` + +In the alert rule, add **Reduce** > **Last** (or **Max**) and **Threshold** > **Is above** 80. + +## Example: Error count alert + +This example fires when the number of error rows in the last 5 minutes exceeds 10. + +**Query (Table format; single row):** + +```sql +SELECT count() AS value +FROM my_app.events +WHERE $__timeFilter(timestamp) + AND level = 'error' +``` + +In the alert rule, set the query **Format** to **Table**, add **Reduce** > **Last**, and **Threshold** > **Is above** 10. + +## Example: No data alert + +You can alert when a query returns no rows (for example, a health check that should always return at least one row). Write a query that returns a row when things are healthy, then in the alert rule configure **Configure no data and error handling** to **Alerting** when there is no data. + +## Best practices + +1. **Use an appropriate evaluation interval** — Set the alert evaluation interval to match how often your data is written. Avoid intervals shorter than your data resolution to prevent flapping or missed data. +2. **Reduce multiple series** — If your query returns multiple time series (e.g. one per host), use **Reduce** to aggregate: **Last**, **Max**, **Mean**, or **Sum** so Grafana can evaluate a single value. +3. **Restrict the time range** — Use **$__timeFilter(column)** in your WHERE clause so the query only reads data in the evaluation window. Avoid full table scans. +4. **Handle no data** — In **Configure no data and error handling**, choose whether no data should keep the alert as-is, fire the alert, or resolve it. Use **Alerting** when no data indicates a problem (e.g. a heartbeat query). +5. **Test the query first** — Run the query in **Explore** with the ClickHouse data source and confirm it returns the expected numeric data before saving the alert rule. + +## Troubleshooting + +If alerts do not fire or evaluate as expected: + +- **Query returns no numeric data** — Confirm the query returns a time column and a numeric column (or a single numeric value). Test in **Explore** and check the result format. +- **Evaluation interval** — Ensure the evaluation interval is long enough for data to be available. Avoid intervals shorter than your data resolution. +- **No data handling** — In **Configure no data and error handling**, choose whether no data should fire the alert, resolve it, or keep the current state. + +For connection errors, timeouts, or other data source issues, see [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/). + +## Next steps + +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) — Macros such as `$__timeFilter` and `$__timeInterval`. +- [Grafana Alerting](https://grafana.com/docs/grafana/latest/alerting/) — Alert rules, contact points, and notification policies. +- [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/) — Common errors and solutions. diff --git a/docs/sources/annotations.md b/docs/sources/annotations.md new file mode 100644 index 000000000..9aeff7728 --- /dev/null +++ b/docs/sources/annotations.md @@ -0,0 +1,114 @@ +--- +description: Use ClickHouse queries to create annotations on dashboards +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source + - annotations +menuTitle: ClickHouse annotations +title: ClickHouse annotations +weight: 50 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# ClickHouse annotations + +Annotations overlay event markers on your dashboard panels. You can use ClickHouse SQL queries to create annotations that mark deployments, alerts, errors, or other events from your data. + +The plugin uses Grafana’s default annotation support: you write a ClickHouse query that returns a time column and a text column. Grafana positions each row as an annotation on the time axis and shows the text when you hover or click. + +For an overview of annotations in Grafana, see [Annotate visualizations](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/). + +## Before you begin + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/). +- Ensure your ClickHouse user has read access to the tables you use in your annotation query. + +## Create an annotation query + +To add a ClickHouse annotation to a dashboard: + +1. Open the dashboard where you want to add annotations. +2. Click **Dashboard settings** (gear icon) in the top navigation. +3. Select **Annotations** in the left menu. +4. Click **Add annotation query**. +5. Enter a **Name** for the annotation (for example, "Deployments", "Errors"). +6. In the **Data source** drop-down, select your ClickHouse data source. +7. In the **Query** field, enter a ClickHouse SQL query that returns the required columns (see [Query requirements](#query-requirements)). +8. Use the **Column mappings** section to map your query columns to **Time**, **Text**, and optionally **Tags** (if your column names differ from Grafana’s defaults). +9. Click **Apply** to save. + +## Query requirements + +Your SQL query must return at least a time column and a text column. Grafana uses these to place and label each annotation. + +| Column | Required | Description | +|--------|----------|-------------| +| **Time** | Yes | The timestamp for the annotation. Grafana uses this to position the marker on the time axis. Use a DateTime or DateTime64 column, or an expression that Grafana can interpret as time. | +| **Text** | Yes | The annotation text shown when you hover over or click the marker. | +| **Tags** | Optional | Additional columns become annotation tags. Use them to filter or group annotations. | + +Always restrict the query to the dashboard time range so annotations load quickly. Use the **$__timeFilter(column)** macro in your WHERE clause; see the [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) Macros section. + +## Annotation query examples + +The following examples show common patterns. Replace the table and column names with your own. + +**Application events (e.g. deployments or status changes):** + +```sql +SELECT + event_time AS time, + message AS text, + environment AS tag +FROM my_app.events +WHERE $__timeFilter(event_time) + AND event_type IN ('deployment', 'status_change') +ORDER BY event_time DESC +LIMIT 100 +``` + +**Query log events (e.g. long-running or failed queries from ClickHouse system tables):** + +```sql +SELECT + event_time AS time, + concat(type, ': ', substring(query, 1, 80)) AS text, + initial_user AS tag +FROM system.query_log +WHERE $__timeFilter(event_time) + AND type IN ('QueryFinish', 'ExceptionWhileProcessing') +ORDER BY event_time DESC +LIMIT 100 +``` + +**Errors or alerts from a custom table:** + +```sql +SELECT + timestamp AS time, + concat(severity, ' - ', message) AS text, + service AS tag +FROM my_app.alerts +WHERE $__timeFilter(timestamp) +ORDER BY timestamp DESC +LIMIT 100 +``` + +## Best practices + +1. **Use a time filter** — Include `$__timeFilter(your_time_column)` in the WHERE clause so the query only returns data in the dashboard time range. +2. **Limit results** — Use `LIMIT` (for example, 100) so the query stays fast and the dashboard does not show too many markers. +3. **Meaningful text** — Use `concat()` or similar so the text column is clear (e.g. event type plus a short description). +4. **Use tags** — Return one or more tag columns (e.g. environment, service, user) so users can filter annotations in the dashboard. +5. **Descriptive names** — Give the annotation a clear name (e.g. "Production deployments", "Query errors") so dashboard users know what it represents. + +## Next steps + +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) — Macros such as `$__timeFilter` and building queries. +- [Annotate visualizations](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/) — Grafana annotation options (colors, which panels show annotations, filters). +- [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/) — Common errors and solutions. diff --git a/docs/sources/configure.md b/docs/sources/configure.md new file mode 100644 index 000000000..dcc6830b7 --- /dev/null +++ b/docs/sources/configure.md @@ -0,0 +1,173 @@ +--- +description: This document outlines configuration options for the ClickHouse data source +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source +menuTitle: Configure the ClickHouse data source +title: Configure the ClickHouse data source +weight: 20 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# Configure the ClickHouse data source + +This document explains how to configure the ClickHouse data source. + +## Before you begin + +Before configuring the data source, ensure you have: + +- **Grafana permissions:** Organization administrator role. +- **Plugin:** The ClickHouse data source plugin installed. For Grafana version compatibility, see [Requirements](/docs/plugins/grafana-clickhouse-datasource//#requirements). +- **ClickHouse:** A running ClickHouse server and a user with read-only access (or the permissions described below). +- **Network access:** The Grafana server can reach the ClickHouse server on the intended port (HTTP: 8123 or 8443 with TLS; Native: 9000 or 9440 with TLS). + +## ClickHouse user and permissions + +Grafana executes queries exactly as written and does not validate or restrict SQL. Use a **read-only ClickHouse user** for this data source to avoid accidental or destructive operations (such as modifying or deleting tables) while still allowing dashboards and queries to run. + +If your ClickHouse administrator has already given you a read-only user and connection details, you can skip to [Add the data source](#add-the-data-source). + +## Recommended permissions + +Create a ClickHouse user with: + +- **readonly** permission enabled +- Access limited to the databases and tables you intend to query +- Permission to modify the **max_execution_time** setting (required by the plugin’s client) + +{{< admonition type="warning" >}} +Grafana does not prevent execution of non-read queries. If the ClickHouse user has sufficient privileges, statements such as `DROP TABLE` or `ALTER TABLE` will be executed by ClickHouse. +{{< /admonition >}} + +## Configure a read-only user + +To configure a suitable read-only user: + +1. Create a user or profile using [Creating users and roles in ClickHouse](https://clickhouse.com/docs/en/operations/access-rights). +1. Set `readonly = 1` for the user or profile. For details, see [Permissions for queries (readonly)](https://clickhouse.com/docs/en/operations/settings/permissions-for-queries#readonly). +1. Allow modification of the **max_execution_time** setting, which is required by the [clickhouse-go](https://github.com/ClickHouse/clickhouse-go/) client so the plugin can enforce query timeouts. + +If you use a **public ClickHouse instance**, do not set `readonly = 2`. Instead: + +- Keep `readonly = 1` +- Set the constraint type for **max_execution_time** to [changeable_in_readonly](https://clickhouse.com/docs/en/operations/settings/constraints-on-settings) + +## ClickHouse protocol support + +The data source supports two transport protocols: **Native** (default) and **HTTP**. Both use ClickHouse’s optimized native data formats and support the same query capabilities. + +## Default ports + +| Protocol | TLS | Port | +|----------|------|------| +| HTTP | No | 8123 | +| HTTP | Yes | 8443 | +| Native | No | 9000 | +| Native | Yes | 9440 | + +When you enable **Secure connection (TLS)** in Grafana, use a port that supports TLS. Grafana does not change the port automatically when TLS is enabled. + +## Add the data source + +To add the data source: + +1. Click **Connections** in the left-side menu. +1. Click **Add new connection**. +1. Type **ClickHouse** in the search bar. +1. Select **ClickHouse**. +1. Click **Add new data source**. + +## Configure settings + +After adding the data source, configure the following: + +| Setting | Description | +|---------|-------------| +| **Name** | The name used to refer to the data source in panels and queries. | +| **Default** | Toggle to make this the default data source for new panels. | +| **Server** | The ClickHouse server host (for example, `localhost`). | +| **Protocol** | **Native** or **HTTP**. | +| **Port** | Port number; depends on protocol and whether TLS is enabled (see default ports above). | +| **Secure connection** | Enable when your ClickHouse server uses TLS. | +| **Username** | ClickHouse user name. | +| **Password** | ClickHouse user password. | + +## Verify the connection + +Click **Save & test** to verify the connection. When the connection test succeeds, you see **Data source is working**. A successful test confirms that Grafana can reach ClickHouse and that the credentials are valid. + +If the test fails, refer to [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/) for common configuration errors and solutions. + +## Provision the data source + +You can define the data source in YAML files as part of the Grafana provisioning system. For more information, refer to [Provisioning Grafana data sources](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources). + +Example ClickHouse data source configuration with basic authentication: + +```yaml +apiVersion: 1 +datasources: + - name: ClickHouse + type: grafana-clickhouse-datasource + jsonData: + host: localhost + port: 9000 + protocol: native + defaultDatabase: database + tlsSkipVerify: false + # secure: + # tlsAuth: + # tlsAuthWithCACert: + # dialTimeout: + # queryTimeout: + # defaultTable: + # httpHeaders: + # - name: X-Example-Header + # secure: false + # value: + secureJsonData: + username: username + password: password + # tlsCACert: + # tlsClientCert: + # tlsClientKey: +``` + +## Provision with Terraform + +You can provision the ClickHouse data source using the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). Example with basic authentication: + +```hcl +resource "grafana_data_source" "clickhouse" { + type = "grafana-clickhouse-datasource" + name = "ClickHouse" + + json_data = { + defaultDatabase = "default" + port = 9000 + host = "localhost" + protocol = "native" + tlsSkipVerify = false + } + + secure_json_data = { + password = var.clickhouse_password + } +} +``` + +For more options and authentication methods, refer to the [Grafana Terraform provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). + +## Next steps + +After configuring the data source: + +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) — Build queries with the SQL editor or query builder. +- [ClickHouse template variables](/docs/plugins/grafana-clickhouse-datasource//template-variables/) — Use variables in dashboards and queries. +- [ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//) — Overview, supported features, and pre-built dashboards. diff --git a/docs/sources/query-editor.md b/docs/sources/query-editor.md new file mode 100644 index 000000000..f934a507b --- /dev/null +++ b/docs/sources/query-editor.md @@ -0,0 +1,197 @@ +--- +description: This document describes the ClickHouse query editor +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source +menuTitle: ClickHouse query editor +title: ClickHouse query editor +weight: 30 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# ClickHouse query editor + +This document explains how to use the ClickHouse query editor to build and run queries. You can access the query editor from [Explore](https://grafana.com/docs/grafana/latest/visualizations/explore/) to run ad hoc queries, or when you add or edit a panel and select the ClickHouse data source. + +## Before you begin + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/). +- Ensure your ClickHouse user has read access to the databases and tables you want to query. + +## Query editor elements + +The query editor appears in [Explore](https://grafana.com/docs/grafana/latest/visualizations/explore/) when you select the ClickHouse data source, or when you add or edit a panel and select ClickHouse. It includes the following elements. + +| Element | Description | +|---------|-------------| +| **Editor type** | Switch between **SQL** (write raw SQL) and **Query builder** (build queries with drop-downs and filters). | +| **Run Query** | Runs the current query and refreshes the panel. In the SQL editor you can also use **Ctrl+Enter** (Windows/Linux) or **Cmd+Enter** (macOS). | +| **Query type** | Choose the result format: **Table**, **Logs**, **Time series**, or **Traces**. This sets how Grafana interprets and visualizes the results. Available in both SQL and Query builder modes. | + +**In SQL mode:** + +- **SQL editor** — A code editor where you write ClickHouse SQL. It provides schema suggestions (databases, tables, columns) as you type. If SQL validation is enabled in the data source settings, the editor marks invalid syntax. +- **Format code** — Use the editor toolbar to format your SQL. +- **Query type** — Select **Table**, **Logs**, **Time series**, or **Traces** so the panel uses the correct visualization. + +**In Query builder mode:** + +- **Database** and **Table** — Select the database and table to query from the drop-downs. +- **Query type** — Select **Table**, **Logs**, **Time series**, or **Traces**. The builder shows options that match the type (for example, time column and value columns for time series; columns, filters, group by, and order by for tables). +- **Type-specific options** — Configure columns, filters, grouping, sorting, limit (max rows), and (for traces) trace ID. The builder generates the SQL for you. +- **SQL preview** — At the bottom of the builder, you can see the generated SQL. You can switch to SQL mode to edit it manually. + +## Build queries + +You can build queries using the **SQL editor** (raw SQL) or the **Query builder**. Queries can include macros for dynamic parts such as time range filters. + +## Time series + +For time series visualizations, your query must include a `datetime` column. Use an alias of `time` for the timestamp column. Grafana treats timestamp rows without an explicit time zone as UTC. Any column other than `time` is treated as a value column. + +## Multi-line time series + +To create multi-line time series, the query must return at least 3 columns in this order: + +1. A `datetime` column with an alias of `time` +1. A column to group by (for example, category or label) +1. One or more metric columns + +Example (replace `mgbench.logs1` with your database and table): + +```sql +SELECT log_time AS time, machine_group, avg(disk_free) AS avg_disk_free +FROM mgbench.logs1 +GROUP BY machine_group, log_time +ORDER BY log_time +``` + +## Tables + +Table visualizations are available for any valid ClickHouse query. Select **Table** in the panel visualization options to view results in tabular form. + +## Visualize logs with the Logs panel + +To use the Logs panel, your query must return a timestamp and string values. To default to the logs visualization in Explore, set the timestamp column alias to `log_time`. + +Example (replace `logs1` with your database and table, for example `mydb.logs`): + +```sql +SELECT log_time AS log_time, machine_group, toString(avg(disk_free)) AS avg_disk_free +FROM logs1 +GROUP BY machine_group, log_time +ORDER BY log_time +``` + +When you don't have a `log_time` column, set **Format** to **Logs** to force logs rendering (available from plugin version 2.2.0). + +## Visualize traces with the Traces panel + +To use the Traces panel, your data must meet the [requirements of the traces panel](https://grafana.com/docs/grafana/latest/explore/trace-integration/#data-api). Set **Format** to **Trace** when building the query (available from plugin version 2.2.0). + +If you use the [OpenTelemetry Collector and ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter), the following query returns the required column names (case sensitive). Replace the trace ID in the WHERE clause with your trace ID or a template variable (for example `$traceId`): + +```sql +SELECT + TraceId AS traceID, + SpanId AS spanID, + SpanName AS operationName, + ParentSpanId AS parentSpanID, + ServiceName AS serviceName, + Duration / 1000000 AS duration, + Timestamp AS startTime, + arrayMap(key -> map('key', key, 'value', SpanAttributes[key]), mapKeys(SpanAttributes)) AS tags, + arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags, + if(StatusCode IN ('Error', 'STATUS_CODE_ERROR'), 2, 0) AS statusCode +FROM otel.otel_traces +WHERE TraceId = '61d489320c01243966700e172ab37081' +ORDER BY startTime ASC +``` + +## Macros + +Macros simplify query syntax and add dynamic parts such as dashboard time range filters. The plugin replaces macros with the corresponding SQL before the query is sent to ClickHouse. + +Example query using a time filter macro (replace `test_data` and `date_time` with your table and timestamp column): + +```sql +SELECT date_time, data_stuff +FROM test_data +WHERE $__timeFilter(date_time) +``` + +| Macro | Description | Output example | +|-------|-------------|----------------| +| `$__dateFilter(columnName)` | Filters by the panel date range using the given column. | `date >= toDate('2022-10-21') AND date <= toDate('2022-10-23')` | +| `$__timeFilter(columnName)` | Filters by the panel time range (seconds). | `time >= toDateTime(1415792726) AND time <= toDateTime(1447328726)` | +| `$__timeFilter_ms(columnName)` | Filters by the panel time range (milliseconds). | `time >= fromUnixTimestamp64Milli(1415792726123) AND time <= fromUnixTimestamp64Milli(1447328726456)` | +| `$__dateTimeFilter(dateColumn, timeColumn)` | Combines date and time filters for separate Date and DateTime columns. | `$__dateFilter(dateColumn) AND $__timeFilter(timeColumn)` | +| `$__fromTime_` | Start of the panel time range as `DateTime`. | `toDateTime(1415792726)` | +| `$__toTime_` | End of the panel time range as `DateTime`. | `toDateTime(1447328726)` | +| `$__fromTime_ms_` | Start of the panel time range as `DateTime64(3)`. | `fromUnixTimestamp64Milli(1415792726123)` | +| `$__toTime_ms_` | End of the panel time range as `DateTime64(3)`. | `fromUnixTimestamp64Milli(1447328726456)` | +| `$__interval_s_` | Panel interval in seconds. | `20` | +| `$__timeInterval(columnName)` | Interval from panel time range (seconds), for grouping. | `toStartOfInterval(toDateTime(column), INTERVAL 20 second)` | +| `$__timeInterval_ms(columnName)` | Interval from panel time range (milliseconds), for grouping. | `toStartOfInterval(toDateTime64(column, 3), INTERVAL 20 millisecond)` | +| `$__conditionalAll(condition, $templateVar)` | Uses the first parameter when the template variable does not select all values; otherwise `1=1`. | `condition` or `1=1` | + +You can also use brace notation `{}` when the macro parameter must contain a query or other expression. + +## Ad hoc filters + +Ad hoc filters let you add key/value filters that are applied to queries that use the ClickHouse data source. You choose filter values from a drop-down in the dashboard without editing the query. Ad hoc filters are supported only with **ClickHouse 22.7 or later**. For an overview, see [Grafana ad hoc filters](https://grafana.com/docs/grafana/latest/variables/variable-types/add-ad-hoc-filters/). + +By default, the ad hoc filter drop-down lists all tables and columns from the data source. If you set a default database in the data source settings, only tables from that database are used. To limit which tables or columns appear (for example, to avoid slow loads), add a dashboard variable of type **Constant** named `clickhouse_adhoc_query`. Set its value to one of: + +- A comma-separated list of databases +- A single database name +- `database.table` to show only columns for one table + +You can hide this variable from the dashboard; it is only used to scope the ad hoc filter options. + +## Use a query to populate ad hoc filters + +You can set `clickhouse_adhoc_query` to a **ClickHouse query** instead of a database or table name. The query results are used to populate the ad hoc filter’s selectable values. For example, set the variable value to: + +```sql +SELECT DISTINCT machine_name FROM mgbench.logs1 +``` + +Then the dashboard filter drop-down lists distinct `machine_name` values, and you can filter queries by the selected machine. + +## Map and JSON types (OpenTelemetry) + +Ad hoc filters work with Map and JSON types for OpenTelemetry data. **Map** is the default and turns merged labels into a filter. To use **JSON** syntax for the filter logic, add a dashboard variable of type **Constant** named `clickhouse_adhoc_use_json`. The variable’s value is ignored; it only needs to exist. + +## Apply ad hoc filters manually with `$__adHocFilters` + +By default, ad hoc filters are applied automatically by detecting the target table from your SQL. For queries that use CTEs, subqueries, or ClickHouse-specific syntax (for example `INTERVAL` or parameterized aggregate functions), automatic detection can fail. In those cases, use the `$__adHocFilters('table_name')` macro to specify where to apply the filters. + +The macro expands to the ClickHouse `additional_table_filters` setting with the currently active ad hoc filter conditions. Place it in the **SETTINGS** clause of your query. + +Example: + +```sql +SELECT * +FROM ( + SELECT * FROM my_complex_table + WHERE complicated_condition +) AS result +SETTINGS $__adHocFilters('my_complex_table') +``` + +When ad hoc filters are active (for example, `status = 'active'` and `region = 'us-west'`), the macro expands to: + +```sql +SETTINGS additional_table_filters={'my_complex_table': 'status = \'active\' AND region = \'us-west\''} +``` + +## Next steps + +- [ClickHouse template variables](/docs/plugins/grafana-clickhouse-datasource//template-variables/) — Use variables in dashboards and queries. +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/) — Connection and authentication options. diff --git a/docs/sources/template-variables.md b/docs/sources/template-variables.md new file mode 100644 index 000000000..718bc69f7 --- /dev/null +++ b/docs/sources/template-variables.md @@ -0,0 +1,171 @@ +--- +description: Use template variables with the ClickHouse data source to build dynamic dashboards +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source + - variables +menuTitle: Template variables +title: ClickHouse template variables +weight: 20 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# ClickHouse template variables + +Template variables let you parameterize your dashboards so you can change databases, tables, environments, or other values from a drop-down without editing each query. This makes dashboards more interactive, reusable, and easier to maintain. + +For an introduction to templating and variable types, see [Templating](https://grafana.com/docs/grafana/latest/variables/) and [Add variables](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/). + +## Before you begin + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/). +- Ensure your ClickHouse user has read access to the databases and tables used in your variable queries (for example, `system.databases`, `system.tables`, and your application tables). + +## Create a query variable + +To create a template variable that gets its values from ClickHouse: + +1. Open the dashboard where you want to add the variable. +2. Click **Dashboard settings** (gear icon) in the top navigation. +3. Select **Variables** in the left menu. +4. Click **Add variable**. +5. Enter a **Name** for your variable (for example, `database`, `table`, or `environment`). Use a name you can reference in queries (for example, `$database`). +6. In the **Type** drop-down, select **Query**. +7. In the **Data source** drop-down, select your ClickHouse data source. +8. In the **Query** field, enter a ClickHouse SQL query that returns the values for the variable. The query can return one column (same label and value) or two columns (value and label). See [How query results become variable options](#how-query-results-become-variable-options) and [Query examples](#query-examples). +9. Click **Run query** to preview the variable options. +10. Set **Refresh** to control when the variable options update (see [Variable refresh options](#variable-refresh-options)). +11. Configure **Multi-value** or **Include All option** if needed. +12. Click **Apply** to save the variable. + +## How query results become variable options + +The plugin uses the query result to build the variable’s drop-down options: + +- **Single column:** Each row becomes one option. Both the displayed label and the value used in queries are that column’s value. +- **Two columns:** The first column is used as the **value** (for example, an id or key). The second column is used as the **text** (the label shown in the drop-down). + +**Example — single column (database names as label and value):** + +```sql +SELECT name FROM system.databases WHERE name NOT IN ('INFORMATION_SCHEMA', 'information_schema') +``` + +You can omit the `WHERE` clause if your ClickHouse instance does not have those databases (for example, a standalone ClickHouse server typically only has `default` and `system`). + +**Example — two columns (id as value, name as label):** + +```sql +SELECT id, name FROM my_app.environments +``` + +Here, the drop-down shows `name`, and queries receive `id` when the variable is used. + +## Variable syntax in queries + +Use variables in your ClickHouse queries by referencing them with `$varname` or `${varname}`. Grafana replaces the variable with the selected value (or values) before the query is sent to ClickHouse. + +For full syntax and options, see [Variable syntax](https://grafana.com/docs/grafana/latest/variables/syntax/). + +## Format options for safe SQL + +To avoid SQL syntax or injection issues, use a **format** when the variable is used inside a string or list: + +- **singlequote** — Wraps each value in single quotes and escapes single quotes inside the value. Use this for string literals and `IN` lists in ClickHouse. + +**Example — filter by one database:** + +```sql +SELECT * FROM system.tables WHERE database = ${database:singlequote} +``` + +**Example — filter by multiple databases (multi-value variable):** + +```sql +SELECT * FROM system.tables WHERE database IN (${database:singlequote}) +``` + +Without `:singlequote`, multi-value variables are comma-separated and can produce invalid SQL. Other formats (for example, **regex** or **pipe**) are described in [Variable syntax](https://grafana.com/docs/grafana/latest/variables/syntax/). + +## Cascading (dependent) variables + +You can make one variable depend on another by using the first variable in the second variable’s query. When the user changes the first variable, the second variable’s options update automatically. + +**Example: database → table** + +1. Create a variable named `database` with query: + + ```sql + SELECT name FROM system.databases WHERE name NOT IN ('INFORMATION_SCHEMA', 'information_schema') + ``` + +2. Create a variable named `table` with query: + + ```sql + SELECT name FROM system.tables WHERE database = ${database:singlequote} + ``` + +When you change the selected database, the table drop-down refreshes with tables from that database. + +## Using the "All" option with `$__conditionalAll` + +If you enable **Include All option** for a variable, selecting **All** sets the variable value to `$__all`. A condition like `WHERE database IN (${database:singlequote})` may not behave as intended when **All** is selected. + +Use the **$__conditionalAll(condition, $variable)** macro so that: + +- When the variable is **not** "All", the macro is replaced by the condition (for example, `database IN ('db1', 'db2')`). +- When the variable **is** "All", the macro is replaced by `1=1` (no filter). + +**Example:** + +```sql +SELECT count() FROM system.tables +WHERE $__conditionalAll(database IN (${database:singlequote}), $database) +``` + +When the user selects one or more databases, the condition filters by those databases. When the user selects **All**, the condition becomes `1=1` and all databases are included. + +See the [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) Macros section for the full list of macros. + +## Query examples + +| Use case | Query | +|----------|--------| +| List databases | `SELECT name FROM system.databases` (optionally add `WHERE name NOT IN ('INFORMATION_SCHEMA', 'information_schema')` to exclude those if present) | +| List tables (for chosen database) | `SELECT name FROM system.tables WHERE database = ${database:singlequote}` | +| List columns (for chosen database and table) | `SELECT name FROM system.columns WHERE database = ${database:singlequote} AND table = ${table:singlequote}` | +| Distinct values for a column | `SELECT DISTINCT environment FROM my_app.events ORDER BY environment` | + +Replace `my_app.events` and column names with your own database, table, and columns. + +## Variable refresh options + +Set **Refresh** to control when the variable’s query runs and the options update: + +| Option | Behavior | +|--------|----------| +| **On dashboard load** | Options refresh when the dashboard loads. Use for data that changes infrequently (for example, database or table lists). | +| **On time range change** | Options refresh when the dashboard time range changes. Use only if your variable query depends on the time range. | + +For dashboards with many variables or heavy variable queries, **On dashboard load** is usually sufficient and avoids unnecessary load. + +## Multi-value variables + +When **Multi-value** is enabled, users can select more than one value. The selected values are typically comma-separated when substituted into the query. Use the **singlequote** format so each value is correctly quoted in SQL: + +```sql +WHERE database IN (${database:singlequote}) +``` + +When one variable’s query uses another variable (cascading variables) and that other variable is multi-value, Grafana often substitutes only the first selected value. Ensure that the first value alone still gives a valid and useful list for the dependent variable. + +## Next steps + +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//query-editor/) — Macros (including `$__timeFilter`, `$__conditionalAll`) and building queries. +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/) — Connection and authentication options. +- [Troubleshoot ClickHouse data source issues](/docs/plugins/grafana-clickhouse-datasource//troubleshooting/) — Common errors and solutions. diff --git a/docs/sources/troubleshooting.md b/docs/sources/troubleshooting.md new file mode 100644 index 000000000..7254b2ab0 --- /dev/null +++ b/docs/sources/troubleshooting.md @@ -0,0 +1,456 @@ +--- +description: Solutions for common errors when using the ClickHouse data source +labels: +products: + - Grafana Cloud + - Grafana OSS + - Grafana Enterprise +keywords: + - data source + - troubleshooting +menuTitle: Troubleshooting +title: Troubleshoot ClickHouse data source issues +weight: 40 +version: 0.1 +last_reviewed: 2026-02-11 +--- + +# Troubleshoot ClickHouse data source issues + +This guide provides solutions for common errors you may encounter when configuring or using the ClickHouse data source for Grafana. + +## Connection Errors + +## Invalid Server Host + +**Error message:** "invalid server host. Either empty or not set" + +**Cause:** The server host field is empty or was not configured in the data source settings. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Verify that the **Server** field contains a valid hostname or IP address. +3. Ensure there are no leading or trailing spaces in the host value. + +--- + +## Invalid Port + +**Error message:** "invalid port" + +**Cause:** The port number is missing, empty, or contains an invalid value. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Verify that the **Port** field contains a valid port number. +3. Use the default port `9000` for native protocol or `8123` for HTTP protocol. +4. Ensure the port value is a number without any special characters. + +--- + +## Failed to Create ClickHouse Client + +**Error message:** "failed to create ClickHouse client" + +**Cause:** The plugin was unable to establish a connection to the ClickHouse server. This can occur due to network issues, incorrect credentials, firewall rules, or server unavailability. + +**Solution:** + +1. Verify that the ClickHouse server is running and accessible from the Grafana server. +2. Check that the hostname, port, username, and password are correct. +3. Ensure there are no firewall rules blocking the connection. +4. If using TLS/SSL, verify that the certificates are correctly configured. +5. Test the connection using `clickhouse-client` from the Grafana server to isolate network issues. +6. Check the Grafana server logs for more detailed error information. + +--- + +## Connection Timeout + +**Error message:** "connection timeout" or "the operation was cancelled before starting: context deadline exceeded" + +**Cause:** The connection to the ClickHouse server timed out before it could be established. + +**Solution:** + +1. Verify that the ClickHouse server is reachable from the Grafana server. +2. Increase the **Dial Timeout** value in the data source settings (default is 10 seconds). +3. Check for network latency or connectivity issues between Grafana and ClickHouse. +4. Ensure no firewall or security group is blocking the connection. + +--- + +## Operation Cancelled During Execution + +**Error message:** "the operation was cancelled during execution: context deadline exceeded" + +**Cause:** The query or connection operation exceeded the configured timeout while running. + +**Solution:** + +1. Increase the **Query Timeout** value in the data source settings (default is 60 seconds). +2. Optimize your query to reduce execution time. +3. Check if the ClickHouse server is under heavy load. +4. Consider adding appropriate indexes to your ClickHouse tables. + +--- + +## Authentication Errors + +## Invalid Username + +**Error message:** "username is either empty or not set" + +**Cause:** The username field is empty or was not configured in the data source settings. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Enter a valid ClickHouse username in the **Username** field. +3. Verify that the user exists in ClickHouse and has appropriate permissions. + +--- + +## Invalid Password + +**Error message:** "password is either empty or not set" + +**Cause:** The password field is empty or was not configured when authentication requires a password. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Enter the correct password in the **Password** field. +3. Verify that the password matches the one configured in ClickHouse for the specified user. + +--- + +## TLS/SSL Certificate Errors + +## Invalid CA Certificate + +**Error message:** "failed to parse TLS CA PEM certificate" + +**Cause:** The CA certificate provided is not in valid PEM format or is corrupted. + +**Solution:** + +1. Verify that the CA certificate is in PEM format (begins with `-----BEGIN CERTIFICATE-----`). +2. Ensure the entire certificate content is copied, including the BEGIN and END markers. +3. Check that there are no extra spaces or line breaks in the certificate. +4. Regenerate the CA certificate if it may be corrupted. + +--- + +## Invalid Client Certificate + +**Error message:** "tls: failed to find any PEM data in certificate input" + +**Cause:** The client certificate or key provided is not in valid PEM format or is empty. + +**Solution:** + +1. Verify that both the client certificate and client key are in PEM format. +2. Ensure the client certificate begins with `-----BEGIN CERTIFICATE-----`. +3. Ensure the client key begins with `-----BEGIN PRIVATE KEY-----` or `-----BEGIN RSA PRIVATE KEY-----`. +4. Check that the certificate and key match (were generated together). +5. Verify that the entire content is copied without truncation. + +--- + +## Protocol Errors + +## Invalid Protocol + +**Error message:** "protocol is invalid, use native or http" + +**Cause:** An unsupported protocol was specified in the data source configuration. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Set the **Protocol** to either `native` or `http`. +3. Use `native` (port 9000) for better performance or `http` (port 8123) for HTTP-based connectivity. + +--- + +## Configuration Parsing Errors + +## Invalid JSON Configuration + +**Error message:** "could not parse json" + +**Cause:** The data source configuration contains invalid JSON syntax. + +**Solution:** + +1. If you are provisioning the data source via YAML/JSON files, validate the JSON syntax. +2. Use a JSON validator to check for syntax errors. +3. Ensure all string values are properly quoted and special characters are escaped. +4. Re-save the data source configuration through the Grafana UI. + +--- + +## Could Not Parse Configuration Values + +**Error messages:** +- "could not parse port value" +- "could not parse secure value" +- "could not parse tlsSkipVerify value" +- "could not parse tlsAuth value" +- "could not parse tlsAuthWithCACert value" +- "could not parse forwardGrafanaHeaders value" + +**Cause:** A configuration value could not be converted to the expected type (boolean or number). + +**Solution:** + +1. Verify that boolean values are set to `true` or `false` (without quotes in JSON, or as strings `"true"`/`"false"`). +2. Verify that numeric values like port are valid integers. +3. Re-configure the data source through the Grafana UI to ensure proper value types. + +--- + +## Invalid Timeout Values + +**Error messages:** +- "invalid timeout: [value]" +- "invalid query timeout: [value]" + +**Cause:** The dial timeout or query timeout value is not a valid integer. + +**Solution:** + +1. Open the data source configuration in Grafana. +2. Ensure the **Dial Timeout** and **Query Timeout** fields contain valid integer values (in seconds). +3. Remove any non-numeric characters from the timeout fields. + +--- + +## Query Errors + +## ClickHouse Database Exception + +**Error message:** "DB::Exception: [error details]" + +**Cause:** ClickHouse returned an error while executing your query. Common causes include syntax errors, missing tables, or permission issues. + +**Solution:** + +1. Review the error message for specific details about the issue. +2. Verify your SQL syntax is correct for ClickHouse. +3. Check that referenced tables and columns exist. +4. Ensure the configured user has permission to access the requested data. +5. Test the query directly in `clickhouse-client` to isolate the issue. + +--- + +## Macro Argument Count Error + +**Error messages:** +- "$__timeFilter: expected 1 argument, received [n]" +- "$__timeFilter_ms: expected 1 argument, received [n]" +- "$__dateFilter: expected 1 argument, received [n]" +- "$__dateTimeFilter: expected 2 arguments, received [n]" +- "$__timeInterval: expected 1 argument, received [n]" +- "$__timeInterval_ms: expected 1 argument, received [n]" + +**Cause:** A Grafana macro was used with the wrong number of arguments. + +**Solution:** + +1. Check the macro syntax in your query: + - `$__timeFilter(column)` - requires 1 argument (the time column) + - `$__timeFilter_ms(column)` - requires 1 argument (the time column for millisecond precision) + - `$__dateFilter(column)` - requires 1 argument (the date column) + - `$__dateTimeFilter(dateColumn, timeColumn)` - requires 2 arguments + - `$__timeInterval(column)` - requires 1 argument (the time column) + - `$__timeInterval_ms(column)` - requires 1 argument (the time column) +2. Ensure arguments are separated by commas if multiple are required. + +--- + +## SQL Parse Error + +**Error message:** Parse error with line and column information + +**Cause:** The SQL query contains syntax errors that could not be parsed. + +**Solution:** + +1. Review the error message for the specific line and column where the error occurred. +2. Check for common SQL syntax issues like missing commas, unmatched parentheses, or incorrect keywords. +3. Verify that ClickHouse-specific syntax is being used correctly. +4. Use the Query Builder mode to construct queries if you're unfamiliar with ClickHouse SQL. + +--- + +## Ad Hoc Filter Errors + +## Unable to Apply Ad Hoc Filters + +**Error message:** "Unable to apply ad hoc filters. Upgrade ClickHouse to >=22.7 or remove ad hoc filters for the dashboard." + +**Cause:** Ad hoc filters require ClickHouse version 22.7 or higher, which introduced the `additional_table_filters` setting. + +**Solution:** + +1. Upgrade your ClickHouse server to version 22.7 or higher. +2. Alternatively, remove the ad hoc filter variable from your dashboard. +3. Use regular template variables as a workaround if upgrading is not possible. + +--- + +## Failed to Get Table from Ad Hoc Query + +**Error message:** "Failed to get table from adhoc query." + +**Cause:** The plugin could not determine which table to apply ad hoc filters to from the query. + +**Solution:** + +1. Ensure your query contains a valid `FROM` clause with a table name. +2. If using a complex query with subqueries or CTEs, consider using a simpler query structure. +3. Explicitly specify the table in the ad hoc filter configuration variable. + +--- + +## Invalid Ad Hoc Filter + +**Error message:** "Invalid adhoc filter will be ignored: [filter details]" + +**Cause:** An ad hoc filter is missing required fields (key, operator, or value). + +**Solution:** + +1. Verify that all ad hoc filters have a key (column name), operator, and value specified. +2. Check the filter configuration in your dashboard variables. +3. Remove any incomplete filter definitions. + +--- + +## Log Context Errors + +## Missing Query for Log Context + +**Error message:** "Missing query for log context" + +**Cause:** The log context feature was invoked without a valid query. + +**Solution:** + +1. Ensure you're using the log context feature with a valid logs query. +2. Verify that the query is using the Builder editor mode. + +--- + +## Missing Log Context Options + +**Error message:** "Missing log context options for query" + +**Cause:** Required options for log context (direction or limit) are missing. + +**Solution:** + +1. This is typically an internal error. Try refreshing the page. +2. If the issue persists, report it as a bug. + +--- + +## Log Context Only Works for Builder Queries + +**Error message:** "Log context feature only works for builder queries" + +**Cause:** The log context feature was invoked on a SQL editor query instead of a Builder query. + +**Solution:** + +1. Switch your query from **SQL Editor** mode to **Builder** mode. +2. Configure your logs query using the Builder interface. + +--- + +## Missing Time Column for Log Context + +**Error message:** "Missing time column for log context" + +**Cause:** The query doesn't have a time column configured, which is required for log context. + +**Solution:** + +1. In the Query Builder, ensure you've selected a column with the **Time** hint. +2. Verify that your logs table has a timestamp column and it's properly configured. + +--- + +## Unable to Match Log Context Columns + +**Error message:** "Unable to match any log context columns" + +**Cause:** None of the configured context columns could be matched from the current log row's data frame. + +**Solution:** + +1. Verify that the **Context Columns** are configured in the data source settings under Logs configuration. +2. Ensure the configured context column names match the actual column names in your query results. +3. Check that the context columns are included in your SELECT statement. + +--- + +## Proxy Errors + +## Unable to Cast SOCKS Proxy Dialer + +**Error message:** "unable to cast SOCKS proxy dialer to context proxy dialer" + +**Cause:** There was an issue initializing the secure SOCKS proxy connection for Private Data Connect (PDC). + +**Solution:** + +1. Verify your PDC configuration is correct. +2. Check that the SOCKS proxy is properly configured and accessible. +3. Review Grafana server logs for more detailed error information. +4. Ensure your Grafana version supports the PDC feature. + +--- + +## Header Parsing Errors + +## Couldn't Parse Message as Args + +**Error message:** "Couldn't parse message as args" + +**Cause:** The plugin could not parse the forwarded headers message. + +**Solution:** + +1. This is typically an internal error related to header forwarding. +2. Check if **Forward Grafana Headers** is enabled and configured correctly. +3. Review Grafana server logs for more details. + +--- + +## Couldn't Parse Grafana HTTP Headers + +**Error message:** "Couldn't parse grafana HTTP headers" + +**Cause:** The Grafana HTTP headers could not be parsed from the request. + +**Solution:** + +1. Verify the header forwarding configuration. +2. Check that custom HTTP headers are properly formatted. +3. Review the data source configuration for any malformed header entries. + +--- + +## Getting More Help + +If you continue to experience issues after trying the solutions in this guide: + +1. Check the [ClickHouse documentation](https://clickhouse.com/docs) for database-specific issues. +2. Review the Grafana server logs for more detailed error messages. +3. Search or create issues in the [grafana-clickhouse-datasource GitHub repository](https://github.com/grafana/clickhouse-datasource). +4. Visit the [Grafana Community forums](https://community.grafana.com/) for community support.