diff --git a/docs/sources/_index.md b/docs/sources/_index.md new file mode 100644 index 00000000..74256ed1 --- /dev/null +++ b/docs/sources/_index.md @@ -0,0 +1,35 @@ +--- +description: This document introduces the ClickHouse data source +labels: +products: + - Grafana Cloud +keywords: + - data source +menuTitle: ClickHouse data source +title: ClickHouse data source +weight: 10 +version: 0.1 +--- + +# Official ClickHouse data source for Grafana + +The ClickHouse data source plugin allows you to query and visualize ClickHouse data in Grafana. + +- [Configure the ClickHouse data source](/docs/plugins/grafana-clickhouse-datasource//configure/) +- [ClickHouse query editor](/docs/plugins/grafana-clickhouse-datasource//editor/) +- [ClickHouse templates and variables](/docs/plugins/grafana-clickhouse-datasource//templates-and-variables/) + +## Version compatibility + +Users on Grafana `v9.x` and higher of Grafana can use `v4`. +Users on Grafana `v8.x` are encouraged to continue using `v2.2.0` of the plugin. + +\* _As of 2.0 this plugin will only support ad hoc filters when using ClickHouse 22.7+_ + +## Get the most out of the ClickHouse data source + +After installing and configuring ClickHouse you can: + +- Add [Annotations](https://grafana.com/docs/grafana/latest/dashboards/annotations/). +- Add [Transformations](https://grafana.com/docs/grafana/latest/panels/transformations/) +- Set up [Alerting](https://grafana.com/docs/grafana/latest/alerting/) diff --git a/docs/sources/ad-hoc-filters.md b/docs/sources/ad-hoc-filters.md new file mode 100644 index 00000000..61198d50 --- /dev/null +++ b/docs/sources/ad-hoc-filters.md @@ -0,0 +1,80 @@ +--- +description: This document outlines ad hoc filter options for the ClickHouse data source +labels: +products: + - Grafana Cloud +keywords: + - data source +menuTitle: Ad hoc filter for the ClickHouse data source +title: ClickHouse ad hoc filters +weight: 20 +version: 0.1 +--- + +### Ad Hoc Filters + +Ad hoc filters are only supported with version 22.7+ of ClickHouse. + +Ad hoc filters allow you to add key/value filters that are automatically added +to all metric queries that use the specified data source, without being +explicitly used in queries. + +By default, Ad Hoc filters will be populated with all Tables and Columns. If +you have a default database defined in the Datasource settings, all Tables from +that database will be used to populate the filters. As this could be +slow/expensive, you can introduce a second variable to allow limiting the +Ad Hoc filters. It should be a `constant` type named `clickhouse_adhoc_query` +and can contain: a comma delimited list of databases, just one database, or a +database.table combination to show only columns for a single table. + +Ad Hoc filters also work with the Map and JSON types for OTel data. +Map is the default, and will automatically convert the merged labels output into a usable filter. +To have the filter logic use JSON syntax, add a dashboard variable with a `constant` type called `clickhouse_adhoc_use_json` (the variable's `value` is ignored, it just has to be present). + +For more information on Ad Hoc filters, check the [Grafana +docs](https://grafana.com/docs/grafana/latest/variables/variable-types/add-ad-hoc-filters/) + +#### Using a query for Ad Hoc filters + +The second `clickhouse_adhoc_query` also allows any valid ClickHouse query. The +query results will be used to populate your ad-hoc filter's selectable filters. +You may choose to hide this variable from view as it serves no further purpose. + +For example, if `clickhouse_adhoc_query` is set to `SELECT DISTINCT +machine_name FROM mgbench.logs1` you would be able to select which machine +names are filtered for in the dashboard. + +#### Manual Ad Hoc Filter Placement with `$__adHocFilters` + +By default, ad-hoc filters are automatically applied to queries by detecting the +target table using SQL parsing. However, for queries that use CTEs or ClickHouse-specific +syntax like `INTERVAL` or aggregate functions with parameters, the automatic +detection may fail. In these cases, you can manually specify where to apply +ad-hoc filters using the `$__adHocFilters('table_name')` macro. + +This macro expands to the ClickHouse `additional_table_filters` setting with the +currently active ad-hoc filters. It should be placed in the `SETTINGS` clause of +your query. + +Example: + +```sql +SELECT * +FROM ( + SELECT * FROM my_complex_table + WHERE complicated_condition +) AS result +SETTINGS $__adHocFilters('my_complex_table') +``` + +When ad-hoc filters are active (e.g., `status = 'active'` and `region = 'us-west'`), +this expands to: + +```sql +SELECT * +FROM ( + SELECT * FROM my_complex_table + WHERE complicated_condition +) AS result +SETTINGS additional_table_filters={'my_complex_table': 'status = \'active\' AND region = \'us-west\''} +``` diff --git a/docs/sources/configure.md b/docs/sources/configure.md new file mode 100644 index 00000000..04106f6a --- /dev/null +++ b/docs/sources/configure.md @@ -0,0 +1,102 @@ +--- +description: This document outlines configuration options for the ClickHouse data source +labels: +products: + - Grafana Cloud +keywords: + - data source +menuTitle: Configure the ClickHouse data source +title: ClickHouse data source +weight: 20 +version: 0.1 +--- + +## Configure the ClickHouse data source + +### ClickHouse user for the data source + +Set up an ClickHouse user account with [readonly](https://clickhouse.com/docs/en/operations/settings/permissions-for-queries#settings_readonly) permission and access to +databases and tables you want to query. +Please note that Grafana does not validate that queries are safe. Queries can contain any SQL statement. +For example, statements like `ALTER TABLE system.users DELETE WHERE name='sadUser'` +and `DROP TABLE sadTable;` would be executed. + +To configure a readonly user, follow these steps: + +1. Create a `readonly` user profile following the [Creating Users and Roles in ClickHouse](https://clickhouse.com/docs/en/operations/access-rights) guide. +2. Ensure the `readonly` user has enough permission to modify the `max_execution_time` setting required by the underlying [clickhouse-go client](https://github.com/ClickHouse/clickhouse-go/). +3. If you're using a public ClickHouse instance, it's not recommended to set `readonly=2` in the `readonly` profile. Instead, leave `readonly=1` and set the constraint type of `max_execution_time` to [changeable_in_readonly](https://clickhouse.com/docs/en/operations/settings/constraints-on-settings) to allow modification of this setting. + +### ClickHouse protocol support + +The plugin supports both `Native` (default) and `HTTP` transport protocols. +This can be enabled in the configuration via the `protocol` configuration parameter. +Both protocols exchange data with ClickHouse using optimized native format. + +Note that the default ports for `HTTP/S` and `Native` differ: + +- HTTP - 8123 +- HTTPS - 8443 +- Native - 9000 +- Native with TLS - 9440 + +### Manual configuration via UI + +Once the plugin is installed on your Grafana instance, follow +[these instructions](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/) +to add a new ClickHouse data source, and enter configuration options. + +### With a configuration file + +It is possible to configure data sources using configuration files with Grafana’s provisioning system. +To read about how it works, refer to +[Provisioning Grafana data sources](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources). + +Here are some provisioning examples for this data source using basic authentication: + +```yaml +apiVersion: 1 +datasources: + - name: ClickHouse + type: grafana-clickhouse-datasource + jsonData: + defaultDatabase: database + port: 9000 + host: localhost + username: username + tlsSkipVerify: false + # tlsAuth: + # tlsAuthWithCACert: + # secure: + # dialTimeout: + # queryTimeout: + # protocol: + # defaultTable: + # httpHeaders: + # - name: X-Example-Header + # secure: false + # value: + # - name: Authorization + # secure: true + # logs: + # defaultDatabase: + # defaultTable: + # otelEnabled: + # otelVersion: + # timeColumn: + # ...Column: + # traces: + # defaultDatabase: + # defaultTable: + # otelEnabled: + # otelVersion: + # durationUnit: + # traceIdColumn: + # ...Column: + secureJsonData: + password: password + # tlsCACert: + # tlsClientCert: + # tlsClientKey: + # secureHttpHeaders.Authorization: +``` diff --git a/docs/sources/editor.md b/docs/sources/editor.md new file mode 100644 index 00000000..ea6f80b6 --- /dev/null +++ b/docs/sources/editor.md @@ -0,0 +1,117 @@ +--- +description: This document describes the ClickHouse query editor +labels: +products: + - Grafana Cloud +keywords: + - data source +menuTitle: ClickHouse query editor +title: ClickHouse query editor +weight: 30 +version: 0.1 +--- + +## Building queries + +Queries can be built using the raw SQL editor or the query builder. +Queries can contain macros which simplify syntax and allow for +dynamic SQL generation. + +### Time series + +Time series visualization options are selectable after adding a `datetime` +field type to your query. This field will be used as the timestamp. You can +select time series visualizations using the visualization options. Grafana +interprets timestamp rows without explicit time zone as UTC. Any column except +`time` is treated as a value column. + +#### Multi-line time series + +To create multi-line time series, the query must return at least 3 fields in +the following order: + +- field 1: `datetime` field with an alias of `time` +- field 2: value to group by +- field 3+: the metric values + +For example: + +```sql +SELECT log_time AS time, machine_group, avg(disk_free) AS avg_disk_free +FROM mgbench.logs1 +GROUP BY machine_group, log_time +ORDER BY log_time +``` + +### Tables + +Table visualizations will always be available for any valid ClickHouse query. + +### Visualizing logs with the Logs Panel + +To use the Logs panel your query must return a timestamp and string values. To default to the logs visualization in Explore mode, set the timestamp alias to _log_time_. + +For example: + +```sql +SELECT log_time AS log_time, machine_group, toString(avg(disk_free)) AS avg_disk_free +FROM logs1 +GROUP BY machine_group, log_time +ORDER BY log_time +``` + +To force rendering as logs, in absence of a `log_time` column, set the Format to `Logs` (available from 2.2.0). + +### Visualizing traces with the Traces Panel + +Ensure your data meets the [requirements of the traces panel](https://grafana.com/docs/grafana/latest/explore/trace-integration/#data-api). This applies if using the visualization or Explore view. + +Set the Format to `Trace` when constructing the query (available from 2.2.0). + +If using the [Open Telemetry Collector and ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter), the following query produces the required column names (these are case sensitive): + +```sql +SELECT + TraceId AS traceID, + SpanId AS spanID, + SpanName AS operationName, + ParentSpanId AS parentSpanID, + ServiceName AS serviceName, + Duration / 1000000 AS duration, + Timestamp AS startTime, + arrayMap(key -> map('key', key, 'value', SpanAttributes[key]), mapKeys(SpanAttributes)) AS tags, + arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags, + if(StatusCode IN ('Error', 'STATUS_CODE_ERROR'), 2, 0) AS statusCode +FROM otel.otel_traces +WHERE TraceId = '61d489320c01243966700e172ab37081' +ORDER BY startTime ASC +``` + +### Macros + +To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros. + +Here is an example of a query with a macro that will use Grafana's time filter: + +```sql +SELECT date_time, data_stuff +FROM test_data +WHERE $__timeFilter(date_time) +``` + +| Macro | Description | Output example | +| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| _$\_\_dateFilter(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the date range of the panel | `date >= toDate('2022-10-21') AND date <= toDate('2022-10-23')` | +| _$\_\_timeFilter(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in seconds | `time >= toDateTime(1415792726) AND time <= toDateTime(1447328726)` | +| _$\_\_timeFilter_ms(columnName)_ | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in milliseconds | `time >= fromUnixTimestamp64Milli(1415792726123) AND time <= fromUnixTimestamp64Milli(1447328726456)` | +| _$\_\_dateTimeFilter(dateColumn, timeColumn)_ | Shorthand that combines $**dateFilter() AND $**timeFilter() using separate Date and DateTime columns. | `$__dateFilter(dateColumn) AND $__timeFilter(timeColumn)` | +| _$\_\_fromTime_ | Replaced by the starting time of the range of the panel casted to `DateTime` | `toDateTime(1415792726)` | +| _$\_\_toTime_ | Replaced by the ending time of the range of the panel casted to `DateTime` | `toDateTime(1447328726)` | +| _$\_\_fromTime_ms_ | Replaced by the starting time of the range of the panel casted to `DateTime64(3)` | `fromUnixTimestamp64Milli(1415792726123)` | +| _$\_\_toTime_ms_ | Replaced by the ending time of the range of the panel casted to `DateTime64(3)` | `fromUnixTimestamp64Milli(1447328726456)` | +| _$\_\_interval_s_ | Replaced by the interval in seconds | `20` | +| _$\_\_timeInterval(columnName)_ | Replaced by a function calculating the interval based on window size in seconds, useful when grouping | `toStartOfInterval(toDateTime(column), INTERVAL 20 second)` | +| _$\_\_timeInterval_ms(columnName)_ | Replaced by a function calculating the interval based on window size in milliseconds, useful when grouping | `toStartOfInterval(toDateTime64(column, 3), INTERVAL 20 millisecond)` | +| _$\_\_conditionalAll(condition, $templateVar)_ | Replaced by the first parameter when the template variable in the second parameter does not select every value. Replaced by the 1=1 when the template variable selects every value. | `condition` or `1=1` | + +The plugin also supports notation using braces {}. Use this notation when queries are needed inside parameters. diff --git a/docs/sources/template-and-variables.md b/docs/sources/template-and-variables.md new file mode 100644 index 00000000..22071c66 --- /dev/null +++ b/docs/sources/template-and-variables.md @@ -0,0 +1,22 @@ +--- +description: This document describes using templates and variables with the ClickHouse data source +labels: +products: + - Grafana Cloud +keywords: + - data source +menuTitle: ClickHouse templates and variables +title: ClickHouse templates and variables +weight: 20 +version: 0.1 +--- + +### Templates and variables + +To add a new ClickHouse query variable, refer to [Add a query +variable](https://grafana.com/docs/grafana/latest/variables/variable-types/add-query-variable/). + +After creating a variable, you can use it in your ClickHouse queries by using +[Variable syntax](https://grafana.com/docs/grafana/latest/variables/syntax/). +For more information about variables, refer to [Templates and +variables](https://grafana.com/docs/grafana/latest/variables/).