Skip to content

Commit c66ffd1

Browse files
gingerwizardgingerwizard
authored
Merge pull request #4702 from ClickHouse/otel-collector-advanced-config
extending otel config
2 parents baea0db + 009ccce commit c66ffd1

File tree

2 files changed

+149
-67
lines changed

2 files changed

+149
-67
lines changed

‎docs/use-cases/observability/clickstack/example-datasets/local-data.md‎

Lines changed: 58 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -10,57 +10,36 @@ keywords: ['clickstack', 'example data', 'sample dataset', 'logs', 'observabilit
1010
---
1111

1212
import Image from '@theme/IdealImage';
13-
import hyperdx from '@site/static/images/use-cases/observability/hyperdx-1.png';
1413
import hyperdx_20 from '@site/static/images/use-cases/observability/hyperdx-20.png';
15-
import hyperdx_3 from '@site/static/images/use-cases/observability/hyperdx-3.png';
16-
import hyperdx_4 from '@site/static/images/use-cases/observability/hyperdx-4.png';
1714
import hyperdx_21 from '@site/static/images/use-cases/observability/hyperdx-21.png';
1815
import hyperdx_22 from '@site/static/images/use-cases/observability/hyperdx-22.png';
1916
import hyperdx_23 from '@site/static/images/use-cases/observability/hyperdx-23.png';
20-
import copy_api_key from '@site/static/images/use-cases/observability/copy_api_key.png';
2117

22-
This getting started guide allows you collect local logs and metrics from your system, sending them to ClickStack for visualization and analysis.
18+
This getting started guide allows you to collect local logs and metrics from your system, sending them to ClickStack for visualization and analysis.
2319

2420
**This example works on OSX and Linux systems only**
2521

26-
The following example assumes you have started ClickStack using the [instructions for the all-in-one image](/use-cases/observability/clickstack/getting-started) and connected to the [local ClickHouse instance](/use-cases/observability/clickstack/getting-started#complete-connection-credentials) or a [ClickHouse Cloud instance](/use-cases/observability/clickstack/getting-started#create-a-cloud-connection).
27-
2822
:::note HyperDX in ClickHouse Cloud
29-
This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only minor adjustments to the flow as noted. If using HyperDX in ClickHouse Cloud, users will require an Open Telemetry collector to be running locally as described in the [getting started guide for this deployment model](/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud).
23+
This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only minor adjustments to the flow as noted. If using HyperDX in ClickHouse Cloud, users will require an OpenTelemetry collector to be running locally as described in the [getting started guide for this deployment model](/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud).
3024
:::
3125

3226
<VerticalStepper>
3327

34-
## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui}
35-
36-
Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu.
28+
## Create a custom OpenTelemetry configuration {#create-otel-configuration}
3729

38-
## Copy ingestion API key {#copy-ingestion-api-key}
39-
40-
:::note HyperDX in ClickHouse Cloud
41-
This step is not required if using HyperDX in ClickHouse Cloud.
42-
:::
43-
44-
Navigate to [`Team Settings`](http://localhost:8080/team) and copy the `Ingestion API Key` from the `API Keys` section. This API key ensures data ingestion through the OpenTelemetry collector is secure.
45-
46-
<Image img={copy_api_key} alt="Copy API key" size="lg"/>
47-
48-
## Create a local OpenTelemetry configuration {#create-otel-configuration}
49-
50-
Create a `otel-local-file-collector.yaml` file with the following content.
51-
52-
**Important**: Populate the value `<YOUR_INGESTION_API_KEY>` with your ingestion API key copied above (not required for HyperDX in ClickHouse Cloud).
30+
Create a `custom-local-config.yaml` file with the following content:
5331

5432
```yaml
5533
receivers:
5634
filelog:
5735
include:
58-
- /var/log/**/*.log # Linux
59-
- /var/log/syslog
60-
- /var/log/messages
61-
- /private/var/log/*.log # macOS
62-
- /tmp/all_events.log # macos - see below
63-
start_at: beginning # modify to collect new files only
36+
- /host/var/log/**/*.log # Linux logs from host
37+
- /host/var/log/syslog
38+
- /host/var/log/messages
39+
- /host/private/var/log/*.log # macOS logs from host
40+
start_at: beginning
41+
resource:
42+
service.name: "system-logs"
6443

6544
hostmetrics:
6645
collection_interval: 1s
@@ -96,29 +75,25 @@ receivers:
9675
network:
9776
processes:
9877

99-
exporters:
100-
otlp:
101-
endpoint: localhost:4317
102-
headers:
103-
authorization: <YOUR_INGESTION_API_KEY>
104-
tls:
105-
insecure: true
106-
sending_queue:
107-
enabled: true
108-
num_consumers: 10
109-
queue_size: 262144 # 262,144 items × ~8 KB per item ≈ 2 GB
110-
11178
service:
11279
pipelines:
113-
logs:
80+
logs/local:
11481
receivers: [filelog]
115-
exporters: [otlp]
116-
metrics:
82+
processors:
83+
- memory_limiter
84+
- batch
85+
exporters:
86+
- clickhouse
87+
metrics/hostmetrics:
11788
receivers: [hostmetrics]
118-
exporters: [otlp]
89+
processors:
90+
- memory_limiter
91+
- batch
92+
exporters:
93+
- clickhouse
11994
```
12095
121-
This configuration collects system logs and metric for OSX and Linux systems, sending the results to ClickStack via the OTLP endpoint on port 4317.
96+
This configuration collects system logs and metrics for OSX and Linux systems, sending the results to ClickStack. The configuration extends the ClickStack collector by adding new receivers and pipelines—you reference the existing `clickhouse` exporter and processors (`memory_limiter`, `batch`) that are already configured in the base ClickStack collector.
12297

12398
:::note Ingestion timestamps
12499
This configuration adjusts timestamps at ingest, assigning an updated time value to each event. Users should ideally [preprocess or parse timestamps](/use-cases/observability/clickstack/ingesting-data/otel-collector#processing-filtering-transforming-enriching) using OTel processors or operators in their log files to ensure accurate event time is retained.
@@ -130,33 +105,53 @@ To avoid this behavior, you can set the start position to `end` in the receiver
130105

131106
For more details on the OpenTelemetry (OTel) configuration structure, we recommend [the official guide](https://opentelemetry.io/docs/collector/configuration/).
132107

133-
:::note Detailed logs for OSX
134-
Users wanting more detailed logs on OSX can run the command `log stream --debug --style ndjson >> /tmp/all_events.log` before starting the collector below. This will capture detailed operating system logs to the file `/tmp/all_events.log`, already included in the above configuration.
135-
:::
136-
137-
## Start the collector {#start-the-collector}
108+
## Start ClickStack with custom configuration {#start-clickstack}
138109

139-
Run the following docker command to start an instance of the OTel collector.
110+
Run the following docker command to start the all-in-one container with your custom configuration:
140111

141112
```shell
142-
docker run --network=host --rm -it \
113+
docker run -d --name clickstack \
114+
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
143115
--user 0:0 \
144-
-v "$(pwd)/otel-local-file-collector.yaml":/etc/otel/config.yaml \
145-
-v /var/log:/var/log:ro \
146-
-v /private/var/log:/private/var/log:ro \
147-
otel/opentelemetry-collector-contrib:latest \
148-
--config /etc/otel/config.yaml
116+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
117+
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
118+
-v /var/log:/host/var/log:ro \
119+
-v /private/var/log:/host/private/var/log:ro \
120+
docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
149121
```
150122

151123
:::note Root user
152124
We run the collector as the root user to access all system logs—this is necessary to capture logs from protected paths on Linux-based systems. However, this approach is not recommended for production. In production environments, the OpenTelemetry Collector should be deployed as a local agent with only the minimal permissions required to access the intended log sources.
125+
126+
Note that we mount the host's `/var/log` to `/host/var/log` inside the container to avoid conflicts with the container's own log files.
153127
:::
154128

129+
If using HyperDX in ClickHouse Cloud with a standalone collector, use this command instead:
130+
131+
```shell
132+
docker run -d \
133+
-p 4317:4317 -p 4318:4318 \
134+
--user 0:0 \
135+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
136+
-e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} \
137+
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
138+
-e CLICKHOUSE_USER=${CLICKHOUSE_USER} \
139+
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
140+
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
141+
-v /var/log:/host/var/log:ro \
142+
-v /private/var/log:/host/private/var/log:ro \
143+
docker.hyperdx.io/hyperdx/hyperdx-otel-collector
144+
```
145+
155146
The collector will immediately begin collecting local system logs and metrics.
156147

148+
## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui}
149+
150+
Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu.
151+
157152
## Explore system logs {#explore-system-logs}
158153

159-
Navigate to the HyperDX UI. The search UI should be populated with local system logs. Expand the filters to select the `system.log`:
154+
The search UI should be populated with local system logs. Expand the filters to select the `system.log`:
160155

161156
<Image img={hyperdx_20} alt="HyperDX Local logs" size="lg"/>
162157

@@ -180,4 +175,4 @@ From the subsequent menu you can select `Percentage` from the `Output format` dr
180175

181176
<Image img={hyperdx_23} alt="Memory % of time" size="lg"/>
182177

183-
</VerticalStepper>
178+
</VerticalStepper>

‎docs/use-cases/observability/clickstack/ingesting-data/collector.md‎

Lines changed: 91 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -100,9 +100,96 @@ With Docker Compose, modify the collector configuration using the same environme
100100
101101
### Advanced configuration {#advanced-configuration}
102102
103-
Currently, the ClickStack distribution of the OTel collector does not support modification of its configuration file. If you need a more complex configuration e.g. [configuring TLS](#securing-the-collector), or modifying the batch size, we recommend copying and modifying the [default configuration](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml) and deploying your own version of the OTel collector using the ClickHouse exporter documented [here](/observability/integrating-opentelemetry#exporting-to-clickhouse) and [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/README.md#configuration-options).
103+
The ClickStack distribution of the OTel collector supports extending the base configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.
104104
105-
The default ClickStack configuration for the OpenTelemetry (OTel) collector can be found [here](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml).
105+
#### Extending the collector configuration {#extending-collector-config}
106+
107+
To add custom receivers, processors, or pipelines:
108+
109+
1. Create a custom configuration file with your additional configuration
110+
2. Mount the file at `/etc/otelcol-contrib/custom.config.yaml`
111+
3. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`
112+
113+
**Example custom configuration:**
114+
115+
```yaml
116+
receivers:
117+
# Collect logs from local files
118+
filelog:
119+
include:
120+
- /var/log/**/*.log
121+
- /var/log/syslog
122+
- /var/log/messages
123+
start_at: beginning
124+
125+
# Collect host system metrics
126+
hostmetrics:
127+
collection_interval: 30s
128+
scrapers:
129+
cpu:
130+
metrics:
131+
system.cpu.utilization:
132+
enabled: true
133+
memory:
134+
metrics:
135+
system.memory.utilization:
136+
enabled: true
137+
disk:
138+
network:
139+
filesystem:
140+
metrics:
141+
system.filesystem.utilization:
142+
enabled: true
143+
144+
service:
145+
pipelines:
146+
# Logs pipeline
147+
logs/host:
148+
receivers: [filelog]
149+
processors:
150+
- memory_limiter
151+
- transform
152+
- batch
153+
exporters:
154+
- clickhouse
155+
156+
# Metrics pipeline
157+
metrics/hostmetrics:
158+
receivers: [hostmetrics]
159+
processors:
160+
- memory_limiter
161+
- batch
162+
exporters:
163+
- clickhouse
164+
```
165+
166+
**Deploy with the all-in-one image:**
167+
```bash
168+
docker run -d --name clickstack \
169+
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
170+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
171+
-v "$(pwd)/custom-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
172+
docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
173+
```
174+
175+
**Deploy with the standalone collector:**
176+
```bash
177+
docker run -d \
178+
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
179+
-e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} \
180+
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
181+
-e CLICKHOUSE_USER=default \
182+
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
183+
-v "$(pwd)/custom-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
184+
-p 4317:4317 -p 4318:4318 \
185+
docker.hyperdx.io/hyperdx/hyperdx-otel-collector
186+
```
187+
188+
:::note
189+
You only define new receivers, processors, and pipelines in the custom config. The base processors (`memory_limiter`, `batch`) and exporters (`clickhouse`) are already defined—reference them by name. The custom configuration is merged with the base configuration and cannot override existing components.
190+
:::
191+
192+
For more complex configurations, refer to the [default ClickStack collector configuration](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml) and the [ClickHouse exporter documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/README.md#configuration-options).
106193

107194
#### Configuration structure {#configuration-structure}
108195

@@ -120,7 +207,7 @@ To further secure your deployment, we recommend:
120207

121208
- Configuring the collector to communicate with ClickHouse over HTTPS.
122209
- Create a dedicated user for ingestion with limited permissions - see below.
123-
- Enabling TLS for the OTLP endpoint, ensuring encrypted communication between SDKs/agents and the collector. **Currently, this requires users to deploy a default distribution of the collector and manage the configuration themselves**.
210+
- Enabling TLS for the OTLP endpoint, ensuring encrypted communication between SDKs/agents and the collector. This can be configured via [custom collector configuration](#extending-collector-config).
124211

125212
### Creating an ingestion user {#creating-an-ingestion-user}
126213

@@ -276,7 +363,7 @@ However, if you require high delivery guarantees or the ability to replay data (
276363
In this case, OTel agents can be configured to send data to Kafka via the [Kafka exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md). Gateway instances, in turn, consume messages using the [Kafka receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkareceiver/README.md). We recommend the Confluent and OTel documentation for further details.
277364

278365
:::note OTel collector configuration
279-
The ClickStack OpenTelemetry collector distribution cannot be used with Kafka as it requires a configuration modification. Users will need to deploy a default OTel collector using the ClickHouse exporter.
366+
The ClickStack OpenTelemetry collector distribution can be configured with Kafka using [custom collector configuration](#extending-collector-config).
280367
:::
281368

282369
## Estimating resources {#estimating-resources}

0 commit comments

Comments
 (0)