Skip to content

update kafka datasource documentation for new batch parameters #781

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 60 additions & 2 deletions crowdsec-docs/docs/log_processor/data_sources/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,30 @@ labels:
type: nginx
```

Adding a batch configuration:

```yaml
source: kafka
brokers:
- "localhost:9093"
topic: "my-topic"
timeout: 5
tls:
insecure_skip_verify: true
client_cert: /path/kafkaClient.certificate.pem
client_key: /path/kafkaClient.key
ca_cert: /path/ca.crt
labels:
type: nginx
batch:
min_bytes: 1024 # 1KB
max_bytes: 1048576 # 1MB
max_wait: 5s
queue_size: 1000
commit_interval: 1s
```


:::info
The reader will always start from the latest offset.
:::
Expand All @@ -60,7 +84,7 @@ Required.

The consumer group id to use.

Cannot be used with `partition`.
Cannot be used with `partition`.

:::warning
It is highly recommended to set this value, or crowdsec will only read logs from the 1st partition of the topic.
Expand All @@ -72,6 +96,12 @@ Read messages from the given partition. Mostly useful for debugging.

Cannot be used with `group_id`.

### `timeout`

Maximum time to wait for new messages before returning an empty read.

Default: 5

### `tls.insecure_skip_verify`

To disable security checks on the certificate.
Expand All @@ -96,6 +126,35 @@ The CA certificate path.

Optional, when you want to enable TLS with client certificate.

### `batch.min_bytes`

Minimum number of bytes to accumulate in the fetch buffer before returning results.

Default: 1

### `batch.max_bytes`

Maximum number of bytes to fetch in one go.

Default: 1048576 (1MB)

### `batch.max_wait`

Maximum time to wait before returning a fetch, even if `batch.min_bytes` isn’t reached.

Default: 250ms

### `batch.queue_size`

Maximum number of messages to buffer internally before processing.

Default: 100

### `batch.commit_interval`

Time interval between automatic commits of consumer offsets.

Default: 0 (commit after every fetch)

### `source`

Expand All @@ -104,4 +163,3 @@ Must be `kafka`
## DSN and command-line

This datasource does not support acquisition from the command line.