-
Notifications
You must be signed in to change notification settings - Fork 5k
[8.19](backport #47154) beatconverter: Ignore disabled elasticsearch output #47559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
## Proposed commit message Similar to the logstash output: https://github.com/elastic/beats/blob/52c5045638c8116550dba017bb455802577a5513/libbeat/otelbeat/beatconverter/beatconverter.go#L82-L85 Fixes #46113 ## How to test this PR locally filebeat.yaml: ```yaml filebeat.inputs: - type: filestream id: my-filestream-id enabled: true paths: - /path/to/file.json output.elasticsearch: enabled: false hosts: ["localhost:9200"] ``` compile: `cd x-pack/filebeat/ && mage buildOtel` test with: 1. `x-pack/filebeat/filebeat -c filebeat.yaml -e` 2. `x-pack/filebeat/filebeat otel --config filebeat-task.yaml` ## Related issues - Closes #46113 ## Logs ### Before ``` 2025-10-16T11:39:52.274+0200 DEBUG elasticsearch/config_otel.go:222 Failed to get queue size: missing field accessing 'queue' 2025-10-16T11:39:52.298+0200 info instance/beat.go:178 Home path: [.] Config path: [.] Data path: [./data] Logs path: [./logs] {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.298+0200 info instance/beat.go:186 Beat ID: 440437c7-3a3f-409a-ba2e-5aa89d0ab790 {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.324+0200 info publisher pipeline/module.go:105 Beat name: laptop {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.325+0200 error modules fileset/modules.go:157 Not loading modules. Module directory not found: module {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.325+0200 warn beater/filebeat.go:174 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning. {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.325+0200 info builders/extension.go:50 Development component. May change in the future. {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "beatsauth", "otelcol.component.kind": "extension"} 2025-10-16T11:39:52.330+0200 info [email protected]/service.go:222 Starting otel... {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "Version": "9.3.0", "NumCPU": 12} 2025-10-16T11:39:52.330+0200 info extensions/extensions.go:41 Starting extensions... {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}} 2025-10-16T11:39:52.330+0200 info extensions/extensions.go:45 Extension is starting... {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "beatsauth", "otelcol.component.kind": "extension"} 2025-10-16T11:39:52.331+0200 info extensions/extensions.go:62 Extension started. {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "beatsauth", "otelcol.component.kind": "extension"} 2025-10-16T11:39:52.331+0200 info [email protected]/service.go:245 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}} 2025-10-16T11:39:52.331+0200 info fbreceiver/receiver.go:27 starting filebeat receiver {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.332+0200 info filebeat memlog/store.go:134 Finished loading transaction log file for '/home/orestis/src/beats/data/registry/filebeat'. Active transaction id=0 {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "store": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.333+0200 warn beater/filebeat.go:406 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning. {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.333+0200 info registrar registrar/registrar.go:103 States Loaded from registrar: 0 {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.333+0200 info crawler beater/crawler.go:76 Loading Inputs: 1 {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.334+0200 info crawler beater/crawler.go:148 Starting input (ID: 1771212043283374986) {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.334+0200 info crawler beater/crawler.go:111 Loading and starting Inputs completed. Enabled inputs: 1 {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.334+0200 info input.filestream compat/compat.go:141 Input 'filestream' starting {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "id": "my-filestream-id", "ecs.version": "1.6.0"} 2025-10-16T11:39:52.335+0200 info input.filestream.metric_registry inputmon/input.go:182 registering {"resource": {"service.instance.id": "1f0db8b8-b116-48be-8dfc-bb2aff2578e6", "service.name": "otel", "service.version": "9.3.0"}, "otelcol.component.id": "filebeatreceiver", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "id": "my-filestream-id", "registry_id": "my-filestream-id", "input_id": "my-filestream-id", "input_type": "filestream", "ecs.version": "1.6.0"} ``` ### After ``` x-pack/filebeat/filebeat otel --config filebeat-task.yaml Error: invalid configuration: no exporter configuration specified in config service::pipelines::logs: must have at least one exporter Usage: filebeat otel [flags] filebeat otel [command] Available Commands: inspect Run this command to inspect the OTel configuration translated from the Beats config Flags: --config string path to filebeat config file (default "filebeat-otel.yml") -h, --help help for otel Global Flags: -E, --E setting=value Configuration overwrite -M, --M setting=value Module configuration overwrite -c, --c string Configuration file, relative to path.config (default "filebeat.yml") -d, --d string Enable certain debug selectors -e, --e Log to stderr and disable syslog/file output --environment environmentVar set environment being ran in (default default) --path.config string Configuration path --path.data string Data path --path.home string Home path --path.logs string Logs path --plugin pluginList Load additional plugins --strict.perms Strict permission checking on config files (default true) -v, --v Log at INFO level Use "filebeat otel [command] --help" for more information about a command. ``` (cherry picked from commit f1267b8) # Conflicts: # libbeat/otelbeat/beatconverter/beatconverter_test.go
|
Cherry-pick of f1267b8 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
|
This pull request has not been merged yet. Could you please review and merge it @orestisfl? 🙏 |
1 similar comment
|
This pull request has not been merged yet. Could you please review and merge it @orestisfl? 🙏 |
|
not sure why |
Proposed commit message
Similar to the logstash output:
beats/libbeat/otelbeat/beatconverter/beatconverter.go
Lines 82 to 85 in 52c5045
Fixes #46113
Checklist
./changelog/fragmentsusing the changelog tool.How to test this PR locally
filebeat.yaml:
compile:
cd x-pack/filebeat/ && mage buildOteltest with:
x-pack/filebeat/filebeat -c filebeat.yaml -ex-pack/filebeat/filebeat otel --config filebeat-task.yamlRelated issues
Logs
Before
After
This is an automatic backport of pull request #47154 done by [Mergify](https://mergify.com).