Zeek logs not showing in OpenSearch

[gormami]

I have a Malcolm instance, and a probe reporting zeek logs from capture. Nothing is showing in the zeek logs of the OpenSearch dashboards.

• I have confirmed the zeek logs are in JSON format
• I have confirmed there is a good data flow from the filebeats process on the probe to port 5044 of the Malcolm instance via tcpdump monitoring.
• The probe is Ubuntu 24.04, not Hedgehog, in case that matters.
• There are good logs in the filebeat machine indicating the harvesters are working on the zeek log files.
• I've confirmed the pipelines are started on the logstash machine (malcolm-zeek and others), no non-running pipelines
• I looked at the logstash pipelines, and noted that zeek-parse filters for tags, and I couldn't find where those tags are input, so I added a tag "_filebeat_zeek" to the filebeat.yml on the probe machine with no results.

It really looks like the logs are being dropped in logstash, but I'm not sure why. Am I missing something silly?

In general, how is the _filebeat_zeek tag supposed to be imposed on the inputs?

[mmguero]

Looks like you've done a lot to debug, which is appreciated! I think you are on the right track with it expecting the tags, but here are some things we can do to confirm.

First, for completeness, you're right that Malcolm's hedgehog sensor is applying a tag to its zeek logs via filebeat (see here). The list of tags Malcolm Zeek logstash pipeline it accepts are here and include:

• _filebeat_zeek
• _filebeat_zeek_live
• _filebeat_zeek_upload
• _filebeat_zeek_hedgehog_live
• _filebeat_zeek_malcolm_live
• _filebeat_zeek_malcolm_upload

For your use case, it shouldn't really matter which you use.

Let's check with logstash on the Malcolm side to see if we can determine if they're being dropped or not:

$ docker compose exec logstash curl -sSL -XGET http://localhost:9600/_node/stats/pipelines | jq -r '.. | .filters? // empty | .[] | objects | select (.events.in > 0) | [.id, .events.in, .events.out, .events.duration_in_millis] | join (";")' | sort -n -t ';' -k2,3 | grep -P "(mutate_filebeat_zeek_forward_noop|drop_not_filebeat_zeek)"
drop_not_filebeat_zeek;9;0;0
mutate_filebeat_zeek_forward_noop;2780;2780;92

That command grabs the pipeline stats from the running logstash instance and gives us the number of records in, the number of records out, and the execution milliseconds for each filter in the pipeline. We filter that for these:

• mutate_filebeat_zeek_forward_noop - these events are passed through to the zeek pipeline
• drop_not_filebeat_zeek - these events are dropped (they may be intended for another pipeline, like the suricata pipeline, or they may just be missing the correct tag)

If you see any going through the "forward" filter, then the problem is later down the line. If you see the "in" count on the "drop" filter increasing as you send events, then the problem is that it's missing a tag. If you don't see the count going up on either then it's more of a connection/plumbing issue we need to debug.

[mikegorman-nf]

Somewhere along the line, I broke the auth for OpenSearch, so I reinstalled rather than chase, as I've been poking quite a bit and didn't take good enough notes.

Running the command above shows the messages are passing through.

mutate_filebeat_zeek_forward_noop;5953;5953;274

I've attached the full pipelines output.
pipeline_stats.txt

The one item that seems potentially concerning is drop_dissect_error, 457,0,6. I can see the if statement in the pipeline, but I'm not sure where the definition of [zeek_cols] is, or if that is actually a text object. I'm not familiar with logstash pipelines.

filter {

  if ([zeek_cols]) {
    # remove unset (-) or "(empty)" top-level field values
    ruby {
      id => "ruby_zeek_remove_empty_values"
      path => "/usr/share/logstash/malcolm-ruby/compact_event.rb"
    }
  } else {
    drop { id => "drop_dissect_error" }
  }

  # ECS - "zeek" -> event.provider
  if (![event][provider]) { mutate { id => "mutate_add_field_event_provider_zeek"
                                     add_field => { "[event][provider]" => "zeek" } } }

  if (![host][name]) and ([agent][hostname]) {
    mutate { id => "mutate_zeek_add_field_host_name_agent_hostname"
             add_field => { "[host][name]" => "%{[agent][hostname]}" } }
  }

  # rename the zeek child array to match the log type
  mutate { id => "mutate_rename_zeek_log_type"
           rename => { "[zeek_cols]" => "[zeek][%{[log_source]}]" } }

[mmguero]

During startup certain plugins are a bit noisy when they're initializing, and the one initializing .opendistro-anomaly-detectors is one of them, that can be ignored.

As far as other points in the pipeline we could check, yeah there are a few:

• you could just grep for drop and see if there are any that jump out there. There will be some, of course (for example, all the zeek events are immediately dropped by the suricata pipeline and vice-versa) but that might give us an idea.
• you could check the input and output filters on each of the pipelines (found here). Each filter, with very few exceptions, has an id field that is returned by the command I showed you. For example, grep for fingerprint which is towards the end of the zeek pipeline, or grep for one of the things in the enrichment pipeline which comes next, maybe mutate_add_field_env_logstash_oui_lookup or mutate_remove_field_beats_useless. All the way on the output end we could check output_opensearch_malcolm or mutate_final_tags_remove.

[gormami]

SO I didn't find exactly what the problem was, but I did grab the filebeat.yml from the malcolm repo and use that one, replacing the variables as necessary, and it is working fine now. It must have been something in the tagging structure. Thank you for the assistance.