troubleshooting Hedgehog forwarding Zeek/Suricata logs to Malcolm

[riley611]

Hello, I am a new Malcolm user.
I have an issue where my Hedgehog sensor is forwarding logs as expected, filtering out itself via BPF, and the Arkime session 3 loading in Malcolm. Zeek and Suricata both running on Hedgehog. My issue is that I am not seeing Zeek Logs nor Suricata populating in my Dashboard. I am using a Security Onion instance as my Elastic Search engine, and installation defaults.
I uploaded a PCAP sample from Fortiphyd and was able to see the Zeek logs populate as expected, but not from Arkime.

[hackortreat]

I am having the same issues. I was running version 24.08.0 and started having suricata logging issues. I decided to upgrade to the newest version, and I am still running into the exact same problem. I'm also new to Malcolm so I am hoping this is easy to fix...

[mmguero]

@riley611 if you're seeing Arkime traffic from the hedgehog but not zeek/suricata, and you've checked with /opt/sensor/sensor_ctl/status that the processes for them are running, my guess is it's some sort of problem in the configuration of forwarding

Arkime connects to Elasticsearch directly, but suricata/zeek logs are sent to Malcolm's logstash instance via a filebeat forwarder. Check out the section called filebeat: Zeek and Suricata log forwarding in the forwarding documentation and double-check those settings. Also, check and make sure the appropriate autostart settings are turned on.

If that doesn't lead to something, we could check the log files in /opt/sensor/sensor_ctl/log/ on the hedgehog and try to get an indication there about what's not working.

[riley611]

Thanks, @mmguero, I've configured the forwarder for arkime-capture, filebeat, and miscbeat. No SSL because I just need this to work first. I do see that the sensor status script is showing all zeekctl to be running but also shows an error. investigating /opt/sensor/sensor_ctl/log/ shows "Running via"/opt/zeek/bin/zeekctl" ( processes) . . .
I also see many empty log files.

[riley611]

Updating /etc/filebeat.yml
output.logstash:
hosts: ["malcolm-server:5044"]

[mmguero]

/etc/filebeat.yml isn't used at all, all of the files that have to do with forwarding are under /opt/sensor/sensor_ctl, and they are all configured by the configure_capture script as described here. Modifying anything in /etc/ or anywhere outside of that directory will have no effect.

[mmguero]

Check running processes with /opt/sensor/sensor_ctl/status and report back which ones are running here?

[riley611]

Logstash connection refused over port 5044

@mmguero , investigating /sensor_ctl now

[mmguero]

Perhaps you need to check your Malcolm configuration too. Malcolm's software firewall is closed off by default, part of the configuration process asks Should Malcolm accept logs and metrics from a Hedgehog Linux sensor or other forwarder? which will open those ports.

I'm pretty confident that if you read and follow the installation example, particularly the steps for Malcolm and Hedgehog capture and forwarding carefully, or watch the corresponding videos for the Malcolm configuration and Hedgehog configuration you will resolve your problems.

[mmguero]

From the ISO they can be installed/configured/run completely without internet access. Just select "No" for the configuration options that have to do with rule updates (updating suricata rules, updating file scanning rules, etc.)

[riley611]

@mmguero, when I initially boot both Malcolm and Hedgehog, I am met with the option to run a Live amd instance. What benefit do these options have over a ground-up installation?

[riley611]

@mmguero , The question "Format non-OS drive(s) for artifact storage" -- does that imply that I should create two virtual hard disks to each installation? I always select Yes to this question, but would it have an impact on the presence of a secondary hard disk?

[riley611]

@mmguero , "Enable index management policies (ILM/ISM) " thread then explains that Hot and Warm indices must be configured in OpenSearch. I'm not sure if this is associated with my particular issue--dashboards not populating. My question is if this configuration of index management is a red-herring to my original question

[mmguero]

1. Live option is not really useful for Malcolm (mainly because of the disk space required to load up the docker images on first boot, which would preclude a live boot scenario) and is more just a convenience for system rescue or something like that. For Hedgehog it's useful for a one-off, but if you're going to have a persistent installation I'd choose Virtual Machine Single Partition Quick Install for a VM.
2. What that means is that if you have multiple disks available to the system, selecting Y will use them, in this order:
   • smallest disk that's at least big enough for the OS gets formatted as the OS disk
   • largest other disk (if present) is used for OpenSearch
   • 2nd-largest other disk (if present) is used for PCAP
   • 3rd-larges other disk (if present) is used for Zeek/Suricata logs and the like
   • This is not required, but it does allow you to segment your storage off for different things.
3. The only thing this question has to do with is Arkime's usage of ILM/ISM. Unless you're specifically wanting to do that, you can probably just select N for this question.

[riley611]

@mmguero, Full rebuild using the latest iso, 25.01, both Hedgehog and Malcolm - Quick Install. I am still not populating dashboards. I have Hedgehog forwarding everything, arkime-capture, filebeat, miscbeat, htpDate for time sync to the Malcolm server. Autostart for all except netsniff and tcpdump. Arkime is showing traffic over 502, Modbus. Default log spaces set for Zeek and Suricata. Logs populating, rules loaded. Zeek set to carve and preserve all traffic. No BPF. OpenSearch and LogStash running locally on Malcolm. Air-gapped installation. no SSL. HTTPS configured.
What could I be missing?

[mmguero]

Just to double-check and make sure the data's not actually there (vs. something wrong with the display itself):

• In arkime, where you have those arkime sessions visible, click the 👁️ icon to the right of the search bar and select Zeek Logs, then Suricata Logs. I assume at that point there are no logs shown in arkime sessions as well, correct?

Assuming they're not, let's check things on the forwarding end. We'll work backwards from Malcolm.

• First, let's check open ports. Run this command on Malcolm. It might look a little different from mine (different version, but something like this:

malcolm-filebeat-1            ghcr.io/idaholab/malcolm/filebeat-oss:25.01.0        "/usr/bin/tini -- /u…"   filebeat            36 minutes ago   Up 35 minutes (healthy)   0.0.0.0:514->514/tcp, 0.0.0.0:5045->5045/tcp, 0.0.0.0:514->514/udp
malcolm-logstash-1            ghcr.io/idaholab/malcolm/logstash-oss:25.01.0        "/usr/bin/tini -- /u…"   logstash            36 minutes ago   Up 35 minutes (healthy)   9001/tcp, 0.0.0.0:5044->5044/tcp, 9600/tcp
malcolm-nginx-proxy-1         ghcr.io/idaholab/malcolm/nginx-proxy:25.01.0         "/sbin/tini -- /usr/…"   nginx-proxy         36 minutes ago   Up 35 minutes (healthy)   0.0.0.0:443->443/tcp, 0.0.0.0:9200->9200/tcp
malcolm-upload-1              ghcr.io/idaholab/malcolm/file-upload:25.01.0         "/usr/bin/tini -- /u…"   upload              36 minutes ago   Up 35 minutes (healthy)   80/tcp, 0.0.0.0:8022->22/tcp

The main thing we're looking for is 0.0.0.0:5044->5044/tcp and 0.0.0.0:5045->5045/tcp in the logstash and filebeat containers, respectively. You could also grep for 0.0.0.0 in your docker-compose.yml which would give us an idea:

$ grep 0.0.0.0 docker-compose.yml
    - 0.0.0.0:5044:5044/tcp
    - 0.0.0.0:5045:5045/tcp
    - 0.0.0.0:514:514/tcp
    - 0.0.0.0:514:514/udp
    - 0.0.0.0:8022:22/tcp
    - 0.0.0.0:443:443:tcp
    - 0.0.0.0:9200:9200/tcp

Another thing you can check for the same thing would be on the hedgehog side: I've got a bash function called portping that can be used like this:

$ portping 172.16.0.20 5044
OPEN
$ portping 172.16.0.20 5045
OPEN

We don't need to check 9200 because we know that's already open, as Arkime is using that one for its sessions, which you're seeing.

So, depending on what we see there, we might have something to investigate. If not, let's go on to the Hedgehog side.

• Forwarding settings on Hedgehog.

I can see what you have enabled with the autostarts. The only thing I would recommend is that you turn off clamav:clamav-updates and fluent-bit:thermal, the first because you're airgapped and can't download antivirus signature updates from the internet, and the second because you're on a VM and generally I don't think VMs are able to get temperatures from thermal sensors. Nothing that will break anything, but something you just might as well turn them off since they'll just silently be failing in the background.

Everything looks like it's running here, but let's sanity check by running pidof zeek and pidof suricata on Hedgehog just to make sure. You should get, probably, something like 3 or more PIDs returned from Zeek, and probably 1 PID from Suricata.

Now, to see if there are actually zeek/suricata logs being created on Hedgehog, let's check a few things:

There's a convenience function on Hedgehog called sensorwatch. When you run it, it will show you, at the top of that output, the latest log files being created:

So in my case you can see that suricata is generating a log file (eve-20250212_174055.json) and so is Zeek (conn.log, notice.log, etc.). You should see something similar.

If you see those log files (eve-*.json and *.log) are there, then Zeek and Suricata are generating logs, they're just not getting fowrarded. If they're NOT generating logs, then we need to debug the capture instead of the forwarding.

Next, let's check forwarding. The one we really care about is filebeat, so let's look at /opt/sensor/sensor_ctl/log/filebeat-*. My guess is we will see something in /opt/sensor/sensor_ctl/log/filebeat-stderr--* that will hopefully give us an idea of what's not working.

[riley611]

Thanks, @mmguero,
Correct, from Arkime viewer, Zeek and Suricata logs are not populating either.
Logstash shows 0.0.0.0:5044->5044/tcp and Filebeat shows 0.0.0.0:5045->5045/tcp
Both portping commands are returning closed
Acknowledged regarding clamav:clamav-updates and fluent-bit:thermal
Running sensorwatch, I do get the .json and .log populating, but also occasionally return a "cannot access '/home/sensor/zeek_logs/extract_files/HTTP-string~~.xml"

And finally, filebeat-stderr--supervisor-string.log outputs multiple messages such as:
"Not loading modules. Module director not found: /opt/sensor/sensor_ctl/filebeat/module"
"Attempting to reconnect to backoff(async(tcp://10.105.1.60:5044)): dial10.105.1.60:5044 tcp :connect: no route to host" with ## reconnect attempt(s)

and lastly: "Filebeat is unable to load the ingest piplelines for the configured modules because ElasticSearch output is not configured/enabled" -- I am no longer using ElasticSearch, only the local OpenSearch so I think that the last message may be a red-herring

[mmguero]

If the portping commands are returning closed from the Hedgehog to Malcolm, then there's the problem. I assume portping x.x.x.x 9200 returns OPEN (again, it must, or you wouldn't have arkime sessions).

If you see those 0.0.0.0:5044->5044/tcp and 0.0.0.0:5045->5045/tcp but portping says CLOSED, then it can only be one of two things:

1. Malcolm's software firewall (sudo ufw status should show you that)
2. You have some firewall or something in between the boxes blocking those ports

[riley611]

Getting closed portping returns on everything, despite ufw rule additions. Our network firewall permits all ports/IP on my 105 and 106 VLANs.

[riley611]

RunningChangelog:
updated /opt/sensor/sensor_ctl/filebeat/filebeat.yml changed from false to true for logging.metrtics.enabled

[riley611]

@mmguero so with the wizards populating those yaml files, when I config with install.py, where does Malcolm store the input for "Should Malcolm accept logs and metrics from a Hedgehog Linux sensor or other forwarder?" and could this solve my issue, because I think this is the crucial step that may be flagging my filebeat/logstash hiccup

[riley611]

@mmguero does anything look funky with my /malcolm/config/filebeat.env

[mmguero]

@mmguero so with the wizards populating those yaml files, when I config with install.py, where does Malcolm store the input for "Should Malcolm accept logs and metrics from a Hedgehog Linux sensor or other forwarder?" and could this solve my issue, because I think this is the crucial step that may be flagging my filebeat/logstash hiccup

Some of that goes into docker-compose.yml in the ports section for filebeat and logstash, and some of it goes into the .env files. That .env file looks fine to me.

Is the filebeat-stderr file we talked about earlier still saying "no route to host"? If it is, that's where we need to focus our energy. No route to host means it can't even route the IP, vs. like a connection refused which would be an issue on the Malcolm side.

[riley611]

@mmguero , I am back in filebeat-stderr--

Starting with the error at the top, file line 155, fileset/modules.go, Not loading modules, Module directory not found /opt/sensor/sensor_ctl/filebeat/module

is there something in the install.py config sequence that should create the Module?

And there is no longer an error message regarding "no route to host"

[mmguero]

That modules.go error is something that can be ignored, I have the same thing on mine. Here's what my filebeat stderr looks like on a working fowarding instance:

Grepping for Harvester and piping to jq to pretty-print, note how it says the various source_file entries like http.log, etc.

grep Harvester /opt/sensor/sensor_ctl/log/filebeat-stderr* | jq

Now grepping for tcp, see how it's connected to my logstash on port 5044:

grep tcp /opt/sensor/sensor_ctl/log/filebeat-stderr* | jq

Do you have messages like that? And you're still not getting logs into Malcolm?

If you're not, would you please run this command and screenshot or share the output:

jq -r '.message' < /opt/sensor/sensor_ctl/log/filebeat-stderr* | uniq

[riley611]

@mmguero , Hey, so I did get it working....on my non-air gapped instance that I set up on a completely different server. I manually changed the DHCP address of the server to a static one and when I reboot my system, I could not longer see any traffic at all. What is the correct procedure for assigning a static IP to Malcolm?

[riley611]

I see there are some python scripts that must be run
I also had Malcolm collecting on its own interface with the single NIC, and I saw my coveted Zeek & Suricata logs. I inherited a DHCP address, and so when I manually updated it, I think the capture interface held on to the DHCP IP and now that's why I can no longer see any traffic because I gave Malcolm a new IP

[mmguero]

Cool, great progress. Malcolm's and Hedgehog's network configuration Utilities are a little bit different, so that can be a bit confusing. For Hedgehog, you run configure-interfaces from the command line or click the corresponding Icon in the toolbar for it. For Malcolm, you'll see a network icon up in the system tray on the right, you can right-click there and select 'Edit connections...' and go from there.

[riley611]

@mmguero That's what I've been doing but I wanted to be sure. Because I notice when I do set a static for Malcolm, after a restart, I will end up losing Malcolm's own capture interface traffic

[riley611]

@mmguero, so I have Malcolm running its own sensor node locally, before I begin forward. I log into the Malcolm's own server instance via localhost on the left, and then from my host, I access the web server. Dashboard showing disparaging information. How is this possible. 10.0.20.71 is DHCP

[riley611]

Usually if you have two different browsers connecting to the same exact malcolm server and they don't see the exact same data, one of them has their time set incorrectly.

You can verify this by checking the time on the boxes, and/or set your query time frame to cover, say, some period in the past to some period in the future:

@mmguero How would you recommend correcting the time for Malcolm?

[riley611]

https://github.com/cisagov/Malcolm/blob/main/docs/time-sync.md
Attempting this

[mmguero]

Yeah you can set up time sync for Malcolm and Hedgehog to use either NTP or htpdate if NTP is not available. At a minimum you could set hedgehog's htpdate configuration to point to it's malcolm server, so at least then they're the same. Or, of course, if you have an internal NTP server you can use that, or some other HTTP server that reports the date/time in its response headers.

[riley611]

@mmguero I'm going to continue to troubleshoot, but my remote browser is seeing time as the Malcolm's VM, (about 7 hours behind), while the Malcolm server itself is roughly two hours behind local time. Running an NTP server from our DC but something is not synching. I don't think this solely a Malcolm issue, but I do find it odd that the Malcolm server and Malcolm ISO it booted from do not synch time by default

[riley611]

I had to install chrony and update the .conf. I also enabled NTP on my hypervisor. I'm 3 minutes behind but that's a lot better than 7 hours.