Zeek/Suricata Alerts without matching PCAP

[MHeezy]

I'm trying to understand if some behavior I'm seeing is normal or if something is wrong with my Malcolm install. When I filter for traffic going to a web-server, I see plenty of Zeek (mainly just 'conn') and Suricata alerts. However, if I want to drill down on an alert and look at the pcap, its not there. I figure if I use the filter "event.provider == arkime", there should be corresponding sessions with pcap for every Zeek/Suricata alert, but that's not the case. However, I still see pcap being generated in the Malcolm/pcap directory.

I am running Malcolm 24.11.0.

[mmguero]

Check out the documentation on correlating Zeek logs and Suricata alerts. The communityId field is a common field across all data source types, and can be used to identify the corresponding Arkime session (and by extension PCAP) for a given Zeek or suricata log. So the process would be identify the Zeek log you're interested in, then click the communityId field and OR that into your search. This video is kind of old, but this part is still applicable and you could watch at this timestamp where I talk about it.

This is, of course, assuming you are getting arkime sessions at all, correct? When you filter on the event provider as Arkime, you do see some arkime sessions, you're just having an issue correlating them? Or are we not seeing any arkime sessions at all, for any time period?

Note that if you're capturing live traffic on a standalone Malcolm instance, the Arkime sessions do get rolled over on a delay (based on PCAP size and/or age, whichever is first) rather than being instantaneous like the Zeek/Suricata logs. That's described more here. So if you're looking at a Zeek log from the last few minutes, you won't necessarily see the Arkime session until up to 10 minutes have passed. That's only with a Malcolm session doing its own capture, and isn't an issue when using a dedicated sensor running Hedgehog Linux.

[MHeezy]

Thanks for the info/resources! However, the issue is that I'm seeing tons of zeek/suricata alerts with no corresponding arkime/pcap. For example, if I use the filter "ip.dst == 192.168.1.250", I'm seeing 220 results in the past 4 days. However, if I add to the filter " && event.provider == arkime", there is only 1 result. If I go to a zeek/suricata alert and copy the communityId string to filter on, I do not see any related arkime data.

So I'm thinking, ok, arkime pcap ingest must be broken, but when I inspect the contents of "Malcolm/pcap/processed/", there are a bunch of pcaps timestamped every day. Additionally, the "status" script shows that all the containers are up and healthy. Also note that I restarted Malcolm on Friday to see if it would fix itself but alas, no bueno.

[mmguero]

One difference that might cause the 220 vs. 1 log is that Arkime combines all of the data from one session into one arkime session record, while Zeek/Suricata split it out. So, for example, if you have a Zeek http.log that was generated with 220 lines that will correspond to 220 entries for event.provider:zeek and event.dataset:http in the Arkime sessions UI, but the corresponding event.provider:arkime will only have one line.

In Arkime, if you go up to the top and click the Files tab, that wills show you all of the files that Arkime has processed Arkime sessions from. If you see the PCAP files you expect there, then the data from those PCAP files is in Arkime.

[mmguero]

Any more luck investigating this?

[MHeezy]

I've been troubleshooting an alternate issue that I think may be connected to this.

On another instance of Malcolm, I wasn't seeing traffic I was expecting to see. Ultimately, the issue seemed to be related to the promiscuity of linux bridges/interfaces in proxmox. The issue was resolved when I did PCIE device pass through directly to the Malcolm VM.

I'm going to replicate this fix on the Malcolm instance re: this thread and report back with results.

[mmguero]

VM networking issues can be tricky for sure. Good luck.