Alerts stop working after a day

[L0rdV0ld3m0rt]

Version

2.4.150

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

32

Storage for /

500GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have a new Security Onion install on a physical server. After initial setup everything works great for about a day and then it randomly stops generating alerts in the console. I initially thought it was because the server was under-spec'd, so I added more RAM and now it seems to still be sending alerts to our SIEM, but they're not showing in our console so I can't tune them. I'm not finding any useful errors, and when I have, the forums and KBs seem to be silent about how to remedy them, or offer a high level suggestion that I have no idea how to implement. Everything on the grid says it's healthy and running, but there is high capture loss in the application. Looking at the Bond interface there is a large amount of traffic but not seeing many drops comparatively. Elastic agent is healthy, have tried adjusting lsheap, have looked at hot/cold settings, soup and so-test have been run and are happy. Restarting the server does not resolve the problem, and the only fix I've found is a full reinstall of SO, and again it only works for a day. This was a straightforward out of the box standalone install, only change was moving the nsm partition to a secondary drive but have seen no issues there.

I'm sure there's got to be some some simple config fix for this, whether with queuing or with packet capture. I'm not a Security Onion expert, just trying to set up a simple server to alert on network traffic. Been spending months on this and at my wits end. Any help would be very greatly appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[L0rdV0ld3m0rt]

I've also tried increasing the settings mentioned here, though there's no direction on what they mean by "adjust". I can't tune the NIDS ruleset because I'm not getting alerts. Went into InfluxDB console and there's a "low pcap retention" alert, but can't find any info on how to correct this.

[L0rdV0ld3m0rt]

Tried removing a couple scanning servers with a BPF, but it broke suricata. I couldn't find any better documentation anywhere to validate those are configured properly. tried adjusting max pending packets and af-packet ring size, though the documentation doesn't specify a proper value for this. ended up having to reinstall SO yet again. Worked for about an hour before it stopped generating alerts again. Some of the documentation suggested issues with Tap/Span port upstream, but is only ingesting 1Gbps on a 10Gbps port, so curious what the suggestions would be to remedy this?

[cm-ops]

A couple of things to check. The screenshot above shows the PCAP BPF, what is in Suricata BPF? Do you see a current eve log in /nsm/suricata? Is there a current Suricata index in Elasticsearch?

Capture loss without packet loss is usually attributed to something upstream, TAP/SPAN traffic. https://docs.securityonion.net/en/2.4/zeek.html#packet-loss-and-capture-loss

[L0rdV0ld3m0rt]

Good morning, thanks for the reply. When I looked at the bond0 span interface I saw almost no dropped packets so assumed issue couldn't be upstream. I noticed though that the 10Gb nic speed was set at 1000 even though both ends were 10Gbps ports. Went into the switch config and found the SFP that was plugged into was only 1Gb :(. Just ordered a new SFP and hoping this will remedy the capture loss. Is the screenshot above the proper way to do the BPF filters? I couldn't find much information on web to validate (SO documentation didn't actually show an example), but what I had above ended up faulting the associated modules.

[L0rdV0ld3m0rt]

[L0rdV0ld3m0rt]

Unfortunately, still running into the same problem where we stop getting alerts after half a day or so. I think most of these problems are coming because the server we deployed is underspecc'ed. With new SPF it's now ingesting avg 3-5 GB/s and the dedicated NSM drive is only 500GB. Getting errors like "Search rejected due to missing shards" & "high disk watermark [85%] exceeded", and if I run commands to delete/clear elastisearch and nsm alerts immediately start coming back in (for a time). It seems pretty clear at this point that we're just eating up the storage too quickly and when it's full, it stops alerting. I've tried adjusting lsheap, and increasing shards to 4 since they were surpassing 50GB. Esheap is set to default of 33% of total RAM, index clean is set to enabled, nsm diskfreepercentage at 21, etc etc, most of the settings I'd think would automatically clear these are set to default.

Since I'm forwarding all alerts to a SIEM I don't really need retention other than long enough to review anything for tuning, but not quite sure how to do that. I thought about looking at ILM and just setting Delete phase to a day, since I don't have enough storage to make use of cold/warm tiers, though it doesn't look like it will let me set any of the min/max ages to less than a day except for hot, which is not going to be very helpful...

Would love any suggestions on how to help keep Elasticsearch and NSM cleaned out at least once a day. Not sure if best to do in the console, or if I could set a cron job to run the clean/delete commands. Thanks for any help

[cm-ops]

How much are you writing in Elastic data per day? If you have index_clean enabled it may be deleting the data streams before the end of 24 hours /opt/so/log/elasticsearch/so-elasticsearch-indices-delete.log

Depending on the amounts, you ILM settings might be as low as 1-2 days to delete.

[L0rdV0ld3m0rt]

I tried setting all of my warm/cold/delete settings to as low as they could go (1d) and set my hot max to 2 hours. Not sure what best practice is there, but seems to be doing the trick. So far changing those settings and increasing the RAM looks to have resolved that problem!

The only problem I have left now is that if I put my forwarder config under "Manager" Pipeline it will successfully forward the logs to my SIEM, but it breaks generating alerts in the local console. If I move the forwarder config to a "Custom" pipeline, my local alerts return, but it ceases to forward to the SIEM. Is there something else I need to do to allow BOTH to function at the same time?

[cm-ops]

Are you doing any filtering in that custom config?

[L0rdV0ld3m0rt]

Yeah, there is. config is (ip redacted):

`filter {
if [event][module] == "suricata" {
prune {
whitelist_names => [ "message" ]
add_tag => [ "Msg_filter" ]
}
}
}
output {

if "Msg_filter" in [tags] {
tcp {
codec => line { format => "%{message}" }
id => "SO_LOG"
host => "10.x.x.x"
port => xxxxx
}
}
} `