Suricata Alerts Not Ingested into Security Onion – Port 5055 Refused, Logstash Configured, Alerts Present in EVE Logs #14826
Replies: 1 comment 3 replies
-
|
Have you tried to |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! Yes I have tried "sudo tcpdump -i any -vvv -X host [manager ip] and port 5055" from the sensor and I am seeing active traffic communicating with the manager on port 5055. Multiple packets are exchanged with valid TCP flags across multiple interfaces (ens and bond). Checksums are marked as correct. The capture shows no dropped packets. From what I can see the network connectivity is working between the sensor and the manager and that port 5055 is open and in use with no signs of packet loss or corruption. |
Beta Was this translation helpful? Give feedback.
-
|
Are there any errors in the |
Beta Was this translation helpful? Give feedback.
-
|
I'm not seeing any errors although there are many warnings about: "An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception." Could this pipeline warning be the culprit? [REDACTED_TIMESTAMP][INFO ][org.logstash.beats.BeatsHandler] [REDACTED_HOST]:5055, remote: [REDACTED_IP]:57040] Handling exception: java.net.SocketException: Connection reset (caused by: java.net.SocketException: Connection reset) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
[root@secon2 log]# grep -iE error redact.txt
redacted_northsouth01_alerts02.txt
redacted_secon2_elastic_2.txt
Version
2.4.160
Installation Method
Security Onion ISO image
Description
configuration
Installation Type
Distributed
Location
on-prem with Internet access
Hardware Specs
Exceeds minimum requirements
CPU
32
RAM
93
Storage for /
222 GB
Storage for /nsm
1.8 TB
Network Traffic Collection
tap
Network Traffic Speeds
Less than 1Gbps
Status
Yes, all services on all nodes are running OK
Salt Status
No, there are no failures
Logs
Yes, there are additional clues in /opt/so/log/ (please provide detail below)
Detail
Hi team,
We are investigating an issue where Suricata alerts are correctly generated on the forward node (northsouth01) but are not appearing in the Security Onion dashboard managed by secon2.
We have confirmed the presence of Suricata alerts in /nsm/suricata/eve-.json, with signatures such as ET EXPLOIT Possible CVE-2020-11899 and ET INFO SMBv2 Protocol Session Setup Observed. These indicate that detection rules are firing and traffic is being captured on bond0. Alerts have expected metadata, severity, and timestamps.
However, no recent suricata- indices are visible via so-elasticsearch-query _cat/indices?v | grep suricata, suggesting the alerts are not being ingested into Elasticsearch.
To troubleshoot, we have:
• We verified that the TLS certificates specified in the Logstash pipeline (elasticfleet-logstash.crt and associated key/CA files) are present, valid, and have not expired.
• Verified Suricata containers are running on northsouth01 forward node (sudo docker ps | grep suricata)
• Confirmed /opt/so/log/suricata on secon2 manager is empty
• Restarted ingestion services:
o sudo salt-call state.apply logstash
o sudo so-logstash-restart
o sudo so-elasticsearch-restart
• Confirmed /nsm has 388 GB free (79% used), so no disk pressure is present
• Ran sudo so-test and sudo so-status with no reported issues
We do have a BPF in place, but alerts in eve.json confirm matching traffic is still logged.
Additionally, Logstash is configured to accept input from Elastic Agents via /opt/so/conf/logstash/pipelines/manager/0012_input_elastic_agent.conf:
input {
elastic_agent {
port => 5055
tags => [ "elastic-agent", "input-secon2" ]
ssl => true
ssl_certificate_authorities => ["/usr/share/filebeat/ca.crt"]
ssl_certificate => "/usr/share/logstash/elasticfleet-logstash.crt"
ssl_key => "/usr/share/logstash/elasticfleet-logstash.key"
ssl_verify_mode => "force_peer"
ecs_compatibility => v8
}
}
filter {
if ![metadata] {
mutate {
rename => {"@metadata" => "metadata"}
}
}
}
However, agents attempting to send logs to tcp://secon2.domain.edu:5055 and udp://secon2.domain.edu:5055 receive either "connection refused" or "no route to host," indicating a port binding issue, SSL verification failure, or firewall block.
We also reviewed logs in /opt/so/log/ and found the following relevant issues:
From /opt/so/log/agents/agentstatus.log:
• "error": 209
From /opt/so/log/elastalert/elastalert.log:
• Dozens of verification_exception errors indicate that ElastAlert is referencing unknown or mismatched fields, such as:
o Unknown column [dns.query.name]
o Unknown column [script_block_text] with no keyword/multi-field
o first argument must be [boolean], found type [keyword]
• A repeated TransportError 504 search_phase_execution_exception indicates slow or failed Elasticsearch queries.
From elastic_agent.filebeat logs (via Fleet):
• Repeated invalid_client errors from Microsoft Defender integration:
• AADSTS7000222: The provided client secret keys for app 'xxxx-xxxx-xxx-xxxx' are expired.
• Multiple write: broken pipe and connection reset by peer messages when publishing events to port 5055
• Failover attempts between tcp://secon2.domain.edu:5055, tcp://secon2:5055, and tcp://fleet:5055 indicate unstable or missing connections
Summary
We believe the root issue lies in the Logstash input on secon2 not binding to port 5055 or failing mutual TLS validation, preventing Elastic Agents (especially httpjson-so-manager_logstash) from publishing. This failure is supported by repeated "broken pipe" and "connection refused/reset" messages, alongside the fact that Suricata logs on the forward node are healthy and populated.
We’ve attached the following logs for review:
• Suricata eve.json snippets from northsouth01
• elastic_agent.filebeat logs from Fleet
• Grepped output of all error/timeout logs from /opt/so/log/
Node Type Breakdown:
Forward Nodes (9 total):
Most show:
CPU: Generally low, between ~0.7% and 8.3%
Memory (Mem): Ranges from ~6% to ~58%
/nsm Usage: Varies, but most are above 85%, indicating high packet logging volume
Status: All report OK
Manager Node (1 total):
CPU: ~15.5%
Mem: ~53.4%
/nsm Usage: ~64.3%
Status: Displays a “Reboot” flag
Network: Actively processing ~34.1 Mbps in and ~498.9 Mbps out
Search Node (1 total):
CPU: Highest load at ~64.6%
Mem: ~62.9%
/nsm Usage: ~74.8%
EPS: Over 5,800 events/sec
Network: ~500.2 Mbps in, ~24.2 Mbps out
Status: OK
Fleet Node (1 total):
Handles Elastic Agent communication
CPU: ~15.9%
/nsm Usage: ~12.3%
Status: OK
Honeypot Nodes (2 total):
CPU & Disk: Very low usage, minimal load
Role: Deployed for IDS baiting
Status: OK
Summary:
Forward: 9 nodes
Honeypot: 2 nodes
Fleet: 1 node
Search: 1 node
Manager: 1 node
Total: 15 nodes
With the exception of the manager needing a reboot, all nodes report healthy status, appropriate resource utilization, and stable performance across the grid.
We would appreciate your guidance in pinpointing the ingestion failure and any recommended fixes for restoring Suricata alert visibility in the SOC.
Guidelines
Beta Was this translation helpful? Give feedback.
All reactions