Zeek 4.0.7 terminating

[samanosuke26]

Trying to get help with zeek issues we've been having on 2.3.90 and now 2.3.130.
We have roughly 300 sensors deployed, and I don't have the numbers, but a lot of them have their zeek containers crashing numerous times throughout the day. Troubleshooting zeek has been difficult since it doesn't throw out meaningful errors or logs unless you configure it that way. Here is an example forward node:

SO: 2.3.130
zeek: 4.0.7
CENTOS 7 (64-bit) on ESXi server 6.5.0
Kernel: 3.10.0-1160.66.1.e17.x86_64
8 CPUs
24GB RAM
250GB HDD
Monitor traffic: 17.7 Mbps

Output of docker logs so-zeek
Error: failed to remove directory /nsm/zeek/spool/installed-scripts-do-not-touch/site: [Errno 13] Permission denied: 'hassh'

Output of a worker's crash diag
No core file found and gdb is not installed. It is recommended to install gdb so that ZeekControl can ouput a backtrace if Zeek crashes.

==== stderr.log
listening on bond0

KILLED
1674163422.587314 received termination signal
1674163422.587314 39963820 packets received on interface bond0, 16 (0.00%) dropped

We are not running any custom scripts or plugins, and can't get zeek to generate a core dump.

Any help would be appreciated.

FUTURE IDEA: Have a containerized zeek for troubleshooting, with all logging turned on and zeek ran with debugging and gdb turned on.

[andre-dessert]

Could you check the output from running:
salt <SENSOR-NAME> zeekctl.check
salt <SENSOR-NAME> zeekctl.status
salt <SENSOR-NAME> zeekctl.diag
on one of the failing sensors?

[samanosuke26]

Working on getting that log airgapped over
but all three came back with no issues, everything was running.
The only "errors" were the gdb errors in the diag log.