PCAP Files keep filling up Hard Drive

[DJNAT10]

Good Morning,

I'm reaching out to you to help with an issue we're having related to Malcolm. We are currently rebuilding due to a hardware failure. Specifically, we've encountered a critical issues:
• The system hard drive becoming full over the weekend, in addition the server stopped working and we weren't able to restart. Restart did work however when we deleted the PCAP files.

Here are the relevant environment configurations we've set:
• MANAGE_PCAP_FILES=true
• ARKIME_FREESPACEG=75%
according to the documentation, Arkime should begin deleting the oldest pcap files once disk space drops below 75% free, but this does not appear to be occurring.
Additionally, for index management:
• We have configured OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=60% in dashboards-helper.env to manage OpenSearch indices, with a limit to prune older indices as needed.

We have a single data partition /Malcolmdata both PCAP and OpenSearch data are writing to this shared 10 Terabyte partition.

Despite these settings, the disk usage continues to grow until it was completely full, and Arkime stopped showing data. We would appreciate your help identifying the problem so we can find the solution.
Please let us know how we can proceed to further troubleshoot or provide more context.

Best regards,
Natasha Jones

[mmguero]

Sure, here are a few things we can check:

• for the PCAP one, Malcolm passed ARKIME_FREESPACEG directly in to arkime's config.ini as freeSpaceG which should manage pcap deletion, and having MANAGE_PCAP_FILES=true should be marking the PCAP files as "deletable" as described here. As suggested by the Arkime FAQ let's try to increase the arkime debug level to see what might be happening. Set ARKIME_DEBUG_LEVEL=2 in arkime.env and restart Malcolm. Then, as the disk starts to fill up watch the logs with ./scripts/logs -s arkime and see if Arkime viewer spits out any error messages or anything with expire in it.
• How you described setting OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT seems correct, unless there is some kind of issue where the code doing the index prune isn't working for some reason. Let's try adding OPENSEARCH_INDEX_SIZE_PRUNE_DEBUG=true to your dashboards-helper.env and restarting Malcolm, then watch ./scripts/logs -s dashboards-helper to see the debug output from that script and we'll try to figure out what's going on.