PCAP uploaded and processed, but no data in dashboards

[commanderFx]

I uploaded a PCAP and the file is visible in the pcap/processed/ folder. But no data is shown in the dashboards (set date to 15y back).
When looking at the upload upload readiness check filebeat_tcp has a red cross.
How to fix this?

[mmguero]

for filebeat_tcp to have a red X isn't a problem, it's only enabled when you select to accept logs from external sensors, so that's not the problem. Let's see if we can figure out what is.

First let's check your settings. These should all be true:

$ grep _AUTO_ANALYZE_PCAP_FILES config/*.env
config/arkime-offline.env:ARKIME_AUTO_ANALYZE_PCAP_FILES=true
config/suricata-offline.env:SURICATA_AUTO_ANALYZE_PCAP_FILES=true
config/zeek-offline.env:ZEEK_AUTO_ANALYZE_PCAP_FILES=true

Next, let's just confirm there's really no data:

• In a browser navigate to https://xxxxxxxxx/mapi/ingest-stats. What is the latest_ingest_age_seconds value?
• In a browser navigate to https://xxxxxxxxx/mapi/agg?from=0. Is there anything under buckets?
• In a browser navigate to https://xxxxxxxxx/mapi/indexes, then search for sessions3. Is there anything there? Those are the indexes where the network log data will be.

If all of those confirm that there's no data, next let's check the file type of the files you've uploaded. Run this, does the file type say pcap capture file?

$ docker compose exec zeek find /pcap -type f
/pcap/processed/some_uploaded_file.pcap
docker compose exec zeek file /pcap/processed/some_uploaded_file.pcap
/pcap/processed/some_uploaded_file.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)

Next let's check logs as well. Run ./scripts/logs -s filebeat. Then look for messages that start with Harvester started for paths. Do you see any of those? Those would let you know if the PCAP was being processed by Zeek.

Finally let's check the logstash pipeline and see if any logs are getting in to that:

docker compose exec logstash curl -sSL -XGET http://localhost:9600/_node/stats/pipelines | jq -r '.. | .filters? // empty | .[] | objects | select (.events.in > 0) | [.id, .events.in, .events.out, .events.duration_in_millis] | join (";")' | sort -n -t ';' -k2,3 | tail -n 10

Should return something like:

...
mutate_replace_zeek_ts;1977;1977;80
mutate_split_zeek_commas_post_parse;1977;1977;61
ruby_zeek_remove_empty_values;1977;1977;1091
ruby_zeek_timestamp_calc;1977;1977;242
mutate_final_tags_remove;2056;2056;142
mutate_filebeat_zeek_forward_noop;3887;3887;113
ruby_zeek_log_source_extract;3887;3887;233
ruby_zeek_log_type_determine_drop;3887;3887;120
drop_not_filebeat_suricata;3925;0;53
drop_not_malcolm_beats;3928;0;35

Those are some places we can start.

[commanderFx]

Step 1

• all True

Step 2
_- {"latest_ingest_age_seconds":0,"sources":{}}

• {"error":"NotFoundError: NotFoundError(404, '{}')"}
• {"arkime_network_index_pattern":"arkime_sessions3-","indices":_[{"docs.count":"444","docs.deleted":"0","health":"green","index":"malcolm_beats_suricata_250917","pri":"1","pri.store.size":"536.5kb","rep":"0","status":"open","store.size":"536.5kb","uuid":"2rs5h3pzQ9iJgXbSd47viA"},{"docs.count":"1","docs.deleted":"0","health":"green","index":".plugins-ml-config","pri":"1","pri.store.size":"4kb","rep":"0","status":"open","store.size":"4kb","uuid":"sClVn5U6Rky0awpVIybo2w"},{"docs.count":"0","docs.deleted":"0","health":"green","index":".opensearch-observability","pri":"1","pri.store.size":"208b","rep":"0","status":"open","store.size":"208b","uuid":"hkP8MiG4RACdSTJOPePlcw"},{"docs.count":"0","docs.deleted":"0","health":"green","index":".ql-datasources","pri":"1","pri.store.size":"208b","rep":"0","status":"open","store.size":"208b","uuid":"BRHBG5_0QRKZwIEWuj8_YQ"},{"docs.count":"162","docs.deleted":"39","health":"green","index":"top_queries-2025.09.17-00414","pri":"1","pri.store.size":"1.4mb","rep":"0","status":"open","store.size":"1.4mb","uuid":"wN6sw7rTR5mWrkqHpYaTbw"},{"docs.count":"7","docs.deleted":"1","health":"green","index":".opendistro-job-scheduler-lock","pri":"1","pri.store.size":"20.3kb","rep":"0","status":"open","store.size":"20.3kb","uuid":"s0ebsGFYRoCeOjB7NXJADg"},{"docs.count":"4","docs.deleted":"0","health":"green","index":"opensearch-ad-plugin-result-dummy-history-2025.09.17-1","pri":"1","pri.store.size":"12kb","rep":"0","status":"open","store.size":"12kb","uuid":"7a5QDFIOQOeuyJDB9D5Z-g"},{"docs.count":"0","docs.deleted":"0","health":"green","index":".opendistro-reports-definitions","pri":"1","pri.store.size":"208b","rep":"0","status":"open","store.size":"208b","uuid":"eZ0LtrLsRgmdgI6WuBJDdA"},{"docs.count":"2714","docs.deleted":"21","health":"green","index":".kibana_1","pri":"1","pri.store.size":"3.1mb","rep":"0","status":"open","store.size":"3.1mb","uuid":"XbJ3IwdkQae80Xub9k3DHw"},{"docs.count":"3","docs.deleted":"0","health":"green","index":".opendistro-reports-instances","pri":"1","pri.store.size":"17.8kb","rep":"0","status":"open","store.size":"17.8kb","uuid":"yyFq52cmSxebSsGZSXB7RQ"},{"docs.count":"1","docs.deleted":"55","health":"green","index":"arkime_queries","pri":"1","pri.store.size":"13.4kb","rep":"0","status":"open","store.size":"13.4kb","uuid":"IxwUoO9iRaaS8_7dcNsaMQ"}],"malcolm_network_index_pattern":"arkime_sessions3-","malcolm_other_index_pattern":"malcolm_beats_*"}

Step 3
malcolm@malcolm:/Malcolm$ docker compose exec zeek find /pcap -type f
/pcap/upload/.gitignore
/pcap/processed/NBSITEID0,test.pcap
/pcap/processed/.gitignore
/pcap/arkime-live/.gitignore
malcolm@malcolm:/Malcolm$ docker compose exec zeek file /pcap/processed/NBSITEID0,test.pcap
/pcap/processed/NBSITEID0,test.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)

Step 4
last lines on the screen...

filebeat-1 | 2025-09-17T18:15:43.007Z Harvester started for paths: [/suricata/live/eve*.json]: /suricata/live/eve-20250917_181538.json
filebeat-1 | 2025-09-17T18:45:22.976Z File is inactive. Closing because close_inactive of 1h30m0s reached: /suricata/live/eve-20250917_161538.json
filebeat-1 | 2025-09-17T19:16:00.137Z Harvester started for paths: [/suricata/live/eve*.json]: /suricata/live/eve-20250917_191558.json
filebeat-1 | 2025-09-17T19:45:27.748Z File is inactive. Closing because close_inactive of 1h30m0s reached: /suricata/live/eve-20250917_171538.json
filebeat-1 | 2025-09-17T20:16:00.616Z Harvester started for paths: [/suricata/live/eve*.json]: /suricata/live/eve-20250917_201558.json

Step 5
mutate_filebeat_suricata_forward_noop;6169;6169;1164
mutate_remove_field_suricata_ja3_strings;6169;6169;127
mutate_remove_field_suricata_useless;6169;6169;216
mutate_rename_suricata_tls;6169;6169;141
mutate_suricata_rename_global_fields;6169;6169;668
ruby_suricata_remove_empty_values;6169;6169;5988
ruby_suricata_tags_extract;6169;6169;748
ruby_suricata_timestamp_calc;6169;6169;2852
mutate_final_tags_remove;7227;7227;4011
drop_not_malcolm_beats;7233;0;692

[mmguero]

I'm still not sure. Let's do this:

• if you're using Malcolm v25.08.1 or higher, add PCAP_PIPELINE_LOGLEVEL=info to ./config/upload-common.env; if it's an earlier version than that, change PCAP_PIPELINE_VERBOSITY to -vv in ./config/upload-common.env
• restart Malcolm (./scripts/restart)
• after it's started up, run ./scripts/logs -s pcap-monitor

There will be a few "errors" at first as it waits for the background services to get started up, then you'll eventually see:

pcap-monitor-1  | 2025-09-22 20:17:36 INFO: pcap_watcher.py:	waiting for 1 to be healthy
pcap-monitor-1  | 2025-09-22 20:17:45 INFO: GET https://opensearch:9200/_cluster/health/arkime_files?wait_for_status=yellow [status:200 request:9.391s]
pcap-monitor-1  | 2025-09-22 20:17:45 INFO: GET https://opensearch:9200/_cluster/health [status:200 request:0.005s]
pcap-monitor-1  | 2025-09-22 20:17:45 INFO: pcap_watcher.py:	binding publisher port 30441
pcap-monitor-1  | 2025-09-22 20:17:45 INFO: pcap_watcher.py:	EventWatcher initialized
pcap-monitor-1  | 2025-09-22 20:17:45 INFO: ۞	started	[1121:1]

Wait 5 minutes or so for the entire system is up (everything is green checkmarks in the upload dialog, minus the filebeat TCP one we talked about not mattering; another way to check to see if things are ready is to run ./scripts/logs -s logstash and then see Pipelines running {:count=>6... at the bottom of those logs) then try your upload. In your ./scripts/logs -s pcap-monitor output you'll see something like:

pcap-monitor-1  | 2025-09-22 20:19:31 INFO: watch-pcap-uploads-folder.py:	👓	/pcap/upload/NBSITEID0,Cyberville.pcap
pcap-monitor-1  | 2025-09-22 20:19:31 INFO: watch-pcap-uploads-folder.py:	🖅	/pcap/upload/NBSITEID0,Cyberville.pcap [application/vnd.tcpdump.pcap][pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)] to /pcap/processed
pcap-monitor-1  | 2025-09-22 20:19:31 INFO: 🖄	processed	/pcap/upload/NBSITEID0,Cyberville.pcap at 10 seconds	[1122:1]
pcap-monitor-1  | 2025-09-22 20:19:43 INFO: pcap_watcher.py:	👓	/pcap/processed/NBSITEID0,Cyberville.pcap
pcap-monitor-1  | 2025-09-22 20:19:43 INFO: POST https://opensearch:9200/arkime_files/_search [status:200 request:0.022s]
pcap-monitor-1  | 2025-09-22 20:19:43 INFO: pcap_watcher.py:	📩	/pcap/processed/NBSITEID0,Cyberville.pcap
pcap-monitor-1  | 2025-09-22 20:19:43 INFO: pcap_watcher.py:	📫	{'name': 'NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
pcap-monitor-1  | 2025-09-22 20:19:43 INFO: 🖄	processed	/pcap/processed/NBSITEID0,Cyberville.pcap at 12 seconds	[1121:1]

If that matches, you can swap out pcap-monitor from that command with arkime, zeek, and suricata which will, respectively, have outputs like this:

arkime-1  | 2025-09-22 20:19:43 INFO: pcap_arkime_processor.py:	📨	{'name': 'NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
arkime-1  | 2025-09-22 20:19:43 INFO: pcap_arkime_processor.py[1]:	🔎	{'name': '/data/pcap/processed/NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['Cyberville']}
arkime-1  | 2025-09-22 20:19:53 INFO: pcap_arkime_processor.py[1]:	✅	NBSITEID0,Cyberville.pcap

zeek-1  | 2025-09-22 20:19:43 INFO: pcap_zeek_processor.py:	📨	{'name': 'NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
zeek-1  | 2025-09-22 20:19:43 INFO: pcap_zeek_processor.py[1]:	🔎	{'name': '/pcap/processed/NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
zeek-1  | 2025-09-22 20:20:07 INFO: pcap_zeek_processor.py[1]:	✅	NBSITEID0,Cyberville.pcap

suricata-1  | 2025-09-22 20:19:43 INFO: pcap_suricata_processor.py:	📨	{'name': 'NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
suricata-1  | 2025-09-22 20:19:43 INFO: pcap_suricata_processor.py[1]:	🔎	{'name': '/data/pcap/processed/NBSITEID0,Cyberville.pcap', 'size': 177023616, 'mime': 'application/vnd.tcpdump.pcap', 'type': 'pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535)', 'node': 'malcolm.sgrover', 'live': False, 'tags': ['NBSITEID0', 'Cyberville']}
suricata-1  | 2025-09-22 20:19:43 INFO: pcap_suricata_processor.py[1]:	📥	Submitting NBSITEID0,Cyberville.pcap to Suricata
suricata-1  | 2025-09-22 20:19:43 INFO: pcap_suricata_processor.py[1]:	✅	NBSITEID0,Cyberville.pcap with suricata-command-1.socket

Hopefully that will help us see what's not right.