Missing Fields

[devilman85]

I recently updated malcolm but I'm having some small problems with Arkime. As you can see from the screenshots it doesn't detect some fields and the views of suricata and

zeek are gone. How can I fix it?

[mmguero]

Ummm... boy, I am not sure, I have not seen this before. Is it possible to ./scripts/wipe your Malcolm instance and then restart it, to see if it's reproducible/persistent? If not, I could try to give you the commands to delete just the Arkime non-log indices and then see if restarting it recreates things.

[devilman85]

I performed the wipe but still gives the same result. Waiting for commands to remove arkime non-logs

[mmguero]

Okay, so the wipe would have removed everything so somehow the initialization code that creates the Zeek/Macolm fields in Arkime isn't happening correctly.

Let's try this:

1. ./scripts/wipe to wipe everything
2. Stop all sources of log data to Malcolm (like, if you have Hedgehog Linux sensors sending data)
3. ./scripts/start to start Malcolm
4. ./scripts/logs -s arkime to watch Arkime come up, share the logs that come out from Arkime here and we'll see if we can help figure out what's going on.

So this is Malcolm v24.04.1, correct? Are you using its embedded OpenSearch instance or a remote OpenSearch or Elasticsearch instance?

[devilman85]

yes I am using malcolm 24.04.1 with remote Elasticsearch Instance but I am not using Hedgehog . I will do all the steps tomorrow. in the meantime thanks

[mmguero]

Ah the with remote elasticsearch instance the wipe won't quite have the same effect. we'll have to delete some indices manually. Hold off on doing anything for now, I'll try to help put together some commands you can do so we can reinitialize.

[devilman85]

Ah the with remote elasticsearch instance the wipe won't quite have the same effect. we'll have to delete some indices manually. Hold off on doing anything for now, I'll try to help put together some commands you can do so we can reinitialize.

I await the commands for elasticsearch

[mmguero]

Rather than doing it from the command line, probably the safest way to do this would be to:

1. Stop Malcolm
2. Go into Kibana on your Elasticsearch cluster
3. Go into Kibana's index management UI (I'm not much of a Kibana user, I assume it has this?)
4. Delete all of the indexes that begin with arkime
5. Start Malcolm back up

Otherwise, use the elasticsearch DELETE api to delete the arkime* indices.

[devilman85]

Resolve. Thanks for supporting