Opensearch backup for migration to Remote Opensearch Cluster

[GeoCoastie]

My network has reached its limit and attempting to scale out for growth. Currently running Malcolm with all its services on a single VM, and looking to maintain the current indicies mapped via Opensearch. I'd like to snapshot and backup the entire Opensearch database and migrate it to a separate Remote Opensearch instance.

Some reasons for this migration:

• Logstash consistently fails, container restart allows the data to flow for a few minutes then crashes and goes unhealthy again.
  • Upgrading to newer Malcolm ver may solve this? Possible bug?
• Wanting a Remote cluster for easier scaling.
• services may be shifting to newer hardware

Currently I have in a test environment Malcolm successfully talking to and ingesting with a Remote OS Cluster. No issues there.

With my current single node OS instance, ive attempted to:

1. Use Dev tools to create a new snapshot - Failed

2. When going into Creating a snapshot from Snapshot Managment -> Snapshots, i do not see any arkime indicies.

If i was to go into the configuration tool, re-run to setup ILM/ISM? Will that change anything?

Attempting to not loose the data, and document the migration path forward if needed and potentially a cold/offsite/remote backup location in the future.

For additional context:
Running Malcolm 24.11.0 ---> Upgrading to 25.05.0

Appreciate the assistance and feedback.

[mmguero]

Sorry for the delay, I was hoping somebody more experienced than I am with this facet of OpenSearch would chime in, because it's not really my area of knowledge.

So internally to the OpenSearch container in Malcolm the paths are going to be different than they are outside on the host. So if you look at docker-compose.yml you'll see the opensearch-backup bind mount, which will be set to some path on your host machine for the source and likely /opt/opensearch/backup for the target. That target path is the one you'd want to specify in configuration for snapshots: OpenSearch won't know anything about /home/malcolm/etc... because it's only seeing the target of the bind mount internally.

Another option is you ought to be able to literally just rsync the contents of your Malcolm's opensearch directory to a new location (on a new machine, or whatever) running Malcolm and make sure that it ends up in the right place and the new Malcolm instance ought to just pick up where the old one left off.

[GeoCoastie]

This was still a test environment and not fully production, I decided to just do an upgrade and adjust settings with the new deployment. Still something ill continue to tinker around and research for the future. Appreciate the suggestions!