Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch #405

khaledk2 · 2023-07-20T14:52:39Z

This PR updates Elasticsearch to version 8 (8.8.1). It also secures the connection between the searchengine and the elasticsearch cluster and between the elasticsearch cluster nodes themselves.
I have tested it locally and it worked fine. I am now testing it in pilot-idr0000-omeroreadwrite.
This PR required changes to the search engine code and I will create a PR for it soon.

…engine and elasticsearch and clustern nodes

khaledk2 · 2023-07-26T11:40:11Z

I have created a seachengine PR (92) to support securing the connection between searchengine and Elasticsearch.

khaledk2 · 2023-07-27T09:05:56Z

@sbesson I have created a docker image for the searchengine (khaledk2/searchengine:teste2_1) and used it to test the deployment.

ansible/group_vars/searchengine-hosts.yml

sbesson · 2023-08-14T10:17:55Z

ansible/group_vars/searchengine-hosts.yml

@@ -5,7 +5,7 @@ database_name: idr
 database_username: omeroreadonly
 database_user_password:  "{{ idr_secret_postgresql_password_ro | default('omero') }}"
 searchenginecache_folder: /data/searchengine/searchengine/cacheddata/
-search_engineelasticsearch_docker_image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
+search_engineelasticsearch_docker_image: docker.elastic.co/elasticsearch/elasticsearch:8.8.1
 searchengine_docker_image: openmicroscopy/omero-searchengine:0.5.2


I assume this PR will require another update with a new version of the Docker image once the corresponding application PR has been reviewed, merged and released

Yes, this is the case.

ansible/idr-elasticsearch.yml

sbesson · 2023-08-14T10:29:40Z

ansible/idr-elasticsearch.yml

      networks:
      - name: "searchengine-net"
+        ipv4_address: 10.11.0.2


What is the requirement for hardcoding this IPv4 address?

It is the first searchengine node, so it should have this IP address. The remaining nodes' IPs depended on the number of nodes in the elasticsearch cluster.

ansible/idr-elasticsearch.yml

ansible/idr-searchengine.yml

ansible/idr-elasticsearch.yml

khaledk2 · 2023-08-17T13:23:02Z

I have passwords as private variables which have default values.
This should work with management_tools 1692

sbesson

Thanks @khaledk2 for separating the private and public variables. No additional comments immediately from my side on the deployment front. I think the next step is to evaluate the corresponding application changes and move towards a new Docker image which can be used for evaluation. I'll leave you and @jburel to coordinate on the latter

sbesson · 2023-09-14T08:15:15Z

ansible/idr-elasticsearch.yml

+
+  - name: Add elastic nodes to instances_nodes
+    set_fact:
+      instances_nodes: "{{instances_nodes + [( {'name' : 'searchengine_elasticsearch_node'+item, 'dns': ['searchengine_elasticsearch_node'+item,'localhost'],'ip': '127.0.0.1'})] }}"


The playbook failed to execute against test120-searchengine with the following error

The task includes an option with an undefined variable. The error was: 'instances_nodes' is undefined The error appears to be in '/tmp/management_tools/idr/deployment/ansible/idr-elasticsearch.yml': line 86, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Add elastic nodes to instances_nodes ^ here"

@sbesson, I have pushed a fix for this issue, could you please try again?

Same error with instances in the next task.

sbesson · 2023-09-14T11:22:33Z

ansible/idr-elasticsearch.yml

-
+  - name: Pause for 1 minutes
+    ansible.builtin.pause:
+      minutes: 1


Rather than pausing for an arbitrary amount of time. Would a different state e.g. stopped in the previous task allow to ensure that the command is completed successfully before executing the rest of the playbook - https://docs.ansible.com/ansible/2.9/modules/docker_container_module.html#parameters?

I have used wait_for to ensure that the ca file is present instead of pause for an arbitrary amount of time.

khaledk2 · 2023-09-14T11:22:58Z

Sorry, it seems that I have posted the following message to the wrong PR.
The Elasticsearch nodes did not run correctly, I have checked that and found out that the CA file was not written to the mapped folder, so I have added a pause for 1 minute and tested that and it should work fine now.
@sbesson I have stopped all the containers deleted the stopped containers and created folders, could you please re-run the playbook again?

khaledk2 · 2023-09-21T10:50:21Z

I have renamed the elasticsearch_nodes vraible inside idr-searchengine.yml to elasticsearch_nodes_urls as idr-elasticsearch.yml has a variable with the same name.
When including the two playbooks in one playbook and running it, it will keep the items which have been added from the first included playbook inside the elasticsearch_nodes list when running the second included playbook rather than create a new list.

sbesson

Proposing to schedule the deployment of the latest version of the search engine onto prod119. As for test120, this will require the deletion and reprovisioning of the searchengine VM but should not affect any import/annotation work happening over the OMERO VMs. @dominikl @francesw are you happy for this work to be carried out anytime or would you suggest a specific timeslot?

khaledk2 · 2023-09-26T09:52:47Z

@sbesson Could you please delete the searchengine folder in the /data before deployment in case you will use the same disk?

sbesson · 2023-09-26T10:04:56Z

@khaledk2 so far I have tested a complete recreation of the VM but if you are confident we can upgrade prod1119-searchengine without recreating the instance, that certainly would reduce the impact and the complexity. Minimally, we would 1- stop all the Docker instances, 2- delete /data before running the playbook. Anything else?

khaledk2 · 2023-09-26T10:09:50Z

@sbesson, Yes, that would be fine, to be on the safe side, we may also delete all the stopped containers.

sbesson · 2023-09-27T08:17:39Z

Deployed on prod119

[sbesson@prod119-searchengine data]$ sudo docker ps
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS          PORTS                                            NAMES
18da47ac08f8   openmicroscopy/omero-searchengine:0.5.3               "bash run_app.sh run…"   25 seconds ago   Up 23 seconds   0.0.0.0:5577->5577/tcp, 8080/tcp                 searchengine
d2507ff519f5   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9203->9200/tcp, 0.0.0.0:9303->9300/tcp   searchengine_elasticsearch_node3
bcf010a6c401   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp   searchengine_elasticsearch_node2
3e0fcc5dd9b0   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp   searchengine_elasticsearch_node1

khaledk2 · 2023-09-27T09:06:34Z

Looks good! thank you.

khaledk2 added 2 commits July 20, 2023 15:37

update to elastic search 8 and secure the connection between search …

43dd559

…engine and elasticsearch and clustern nodes

Use IP address for the elasticsearch nodes

a0aea10

khaledk2 mentioned this pull request Jul 26, 2023

Secure the connection with the elsticsearch ome/omero_search_engine#92

Merged

sbesson requested changes Aug 14, 2023

View reviewed changes

khaledk2 added 2 commits August 14, 2023 14:44

delete commented code

94ebcc8

set searchengine elasticsearchprivate vars

b949259

sbesson reviewed Aug 17, 2023

View reviewed changes

sbesson requested changes Sep 14, 2023

View reviewed changes

khaledk2 added 3 commits September 14, 2023 09:42

add instances_nodes

bc382dd

add instances

d77f9c3

add pause to allow write CA file

47cd12d

sbesson reviewed Sep 14, 2023

View reviewed changes

khaledk2 added 2 commits September 14, 2023 14:56

use wait for ca file present instead of pause

a4a7141

rename elasticsearch_nodes var to elasticsearch_nodes_urls

22672b1

Update the search engine image tag

bd9d263

sbesson approved these changes Sep 25, 2023

View reviewed changes

sbesson merged commit 624d889 into IDR:master Sep 27, 2023
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch #405

Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch #405

khaledk2 commented Jul 20, 2023

khaledk2 commented Jul 26, 2023

khaledk2 commented Jul 27, 2023

sbesson Aug 14, 2023

khaledk2 Aug 14, 2023

sbesson Aug 14, 2023

khaledk2 Aug 14, 2023

khaledk2 commented Aug 17, 2023

sbesson left a comment

sbesson Sep 14, 2023

khaledk2 Sep 14, 2023

sbesson Sep 14, 2023

sbesson Sep 14, 2023

khaledk2 Sep 14, 2023 •

edited

Loading

khaledk2 commented Sep 14, 2023

khaledk2 commented Sep 21, 2023 •

edited

Loading

sbesson left a comment

khaledk2 commented Sep 26, 2023

sbesson commented Sep 26, 2023

khaledk2 commented Sep 26, 2023

sbesson commented Sep 27, 2023

khaledk2 commented Sep 27, 2023

Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch #405

Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch #405

Conversation

khaledk2 commented Jul 20, 2023

khaledk2 commented Jul 26, 2023

khaledk2 commented Jul 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khaledk2 commented Aug 17, 2023

sbesson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khaledk2 Sep 14, 2023 • edited Loading

Choose a reason for hiding this comment

khaledk2 commented Sep 14, 2023

khaledk2 commented Sep 21, 2023 • edited Loading

sbesson left a comment

Choose a reason for hiding this comment

khaledk2 commented Sep 26, 2023

sbesson commented Sep 26, 2023

khaledk2 commented Sep 26, 2023

sbesson commented Sep 27, 2023

khaledk2 commented Sep 27, 2023

khaledk2 Sep 14, 2023 •

edited

Loading

khaledk2 commented Sep 21, 2023 •

edited

Loading