[18.0-fr3] Fetch up-to-date gcomm members list during a failover #349
+60
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
One chainsaw test consists in abruptly cutting one galera node away from the galera cluster and verify that the active endpoint moves to one of the remaining two galera instances. In doing so, we currently kill -9 the target mysqld server. By design, this can take by default up to 15s for the remaining galera nodes to acknowlege the node went away and react to that. This is a problem for the test as if the pod comes back online before the 15s, the galera cluster won't move the endpoint and the test will fail. To prevent flaky result in the unit test, use the STOP signal instead of the KILL signal. This doesn't kill the pod, and by default galera will mark the node as not responding after 3s, and switch the endpoint. This achieves the same result, which is to make sure that an unexpected disconnection still trigger a endpoint switch.
The operator script that implements service endpoint failover contains internal logic to probe the up-to-date state of the gcomm cluster. This is done when the script starts, or when a command failed and is retried. The list of members was incorrectly extracted from a mysql table which is not guaranteed to be up-to-date when e.g. a node disappears from the cluster due to a network partition. Instead, we must rely on the mysql status, that always exposes the up-to-date gcomm state, in particular the members that are still connected to the primary partition. Jira: OSPRH-18408
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: openshift-cherrypick-robot, stuggi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
bot
merged commit e3a64bd
into
openstack-k8s-operators:18.0-fr3
7 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an automated cherry-pick of #348
/assign lmiccini