Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support starting existing cluster from 0 controller nodes #561

Open
pschichtel opened this issue Sep 20, 2023 · 0 comments
Open

Support starting existing cluster from 0 controller nodes #561

pschichtel opened this issue Sep 20, 2023 · 0 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@pschichtel
Copy link

pschichtel commented Sep 20, 2023

k0sctl version v0.15.5 on NixOS, k0s 1.27.5 on Rocky 9.

We've setup a 3 node controller+worker cluster. Due to a misconfiguration of the storage extension k0sctl took down all 3 nodes at once (another bug I guess). After that it was unable to start the cluster back up, because it attempted to start the nodes one by one, however the etcd obviously never get's a quorum with 1 out of 3 members. To fix the situation we manually started 2 of the 3 nodes simultaneously, so that etcd would become operational.

So to reproduce:

  1. Setup a cluster with 3 controllers
  2. Stop all 3 controllers
  3. Try applying the k0sctl config

Expected:

k0sctl realizes that not enough controllers are running to get a quorum and decides to start multiple nodes at once.

Actual:

k0sctl tries to start first node which eventually fails.

@kke kke added enhancement New feature or request bug Something isn't working labels Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants