Multiple master nodes after docker-compose restart

[jbumgit]

Hello.

I'm new to kubernetes and am using k3s as my learning tool (a great one at that). I'm using the official docker image of k3s (v1.23.1_k3s1) to serve as my master on my Linux laptop. I have two worker nodes running on a couple Raspberry Pi 2s and joined to the cluster without issues.

The problem is that if I bring the k3s container down and then back up (docker-compose down/up -d), I see another master node being created. The previous running node goes to "NotReady" status and the new instance is "Ready". Every docker-compose down and up seems to bring another master node instance and puts all the previous ones to "NotReady". I've been deleting the ones that are not running manually (delete node).

NAME           STATUS     ROLES                  AGE     VERSION
2b9de5f4c803   NotReady   control-plane,master   17m     v1.23.1+k3s1
worker1        NotReady   <none>                 40h     v1.23.1+k3s1
7fbe9ac53390   NotReady   control-plane,master   21m     v1.23.1+k3s1
97975d8ecbb1   NotReady   control-plane,master   22m     v1.23.1+k3s1
02fa9a32a700   NotReady   control-plane,master   15m     v1.23.1+k3s1
cb0de985258b   NotReady   control-plane,master   14m     v1.23.1+k3s1
69faed564b89   Ready      control-plane,master   8m52s   v1.23.1+k3s1
worker2        Ready      <none>                 40h     v1.23.1+k3s1

Is this expected behavior? My docker-compose.yml is shown below. I removed the "agent" section but otherwise it's more or less the same as the official copy.

version: '3'
services:
server:
image: nasberry:5000/rancher/k3s:v1.23.1-k3s1
command: server --tls-san t400
tmpfs:
- /run
- /var/run
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
privileged: true
restart: always
environment:
- K3S_TOKEN=<some value, not printed here>
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
- /srv/k3s/var/lib/rancher/k3s:/var/lib/rancher/k3s
# This is just so that we get the kubeconfig file out
- .:/output
ports:
- 6443:6443 # Kubernetes API Server
- 80:80 # Ingress controller port 80
- 443:443 # Ingress controller port 443

Any help would be greatly appreciated.

[brandond]

You're not assigning a fixed hostname to your Docker container, so every time you stop and start it, the node name changes. Thats why you see a new node in the cluster every time you restart.

[jbumgit]

Ah... thank you for the quick tip. Set the hostname in my docker compose file and everything appears to be working now. Thank you so much!

[brandond]

I would definitely recommend k3d over k3s in docker-compose though.

[tamama]

Thank you!

This saves my day.

For us, k3d is not an option for us, because we want k3s to be a sidecar for every data pipeline we run. We do not want k3s as a separate silo - otherwise we need to provide additional SOP guide for our DevOps team. To us, k3s has to quietly sneak into our ecosystem. :)

[tamama]

@brandond ^ thanks for your hint!

[ChristianCiach]

brandond surely already gave you the right answer. But if you just want to learn how to use kubernetes, you should take a look at https://k3d.io . This will give you k3s in docker without any headaches.

[jbumgit]

Good to know. I may have misunderstood its capabilities then. I thought it was more like minikube - multiple nodes but on single physical host. If I run into further problems with k3s will definitely give k3d a try next.

Thanks for all the help and input.

[ChristianCiach]

Oh, yes, you are right about that. I don't think you can use k3d to build a cluster across multiple physical hosts. But you also cannot do that with docker-compose, can you?

[ChristianCiach]

I see now what you are trying to do. "docker-compose" is actually not important in your scenario. You may just as well just use docker run.

Well, yeah. If you want to use multiple raspberry pis to build a cluster, you may just as well ditch docker altogether and just use the normal install script to properly install the k3s binary. Just make sure that you are using a Linux distribution that uses systemd.

There may be other pitfalls when using k3s on a raspberry pi. I actually never tried that. If the docker installation works well for you, I guess there is also nothing wrong with that :)

[jbumgit]

Right, you can't with docker-compose either. My docker setup is just for the master server. For the worker nodes, I run k3s manually/separately on my raspberry pi devices. I have a bunch of other things I'm doing on my dev laptop so I chose docker/docker-compose to isolate the k3s work/changes.

[jbumgit]

k3s on raspberry pi has been surprisingly robust thus far. Regarding systemd, I'm actually on Slackware :) so that might have been another reason I went to the k3s docker image for the master node.