Cannot start k0s inside LXD container.

[a-prokopyev-resume]

Hello,

I have some positive experience using your k0s in Docker container on the host, but unfortunately still cannot succeed with running k0s inside LXD container.

Here is output of the lxc profile show k8s command:

config:
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter,nf_conntrack,iscsi_tcp
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw cgroup:rw:force
    lxc.cgroup.devices.allow=a
    lxc.cgroup2.devices.allow=c 10:200 rwm
    lxc.cap.drop=
    lxc.seccomp.profile=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable2:
    path: /dev/zfs
    source: /dev/zfs
    type: disk
  aadisable3:
    path: /dev/kmsg
    source: /dev/kmsg
    type: unix-char
  aadisable4:
    path: /sys/fs/bpf
    source: /sys/fs/bpf
    type: disk
  aadisable5:
    path: /proc/sys/net/netfilter/nf_conntrack_max
    source: /proc/sys/net/netfilter/nf_conntrack_max
    type: disk
  root:
    path: /
    pool: default
    type: disk
name: k8s
used_by:
- /1.0/instances/wp-1
- /1.0/instances/alpine2
- /1.0/instances/alpine3
- /1.0/instances/alpine4
- /1.0/instances/alpine5

[a-prokopyev-resume]

Output of k0s sysinfo on alpine5 instance:

Total memory: 964.7 MiB (warning: 1.0 GiB recommended)
File system of /var/lib: unknown (warning)
Disk space available for /var/lib/k0s: 3.4 GiB (pass)
Relative disk space available for /var/lib/k0s: 93% (pass)
Name resolution: localhost: [127.0.0.1 ::1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.151-gnu (pass)
  Max. file descriptors per process: current: 65536 / max: 65536 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /sbin/modprobe (pass)
  Executable in PATH: mount: /bin/mount (pass)
  Executable in PATH: umount: /bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": unavailable (rejected)
    cgroup controller "cpuacct": unavailable (rejected)
    cgroup controller "cpuset": unavailable (rejected)
    cgroup controller "memory": unavailable (rejected)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": unavailable (rejected)
    cgroup controller "hugetlb": unavailable (warning)
    cgroup controller "blkio": unavailable (warning)
  CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
  CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
  CONFIG_NET: Networking support: no kernel config found (warning)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
  CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)
Error: sysinfo failed

[a-prokopyev-resume]

Output of k0s sysinfo in Debian v12 wp-1 container on the same LXD host with the same k8s LXD profile:

Total memory: 964.7 MiB (warning: 1.0 GiB recommended)
File system of /var/lib: unknown (warning)
Disk space available for /var/lib/k0s: 3.3 GiB (pass)
Relative disk space available for /var/lib/k0s: 77% (pass)
Name resolution: localhost: [127.0.0.1 ::1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.151-gnu (pass)
  Max. file descriptors per process: current: 65536 / max: 65536 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 1 (pass)
    cgroup controller "cpu": available (pass)
    cgroup controller "cpuacct": available (pass)
    cgroup controller "cpuset": available (pass)
    cgroup controller "memory": available (pass)
    cgroup controller "devices": available (pass)
    cgroup controller "freezer": available (pass)
    cgroup controller "pids": available (pass)
    cgroup controller "hugetlb": available (pass)
    cgroup controller "blkio": available (pass)
  CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
  CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
  CONFIG_NET: Networking support: no kernel config found (warning)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
  CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)

Several times when I tried to install with option --enable-worker, kublet could not run later after k0s start.
In htop I could see kublet tried to start and then terminated soon after starting many times in a cycle.

Today I tried following:

k0s install controller --single --force

It seems, it works:

k0s kubectl get node
NAME   STATUS   ROLES    AGE   VERSION
wp-1   Ready    <none>   30s   v1.31.1+k0s

And even able to install Prometheus stack and Grafana works.

Not sure, what happened, why it works now.
I tried to disable distro containerd by systemctl, stopped it (virtually "rebooted" LXD container).
And after that (not sure if it is related) k0s began to work.
Then I tried to start distro containerd again for a test, but k0s still works fine even with it.

May be k0s install controller --enable-worker (which I used earlier) does not work, will test it later again.
Do I understand correct that a node installed by k0s install controller --enable-worker command runs both control plane and work load and able to join more worker nodes? So without additional nodes joined yet, it is like a --single ?

[a-prokopyev-resume]

Btw., what do you think about making your own provisioning script (k0sctl, Ansible or Terraform provider) to install a k0s cluster in Docker containers? I think that installation of docker images is more predictable than trying to install your k0s on different Linux distribution.

Something like a k0s inside Docker Swarm, I even have seen some Ansible playbooks to install something like this on Github, but they seem to be over-complicated:

https://github.com/zengzhengrong/k0s-stack

https://github.com/tldr-devops/k0s-in-docker

When performance is not very important but needing a real multi-node k0s cluster then why not using many Docker containers on many different physical hosts to build a k0s cluster?

[ricardomaraschini]

I have been able to install k0s on lxd containers using the following raw.lxc configuration in the profile:

lxc.apparmor.profile=unconfined
lxc.mount.auto=proc:rw sys:rw cgroup:rw
lxc.cgroup.devices.allow=a
lxc.cap.drop=

Unfortunately I don't have the actual profile YAML but this is the Go code that generates the profile and I hope you can infer the YAML out of this.

	request := api.ProfilesPost{
		Name: fmt.Sprintf("profile-%s", in.id),
		ProfilePut: api.ProfilePut{
			Description: "test",
			Config: map[string]string{
				"raw.lxc":              profileConfig,
				"security.nesting":     "true",
				"security.privileged":  "true",
				"linux.kernel_modules": "br_netfilter,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay",
			},
			Devices: map[string]map[string]string{
				"kmsg": {
					"path":   "/dev/kmsg",
					"source": "/dev/kmsg",
					"type":   "unix-char",
				},
			},
		},
	}

Note

profileConfig is what I referred before as raw.lxc.