Failed to start k0s on arch fresh installation

[hehehenri]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.4.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 27 Jul 2023 22:02:18 +0000 x86_64 GNU/Linux
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Version

v1.27.4+k0s.0

Sysinfo

`k0s sysinfo`

Machine ID: "9053313309a029437f8acf5805b6351c0af0e835b945656df826f90d015fde8c" (from machine) (pass)
Total memory: 38.4 GiB (pass)
Disk space available for /var/lib/k0s: 735.0 GiB (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.4.7-arch1-1 (pass)
  Max. file descriptors per process: current: 524288 / max: 524288 (pass)
  Executable in path: modprobe: /usr/bin/modprobe (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (pass)
    cgroup controller "memory": available (pass)
    cgroup controller "devices": available (assumed) (pass)
    cgroup controller "freezer": available (assumed) (pass)
    cgroup controller "pids": available (pass)
    cgroup controller "hugetlb": available (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

Got error: status: can't do http request: /run/k0s/status.sock status after running sudo k0s status on a fresh install.

$ journalctl -u k0scontroller

Aug 06 23:37:25 archlinux k0s[170281]: Error: error detecting local IP: lookup localhost: Try again
Aug 06 23:37:25 archlinux systemd[1]: k0scontroller.service: Main process exited, code=exited, statu>
Aug 06 23:37:25 archlinux systemd[1]: k0scontroller.service: Failed with result 'exit-code'.

Steps to reproduce

1. curl -sSLf https://get.k0s.sh | sudo sh
2. sudo k0s install controller --single
3. sudo k0s start
4. sudo k0s status

Expected behavior

Version: v1.27.4+k0s.0
Process ID: 4315
Role: controller
Workloads: true
SingleNode: true
Kube-api probing successful: true
Kube-api probing last error:

Actual behavior

error: status: can't do http request: /run/k0s/status.sock status

Screenshots and logs

No response

Additional context

It worked after creating the config file

mkdir -p /etc/k0s
k0s config create > /etc/k0s/k0s.yaml

[jnummelin]

hmm, looking where the error stems from:

addrs, err := resolver.LookupIPAddr(ctx, "localhost")

That errors out for some reason. Is the machine configured to have any mapping for localhost?

[hehehenri]

hmm, looking where the error stems from:

addrs, err := resolver.LookupIPAddr(ctx, "localhost")

That errors out for some reason. Is the machine configured to have any mapping for localhost?

I thought about the same thing and added the localhost entry to /etc/hosts file, then reinstalled k0s from scratch again but I still got the same error running sudo k0s status.

This is my /etc/hosts

# Static table lookup for hostnames.
# See hosts(5) for details.

127.0.0.1 localhost

[kke]

The http error in k0s status just means k0s isn't running, it uses local unix sockets, no hostname lookup.

sudo k0s reset may be required between installation attempts.

The lookup happens when k0s tries to generate SAN certificate:

	hostnames := []string{
		"kubernetes",
		"kubernetes.default",
		"kubernetes.default.svc",
		"kubernetes.default.svc.cluster",
		fmt.Sprintf("kubernetes.svc.%s", c.ClusterSpec.Network.ClusterDomain),
		"localhost",
		"127.0.0.1",
	}

	localIPs, err := detectLocalIPs(ctx)
	if err != nil {
		return fmt.Errorf("error detecting local IP: %w", err) // <-- here the error is decorated
	}
	hostnames = append(hostnames, localIPs...)
	hostnames = append(hostnames, c.ClusterSpec.API.Sans()...)

        ....
func detectLocalIPs(ctx context.Context) ([]string, error) {
	resolver := net.DefaultResolver

	addrs, err := resolver.LookupIPAddr(ctx, "localhost") /// <-- this triggers the error
	if err != nil {
		return nil, err
	}

It appears k0s adds localhost and 127.0.0.1 anyway and whatever comes from detectLocalIPs and spec.api.sans (could contain duplicates).

This happens every time a k0s controller is started and having /etc/k0s/k0s.yaml or not shouldn't make any difference.

The Try again in the error message kind of hints that maybe k0s could retry the lookup.

[twz123]

A failure to lookup localhost smells a bit like a more general name resolution issue to me. Note that it's not resolving to an empty list of addresses (which would be fine for k0s), but it errors out with a hard error. What results do you get when trying to lookup localhost with other tools, say nslookup or dig? Do other names work, like your machine's hostname or, say, github.com? There are a variety of things that could influence name resolution. To name a few:

• some problematic settings in /etc/resolv.conf
• some problematic settings in /etc/nsswitch.conf
• some issues with systemd-resolved

I prepared PR #3366 that'll ignore and log the localhost resolution error. However, if resolving localhost continues to fail, I'm pretty sure that other components will fail as well. So not sure if the PR will help much. I quickly checked the sources, there are quite a few references to localhost, e.g. in some etcd certificates, konnectivity, kube-proxy, NLLB. They all assume that localhost resolves to something in the loopback range of 127.x.x.x.

[twz123]

I can repro this on an Arch Linux VM. So I have something to test on.

[twz123]

The error doesn't occur on main, I bisected it and found out that #3115 fixed it. Probably something in musl. @ncopa Do you maybe remember some noteworthy changes in musl that would affect name resolution of localhost that landed in Alpine 3.18?

Maybe this?

It also makes a number of other bug fixes and improvements in DNS and related functionality, including making both the modern and legacy API results differentiate between NODATA and NxDomain conditions so that the caller can handle them differently.

[twz123]

Note that I cannot repro this with the docker.io/library/archlinux Docker image. My host machine is running a 6.1 kernel, the Arch VM is running 6.4. I also tried it in another VM using Linux 6.4.8 #1-NixOS SMP PREEMPT_DYNAMIC Thu Aug 3 08:26:15 UTC 2023 x86_64 GNU/Linux. Works. So it seems to me that there is some peculiarity in the way Arch builds/configures the kernel that triggers this?

[ncopa]

I was able to reproduce this in arch linux vm as well. However, when I added 127.0.0.1 localhost to /etc/hosts it started up.

I also notice that the docker image has localhost in its /etc/hosts.

$ docker run --rm -it archlinux grep 
localhost /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback

I think this is related systemd's /etc/nsswitch.conf:

hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns

[ncopa]

This is the problem: https://www.openwall.com/lists/musl/2022/08/31/5 caused by bug in musl libc when there is a search . in /etc/resolv.conf. (introduced in systemd to workaround glibc behavior)

It does not look like the fix was backported to alpine 3.17. I suppose we should make sure we use alpine:3.18 which has the fix.