Adding talos support

[wasurerarenai]

Thank you @onedr0p for adding initial talos support to the template repository.

Running through the install procedure testing I ran into a few unexpected issues. I'm working on resolving these in my fork and expect to have a PR available soon.

1. The configure task requires a username be defined for nodes in config.yaml but talos doesn't allow authenticating as an OS level user. This should likely be updated to remove the username check for repositories configured for the Talos distribution.
2. The configure task requires a diskSerial value be defined nodes in config.yaml but not all disks provide one at /sys/block/<dev>/serial (from Talos docs).
3. The talconfig.yaml.j2 file disables DHCP, but doesn't set a static IP assignment. This was preventing the "vip" address from being used for me.
4. The talconfig.yaml.j2 assumes we are not using Predictable Interface Names and should be adding the net.ifnames=0 kernel argument. This can be set by talhelper in a schematic with controlPlane.schematic.customization.extraKernelArgs: - net.ifnames=0 but not sure how portable this would be.
5. The talos:kubeconfig task fails with a nodes are not set for the command: please use --nodes flag or configuration file to set the nodes to run the command against error.
6. The talos:apply-extras task fails with error: must build at directory: not a valid directory: evalsymlink failure on './cni' : lstat <path/to/repo>/kubernetes/talos/cni: no such file or directory and error: no objects passed to apply errors.

[wasurerarenai]

To resolve 2. I created the following two commits to add a "diskDevice" variable for nodes meant to pass a /dev/sda or similar to the "installDisk" talconfig.yaml setting. This was needed in my testing when I found disks not exposing a serial number for an attached disk. This was found in Proxmox Qemu VMs and one Intel NUC5 on-metal host install.

https://github.com/Wasurerarenai/flux-cluster-template/commit/1dc1fe3ec32ebe3d07827de1bbcab430a5698bed
https://github.com/Wasurerarenai/flux-cluster-template/commit/3fbdb34481fdc2fa487e789c17fc7d9385693416

[onedr0p]

Maybe the nodes need a talos section? e.g.

bootstrap_nodes:
  # Use only 1, 3 or more odd master nodes, recommended is 3
  master:
    # - name:       # name or hostname of the master node (must match [a-z0-9-\.]+)
    #   address:    # ip address of the master node
    #   username:   # ssh username of the master node (omit for talos)
    #   talos:
    #     diskDevice: # device, or
    #     # diskSerial: # serial
    # ...

[wasurerarenai]

Would be nice to be able to drop other Talos node-specific yaml under the talos: key. Looking to extend the talconfig.yaml.j2 file with something that can drop a item.talos.nodes_yaml or similar block into the talconfig.yaml nodes equivalent section. Doesn't look like to_yaml is usable here from testing.

bootstrap_nodes:
  # Use only 1, 3 or more odd master nodes, recommended is 3
  master:
    - name: talos-1
      talos:
        nodes_yaml:
...
          schematic:
            customization:
              extraKernelArgs:
                - net.ifnames=0

can translate to:

nodes:
  - hostname: "talos-1"
...
    schematic:
      customization:
        extraKernelArgs:
          - net.ifnames=0

[wasurerarenai]

The controlPlane: and workers: sections may also be extended without overwriting via reapplying the task configure task. Following https://github.com/budimanjojo/talhelper/blob/master/example/talconfig.yaml#L134 we can add a talconfig-controlPlane-patches.yaml and talconfig-worker-patches.yaml (or similarly named/stored in a patches directory) inclusion.

controlPlane:
  patches:
...
    - "@./talconfig-controlPlane-patches.yaml"
worker:
  patches:
...
    - "@./talconfig-worker-patches.yaml"

[wasurerarenai]

After the config refactor, I've added talos node-specific YAML processing with a simple block read. Since this is custom for the user, not much validation is needed from the template repo's scripts. The block was the simplest without needing to add a to_yaml or similar function for the jinja templating. I'll be submitting a PR for this one.

      - name: "talos-1"
...
        talos_disk_device: "/dev/sda"
        talos_node_config: |-
          schematic:
            customization:
              systemExtensions:
                officialExtensions:
                  - siderolabs/intel-ucode

wasurerarenai@593856e
wasurerarenai@ebeb3d0

[wasurerarenai]

PR submitted. #1273

[wasurerarenai]

To resolve 3. I created https://github.com/Wasurerarenai/flux-cluster-template/commit/34c349d3453b99d523b3ba2107bbb1c7d6c88026 which uses your existing node.address field and adds it as a static IP address defined for the default eth0 network interface. As long as the net.ifnames=0 kernel argument is set, this got the network device selector matching correctly and allowed the Virtual IP to be successfully brought up. Without out it I couldn't continue complete the talos:apply-extras task as kubectl was unable to access the VIP configured in bootstrap_kube_api_addr of the config.yaml file.

[wasurerarenai]

To workaround 5. I created https://github.com/Wasurerarenai/flux-cluster-template/commit/7441aad1af6d87f94abbd60d7eefb5d41db06ed1 which isn't perfect and requires the user to run task talos:kubeconfig node=<ip_address> but gives the user a hint towards whats needed to successfully complete that step. We might be able to use a loop or similar to try each of the already configured bootstrap_nodes.master[].address values until one is generated.

[wasurerarenai]

Not sure if this makes anything messy for other distribution support, but I added https://github.com/Wasurerarenai/flux-cluster-template/commit/e0d8895669ec20b92ecca8910eedc2025805b2cf which allows running talosctl commands (while the shell is in the repository directory or subdirectory) without needing to add the --talosconfig <path/to/repo/kubernetes/talos/clusterconfig/talosconfig> argument to allow the client to authenticate with the nodes. I noticed this adding a distribution-specific section for the .envrc and thought this would ease things for talos users.

[onedr0p]

That would be great! The .envrc can have anything for any distro. If they don't use talos it will simply be exported to the env and ignored.

[wasurerarenai]

To resolve 6. I created https://github.com/Wasurerarenai/flux-cluster-template/commit/257a2e13ef1b9f816eee66b9b60b6880e7ed0090 to use the kubernetes/talos/cilium/ directory instead of kubernetes/talos/cni/.

[onedr0p]

I have a PR up that will probably get merged soon.

#1253

[wasurerarenai]

Thanks. Successfully rolled a test cluster start to finish using the new updates.

[wasurerarenai]

Linking in Talos system extensions issue #1248