Skip to content

Commit d093d0a

Browse files
brunnelsonedr0p
andauthored
add k0s support (#1123)
* ignore idea * Initial work on support for k0s * * Changed config param to bootstrap_distribution * Replaced democratic-csi with openebs * Added notes about system-upgrade-controller being ignored with k0s * Removed password for nodes * * Switched to single storage path for openebs * Removed democratic-csi helm repo * Added missing openebs helm repo * Fixed template formatting * Made bootstrap_private_github_repo hidden/undocumented * Updated flux cluster config to accommodate bootstrap_private_github_repo * * added the storageclass name to be declarative * * added newline * set chart wait false * * fixed spacing in K0sTasks * fixed openebs path * properly added openebs helm repo * * updated configure task to use venv * renamed bootstrap_kube_vip_addr to bootstrap_kubeapi_addr * renamed bootstrap_kube_vip_addr derived variables accordingly * * tweaked configure task cmd * started work on k0s-config.yaml template * * tweaked k0s task cmd * moved coredns under addons rather than optional * fixed typo in coredns task * normalized formatting in k0s-config * disabled telemetry in k0s-config * added renovate to k0s-config * removed tunnel and added routing mode in k0s-config cilium chart values * updated docs in config example * fixed issue with k0s-config controller role so it didn't default to worker+controller if more than one master * * fixed whitespace * add k0sctl to brew tasks * Update bootstrap/vars/config.sample.yaml * Update bootstrap/vars/config.sample.yaml * Update k0s-config.yaml.j2 * Update config.sample.yaml * controllerManager and scheduler bind to `0.0.0.0` for metrics * Update custom-cilium-helmchart.yaml.j2.j2 * Update helmvalues.yaml.j2 * Update bootstrap/tasks/addons/main.yaml * first pass at readme updates * update placement of k0sconfig options * disable telemetry on k0sctl too * disable k0s telemetry in env too * update: jinja spacing in k0sconfig Signed-off-by: Devin Buhl <[email protected]> * fix: ensure bootstrap_local_storage_path is delete on nuke Signed-off-by: Devin Buhl <[email protected]> * fix: update renovate regex for k3s/k0s Signed-off-by: Devin Buhl <[email protected]> * fix: update nuke playbook and only include k3s task when k3s Signed-off-by: Devin Buhl <[email protected]> * fix: update nuke playbook and only include k3s task when k3s and include jinja comments Signed-off-by: Devin Buhl <[email protected]> --------- Signed-off-by: Devin Buhl <[email protected]> Co-authored-by: Devin Buhl <[email protected]> Co-authored-by: Devin Buhl <[email protected]>
1 parent 1b73543 commit d093d0a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+355
-153
lines changed

.envrc

+2
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,5 @@ export ANSIBLE_COLLECTIONS_PATH=$(expand_path ./.venv/galaxy)
88
export ANSIBLE_ROLES_PATH=$(expand_path ./.venv/galaxy/ansible_roles)
99
export ANSIBLE_VARS_ENABLED="host_group_vars,community.sops.sops"
1010
export K8S_AUTH_KUBECONFIG="$(expand_path ./kubeconfig)"
11+
# k0s
12+
export DISABLE_TELEMETRY="true"

.github/renovate.json5

+3-3
Original file line numberDiff line numberDiff line change
@@ -78,10 +78,10 @@
7878
},
7979
// custom versioning
8080
{
81-
"description": "Use custom versioning for k3s",
81+
"description": "Use custom versioning for k0s/k3s",
8282
"matchDatasources": ["github-releases"],
83-
"versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)(?<compatibility>\\+k3s)(?<build>\\d+)$",
84-
"matchPackagePatterns": ["k3s"]
83+
"versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)(?<compatibility>\\+k.s)\\.?(?<build>\\d+)$",
84+
"matchPackagePatterns": ["k0s", "k3s"]
8585
},
8686
// commit message topics
8787
{

.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,6 @@ kubeconfig
1212
.venv*
1313
# Taskfile
1414
.tasks
15+
# intellij
16+
.idea
17+

.taskfiles/BrewTasks.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ tasks:
2020
fluxcd/tap/flux
2121
helm
2222
jq
23+
k0sproject/tap/k0sctl
2324
k9s
2425
kubernetes-cli
2526
kustomize

.taskfiles/FluxTasks.yaml

+7-1
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,15 @@ tasks:
2626
- kubectl apply --server-side --filename {{.KUBERNETES_DIR}}/flux/vars/cluster-settings-user.yaml
2727
- kubectl apply --server-side --kustomize {{.KUBERNETES_DIR}}/flux/config
2828
preconditions:
29-
- { msg: "Flux already appears installed", sh: "exit $(( ! $(kubectl get namespace flux-system) ))" }
29+
- { msg: "Flux already appears installed", sh: "kubectl get namespace flux-system &>/dev/null && exit 1 || exit 0" }
3030
- { msg: "Age private key not found", sh: "test -f {{.ROOT_DIR}}/age.key" }
3131

32+
github-deploy-key:
33+
cmds:
34+
- sops --decrypt {{.KUBERNETES_DIR}}/bootstrap/github-deploy-key.sops.yaml | kubectl apply -f -
35+
preconditions:
36+
- { msg: "Flux is not installed", sh: "kubectl get namespace flux-system &>/dev/null && exit 0 || exit 1" }
37+
3238
apply:
3339
desc: Apply a Flux Kustomization resource for a cluster
3440
summary: |

.taskfiles/K0sTasks.yaml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
version: "3"
3+
4+
env:
5+
DISABLE_TELEMETRY: "true"
6+
7+
tasks:
8+
9+
kubeconfig:
10+
desc: Gets k0s cluster kubeconfig
11+
cmds:
12+
- k0sctl kubeconfig --config k0s-config.yaml > kubeconfig
13+
preconditions:
14+
- { msg: "k0s-config.yaml not found", sh: "test -f {{.ROOT_DIR}}/k0s-config.yaml" }
15+
16+
apply:
17+
desc: Apply k0s cluster k0s-config.yaml
18+
cmds:
19+
- k0sctl apply --config k0s-config.yaml
20+
- task: kubeconfig
21+
preconditions:
22+
- { msg: "k0s-config.yaml not found", sh: "test -f {{.ROOT_DIR}}/k0s-config.yaml" }
23+
24+
reset:
25+
desc: Resets the k0s cluster
26+
cmd: k0sctl reset --config k0s-config.yaml
27+
preconditions:
28+
- { msg: "k0s-config.yaml not found", sh: "test -f {{.ROOT_DIR}}/k0s-config.yaml" }

README.md

+9-6
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Deploy a Kubernetes cluster backed by Flux
22

3-
Welcome to my highly opinionated template for deploying a single Kubernetes ([k3s](https://k3s.io)) cluster with [Ansible](https://www.ansible.com) and using [Flux](https://toolkit.fluxcd.io) to manage its state.
3+
Welcome to my highly opinionated template for deploying a single Kubernetes ([k3s](https://k3s.io) or [k0s](https://github.com/k0sproject/k0s)) cluster with [Ansible](https://www.ansible.com) and using [Flux](https://toolkit.fluxcd.io) to manage its state.
44

55
## 👋 Introduction
66

@@ -258,7 +258,7 @@ Once you have installed Debian on your nodes, there are six stages to getting a
258258
> └─📁 apps # Apps deployed into the cluster grouped by namespace
259259
> ```
260260
261-
### ⚡ Stage 4: Prepare your nodes for k3s
261+
### ⚡ Stage 4: Prepare your nodes for Kubernetes
262262
263263
📍 _Here we will be running an Ansible playbook to prepare your nodes for running a Kubernetes cluster._
264264
@@ -282,7 +282,7 @@ Once you have installed Debian on your nodes, there are six stages to getting a
282282
task ansible:run playbook=cluster-prepare
283283
```
284284
285-
### ⛵ Stage 5: Use Ansible to install k3s
285+
### ⛵ Stage 5: Install Kubernetes
286286
287287
📍 _Here we will be running a Ansible Playbook to install [k3s](https://k3s.io/) with [this](https://galaxy.ansible.com/xanmanning/k3s) Ansible galaxy role. If you run into problems, you can run `task ansible:run playbook=cluster-nuke` to destroy the k3s cluster and start over from this point._
288288
@@ -298,13 +298,16 @@ Once you have installed Debian on your nodes, there are six stages to getting a
298298
task ansible:ping
299299
```
300300
301-
3. Install k3s with Ansible
301+
3. Install Kubernetes
302302
303303
```sh
304+
# k3s
304305
task ansible:run playbook=cluster-installation
306+
# k0s
307+
task k0s:apply
305308
```
306309
307-
4. Verify the nodes are online
310+
5. Verify the nodes are online
308311
309312
📍 _If this command **fails** you likely haven't configured `direnv` as mentioned previously in the guide._
310313
@@ -315,7 +318,7 @@ Once you have installed Debian on your nodes, there are six stages to getting a
315318
# k8s-1 Ready worker 1h v1.27.3+k3s1
316319
```
317320
318-
5. The `kubeconfig` for interacting with your cluster should have been created in the root of your repository.
321+
6. The `kubeconfig` for interacting with your cluster should have been created in the root of your repository.
319322
320323
### 🔹 Stage 6: Install Flux in your cluster
321324

Taskfile.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ includes:
1717
aliases: ["k8s"]
1818
taskfile: .taskfiles/KubernetesTasks.yaml
1919
flux: .taskfiles/FluxTasks.yaml
20+
k0s: .taskfiles/K0sTasks.yaml
2021

2122
tasks:
2223

@@ -45,7 +46,6 @@ tasks:
4546
configure:
4647
desc: Configure repository from Ansible vars
4748
prompt: Any conflicting config in the root kubernetes and ansible directories will be overwritten... continue?
48-
dir: "{{.BOOTSTRAP_DIR}}"
49-
cmd: ansible-playbook configure.yaml
49+
cmd: ./.venv/bin/ansible-playbook {{.BOOTSTRAP_DIR}}/configure.yaml
5050
env:
5151
ANSIBLE_DISPLAY_SKIPPED_HOSTS: "false"

bootstrap/configure.yaml

+7-3
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,22 @@
1717
ansible.builtin.set_fact:
1818
repository_path: "{{ repository.stdout }}"
1919

20-
- name: Override kube-vip address when there is a single master node and no address is defined
21-
when: bootstrap_nodes.master | length == 1 and not bootstrap_kube_vip_addr
20+
- name: Override kubeapi address when there is a single master node and no address is defined
21+
when: bootstrap_nodes.master | length == 1 and not bootstrap_kubeapi_addr
2222
ansible.builtin.set_fact:
2323
bootstrap_kube_vip_enabled: false
24-
bootstrap_kube_vip_addr: "{{ bootstrap_nodes.master[0].address }}"
24+
bootstrap_kubeapi_addr: "{{ bootstrap_nodes.master[0].address }}"
2525

2626
- name: Verify configuration
2727
ansible.builtin.include_tasks: tasks/validation/main.yaml
2828

2929
- name: Template Sops configuration
3030
ansible.builtin.include_tasks: tasks/sops/main.yaml
3131

32+
- name: Template k0s configuration
33+
when: bootstrap_distribution == "k0s"
34+
ansible.builtin.include_tasks: tasks/k0s/main.yaml
35+
3236
- name: Template Ansible configuration
3337
ansible.builtin.include_tasks: tasks/ansible/main.yaml
3438

bootstrap/tasks/addons/coredns.yaml

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
- name: Set addon facts
3+
ansible.builtin.set_fact:
4+
addon_name: coredns
5+
addon_namespace: kube-system
6+
7+
- name: Ensure directories exist for {{ addon_namespace }}/{{ addon_name }}
8+
when: item.state == 'directory'
9+
ansible.builtin.file:
10+
path: "{{ repository_path }}/kubernetes/apps/{{ addon_namespace }}/{{ addon_name }}/{{ item.path }}"
11+
state: directory
12+
mode: "0755"
13+
with_community.general.filetree: ["../templates/addons/{{ addon_name }}/"]
14+
15+
- name: Template unencrypted files for {{ addon_namespace }}/{{ addon_name }}
16+
when: item.state == 'file' and 'sops' not in item.path
17+
ansible.builtin.template:
18+
src: "{{ item.src }}"
19+
dest: "{{ repository_path }}/kubernetes/apps/{{ addon_namespace }}/{{ addon_name }}/{{ item.path | replace('.j2', '') }}"
20+
mode: "0644"
21+
with_community.general.filetree: ["../templates/addons/{{ addon_name }}/"]
22+
23+
- name: Template encrypted files for {{ addon_namespace }}/{{ addon_name }}
24+
block:
25+
- name: Template encrypted files
26+
when: item.state == 'file' and 'sops' in item.path
27+
community.sops.sops_encrypt:
28+
path: "{{ repository_path }}/kubernetes/apps/{{ addon_namespace }}/{{ addon_name }}/{{ item.path | replace('.j2', '') }}"
29+
encrypted_regex: ^(data|stringData)$
30+
age: ["{{ bootstrap_age_public_key }}"]
31+
content_yaml: "{{ lookup('ansible.builtin.template', item.src) | from_yaml }}"
32+
mode: "0644"
33+
force: true
34+
with_community.general.filetree: ["../templates/addons/{{ addon_name }}/"]

bootstrap/tasks/addons/main.yaml

+7-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
---
22

3+
- name: Process optional coredns
4+
when: bootstrap_distribution == "k3s"
5+
ansible.builtin.include_tasks: coredns.yaml
6+
37
- name: Process addon csi-driver-nfs
48
when: csi_driver_nfs.enabled | default(false)
59
ansible.builtin.include_tasks: csi_driver_nfs.yaml
@@ -21,7 +25,9 @@
2125
ansible.builtin.include_tasks: kube_prometheus_stack.yaml
2226

2327
- name: Process addon system-upgrade-controller
24-
when: system_upgrade_controller.enabled | default(false)
28+
when:
29+
- bootstrap_distribution == "k3s"
30+
- system_upgrade_controller.enabled | default(false)
2531
ansible.builtin.include_tasks: system_upgrade_controller.yaml
2632

2733
- name: Process addon weave-gitops

bootstrap/tasks/ansible/main.yaml

+1-10
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,4 @@
2727
mode: "0644"
2828
force: true
2929
with_community.general.filetree: ["../templates/ansible/"]
30-
- name: Template encrypted node secrets
31-
community.sops.sops_encrypt:
32-
path: "{{ repository_path }}/ansible/inventory/host_vars/{{ item.name }}.sops.yaml"
33-
age: ["{{ bootstrap_age_public_key }}"]
34-
content_yaml: "{{ lookup('ansible.builtin.template', 'templates/node.sops.yaml.j2', template_vars=dict(password=item.password)) | from_yaml }}"
35-
mode: "0644"
36-
force: true
37-
loop: "{{ bootstrap_nodes.master + bootstrap_nodes.worker | default([]) }}"
38-
loop_control:
39-
label: "{{ item.address }}"
30+

bootstrap/tasks/k0s/main.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
- name: Template k0s configuration file
3+
ansible.builtin.template:
4+
src: "templates/k0s-config.yaml.j2"
5+
dest: "{{ repository_path }}/k0s-config.yaml"
6+
mode: "0644"

bootstrap/tasks/validation/github.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
fail_msg: Github user {{ bootstrap_github_username }} does not exist
1515

1616
- name: Query Github repo
17+
when: not bootstrap_private_github_repo | default(false)
1718
ansible.builtin.uri:
1819
url: https://api.github.com/repos/{{ bootstrap_github_username }}/{{ bootstrap_github_repository_name }}
1920
timeout: 5
@@ -22,12 +23,14 @@
2223
register: result
2324

2425
- name: Check if repo exists
26+
when: not bootstrap_private_github_repo | default(false)
2527
ansible.builtin.assert:
2628
that: result.json.full_name == bootstrap_github_username + '/' + bootstrap_github_repository_name
2729
success_msg: Github repo {{ bootstrap_github_username }}/{{ bootstrap_github_repository_name }} exists
2830
fail_msg: Github repo {{ bootstrap_github_username }}/{{ bootstrap_github_repository_name }} does not exist
2931

3032
- name: Query Github repo branch
33+
when: not bootstrap_private_github_repo | default(false)
3134
ansible.builtin.uri:
3235
url: https://api.github.com/repos/{{ bootstrap_github_username }}/{{ bootstrap_github_repository_name }}/branches/{{ bootstrap_github_repository_branch | default('main', true) }}
3336
timeout: 5
@@ -36,6 +39,7 @@
3639
register: result
3740

3841
- name: Check if repo branch exists
42+
when: not bootstrap_private_github_repo | default(false)
3943
ansible.builtin.assert:
4044
that: result.json.name == bootstrap_github_repository_branch | default('main', true)
4145
success_msg: Github repo branch {{ bootstrap_github_repository_branch | default('main', true) }} exists

bootstrap/tasks/validation/net.yaml

+14-14
Original file line numberDiff line numberDiff line change
@@ -98,17 +98,17 @@
9898
success_msg: external ingress address {{ bootstrap_external_ingress_addr }} is within {{ bootstrap_node_cidr }}.
9999
fail_msg: external ingress address {{ bootstrap_external_ingress_addr }} is not within {{ bootstrap_node_cidr }}.
100100

101-
- name: Verify kube-vip
101+
- name: Verify kubeapi address
102102
ansible.builtin.assert:
103-
that: bootstrap_kube_vip_addr is ansible.utils.ipv4
104-
success_msg: kube-vip address {{ bootstrap_kube_vip_addr }} is valid.
105-
fail_msg: kube-vip address {{ bootstrap_kube_vip_addr }} is invalid.
103+
that: bootstrap_kubeapi_addr is ansible.utils.ipv4
104+
success_msg: kubeapi address {{ bootstrap_kubeapi_addr }} is valid.
105+
fail_msg: kubeapi address {{ bootstrap_kubeapi_addr }} is invalid.
106106

107-
- name: Verify kube-vip in node CIDR
107+
- name: Verify kubeapi address in node CIDR
108108
ansible.builtin.assert:
109-
that: bootstrap_node_cidr | ansible.utils.network_in_usable(bootstrap_kube_vip_addr)
110-
success_msg: kube-vip address {{ bootstrap_kube_vip_addr }} is within {{ bootstrap_node_cidr }}.
111-
fail_msg: kube-vip address {{ bootstrap_kube_vip_addr }} is not within {{ bootstrap_node_cidr }}.
109+
that: bootstrap_node_cidr | ansible.utils.network_in_usable(bootstrap_kubeapi_addr)
110+
success_msg: kubeapi address {{ bootstrap_kubeapi_addr }} is within {{ bootstrap_node_cidr }}.
111+
fail_msg: kubeapi address {{ bootstrap_kubeapi_addr }} is not within {{ bootstrap_node_cidr }}.
112112

113113
- name: Verify all IP addresses are unique
114114
ansible.builtin.assert:
@@ -117,7 +117,7 @@
117117
bootstrap_k8s_gateway_addr,
118118
bootstrap_external_ingress_addr,
119119
bootstrap_internal_ingress_addr,
120-
bootstrap_kube_vip_addr
120+
bootstrap_kubeapi_addr
121121
] | unique | length == 4
122122
success_msg: All IP addresses are unique.
123123
fail_msg: All IP addresses are not unique.
@@ -133,12 +133,12 @@
133133
loop_control:
134134
label: "{{ item.address }}"
135135

136-
- name: Verify nodes are not the same IPs as k8s_gateway, ingress external/internal or kube-vip
137-
when: bootstrap_kube_vip_enabled | default(true)
136+
- name: Verify nodes are not the same IPs as k8s_gateway, ingress external/internal or kubeapi address
137+
when: (bootstrap_distribution == "k3s") and (bootstrap_kube_vip_enabled | default(true))
138138
ansible.builtin.assert:
139-
that: item.address not in (bootstrap_k8s_gateway_addr, bootstrap_external_ingress_addr, bootstrap_internal_ingress_addr, bootstrap_kube_vip_addr)
140-
success_msg: Node address {{ item.address }} is different than k8s_gateway, ingress-nginx or kube-vip.
141-
fail_msg: Node address {{ item.address }} is not different than k8s_gateway, ingress-nginx or kube-vip.
139+
that: item.address not in (bootstrap_k8s_gateway_addr, bootstrap_external_ingress_addr, bootstrap_internal_ingress_addr, bootstrap_kubeapi_addr)
140+
success_msg: Node address {{ item.address }} is different than k8s_gateway, ingress-nginx or kubeapi.
141+
fail_msg: Node address {{ item.address }} is not different than k8s_gateway, ingress-nginx or kubeapi.
142142
quiet: true
143143
loop: "{{ bootstrap_nodes.master + bootstrap_nodes.worker | default([]) }}"
144144
loop_control:

bootstrap/tasks/validation/vars.yaml

+6-5
Original file line numberDiff line numberDiff line change
@@ -16,23 +16,24 @@
1616
- bootstrap_cloudflare_tunnel_id
1717
- bootstrap_cloudflare_tunnel_secret
1818
- bootstrap_cluster_cidr
19+
- bootstrap_distribution
20+
- bootstrap_external_ingress_addr
1921
- bootstrap_flux_github_webhook_token
20-
- bootstrap_github_repository_name
2122
- bootstrap_github_repository_branch
23+
- bootstrap_github_repository_name
2224
- bootstrap_github_username
23-
- bootstrap_external_ingress_addr
2425
- bootstrap_internal_ingress_addr
2526
- bootstrap_ipv6_enabled
2627
- bootstrap_k8s_gateway_addr
27-
- bootstrap_kube_vip_addr
28-
- bootstrap_local_path_provisioner_path
28+
- bootstrap_kubeapi_addr
29+
- bootstrap_local_storage_path
2930
- bootstrap_node_cidr
3031
- bootstrap_service_cidr
3132
- bootstrap_timezone
3233

3334
- name: Verify bootstrap node names are valid
3435
ansible.builtin.assert:
35-
that: item.name is match('^[a-z0-9-]+$')
36+
that: item.name is match('^[a-z0-9-\.]+$')
3637
success_msg: Node name {{ item.name }} is valid
3738
fail_msg: Node name {{ item.name }} is not valid
3839
loop: "{{ bootstrap_nodes.master + bootstrap_nodes.worker | default([]) }}"

bootstrap/templates/addons/grafana/app/helmrelease.yaml.j2

+2-2
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ spec:
2424
uninstall:
2525
keepHistory: false
2626
dependsOn:
27-
- name: local-path-provisioner
27+
- name: openebs
2828
namespace: storage
2929
values:
3030
deploymentStrategy:
@@ -168,6 +168,6 @@ spec:
168168
- *host
169169
persistence:
170170
enabled: true
171-
storageClassName: local-hostpath
171+
storageClassName: openebs-hostpath
172172
testFramework:
173173
enabled: false

bootstrap/templates/addons/kube-prometheus-stack/app/helmrelease.yaml.j2

+1-1
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ spec:
2727
uninstall:
2828
keepHistory: false
2929
dependsOn:
30-
- name: local-path-provisioner
30+
- name: openebs
3131
namespace: storage
3232
valuesFrom:
3333
- name: kube-prometheus-stack-values

0 commit comments

Comments
 (0)