Skip to content

Commit 3699457

Browse files
matofedermichal-gubrickychess-knight
authored
✨ Add zuul e2e for openstack (#53)
* Add zuul e2e for openstack Signed-off-by: Matej Feder <[email protected]> * Install and build csctl-openstack from separate repository Signed-off-by: michal.gubricky <[email protected]> * Use openstack-csp-helper chart Signed-off-by: michal.gubricky <[email protected]> * Add cluster-stack template Signed-off-by: michal.gubricky <[email protected]> * Add cluster template Signed-off-by: michal.gubricky <[email protected]> * Add capo_version into variables Signed-off-by: michal.gubricky <[email protected]> * Cleanup workload cluster Signed-off-by: michal.gubricky <[email protected]> * Update csctl plugin branch Signed-off-by: michal.gubricky <[email protected]> * Add block in ansible to create workload cluster and execute checks Signed-off-by: michal.gubricky <[email protected]> * Import sonobuoy tests Signed-off-by: michal.gubricky <[email protected]> * Import scs-compliance pre-tasks and tasks Signed-off-by: michal.gubricky <[email protected]> * Debug sonobouy retrieve error in zuul pipeline Signed-off-by: michal.gubricky <[email protected]> * Enable scs-compliance tests in zuul Signed-off-by: michal.gubricky <[email protected]> * Increase pause to be able to retrieve results from sonobuoy Signed-off-by: michal.gubricky <[email protected]> * Ensure that pip3 is installed Signed-off-by: michal.gubricky <[email protected]> * Add zuul pipeline timeout Signed-off-by: michal.gubricky <[email protected]> * Wait for sonobuoy results and insert them to the warning message Signed-off-by: michal.gubricky <[email protected]> * Change how to get kubeconfig of the workload cluster Signed-off-by: michal.gubricky <[email protected]> * Increase wait_for_cluster timeout Signed-off-by: michal.gubricky <[email protected]> * Override cluster stack if defined in Zuul config Signed-off-by: michal.gubricky <[email protected]> * Add server groups to nodes Signed-off-by: michal.gubricky <[email protected]> * Use helm to create secrets and ClusterResourceSet Signed-off-by: michal.gubricky <[email protected]> * Import task for creating a server groups Signed-off-by: michal.gubricky <[email protected]> * Use helm via shell to create secrets and ClusterResourceSet Signed-off-by: michal.gubricky <[email protected]> * Change zuul config and also organization to SCS Signed-off-by: michal.gubricky <[email protected]> * Update .ansible-lint Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update .zuul.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update .zuul.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update .zuul.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/tasks/create_server_groups.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/openstack/templates/cluster-stack-template.yaml.j2 Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/dependencies.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Update playbooks/openstack/e2e.yaml Co-authored-by: Roman Hros <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> * Fix zuul config Signed-off-by: michal.gubricky <[email protected]> * Remove unnecessary --os-compute-api-version flag Signed-off-by: michal.gubricky <[email protected]> * Remove hardcoded timeouts and use variables instead Signed-off-by: michal.gubricky <[email protected]> * Parse k8s version to format major.minor Signed-off-by: michal.gubricky <[email protected]> * Parse cluster_stack_version_name from zuul config Signed-off-by: michal.gubricky <[email protected]> * Wait for clusteraddons resource to become ready Signed-off-by: michal.gubricky <[email protected]> * Workaround until the csctl binary can be built from the main branch again Signed-off-by: michal.gubricky <[email protected]> * Add some retries to the task called Get kubeadmcontrolplane name Signed-off-by: michal.gubricky <[email protected]> * Add cleanup run into the pipeline Signed-off-by: michal.gubricky <[email protected]> * Add wait for control-plane machines to be ready Signed-off-by: michal.gubricky <[email protected]> * Add cloud_name variable to cleanup.yaml playbook Signed-off-by: michal.gubricky <[email protected]> * Install csctl and csctl-openstack via releases Signed-off-by: michal.gubricky <[email protected]> * Fix retries keyword issue for ansible versions older then 9 Signed-off-by: michal.gubricky <[email protected]> * Add cleanup into always block as well Signed-off-by: michal.gubricky <[email protected]> --------- Signed-off-by: Matej Feder <[email protected]> Signed-off-by: michal.gubricky <[email protected]> Signed-off-by: Michal Gubricky <[email protected]> Co-authored-by: michal.gubricky <[email protected]> Co-authored-by: Roman Hros <[email protected]>
1 parent 114ea2b commit 3699457

19 files changed

+908
-0
lines changed

‎.ansible-lint

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
use_default_rules: true
3+
skip_list:
4+
- yaml # disabled because we use yamllint
5+
# Roles and modules imported from https://opendev.org/zuul/zuul-jobs
6+
mock_roles:
7+
- ensure-docker
8+
- ensure-go
9+
mock_modules:
10+
- zuul_return

‎.github/workflows/ansible-lint.yml

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
name: Ansible lint
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
paths:
9+
- 'playbooks/**'
10+
pull_request:
11+
paths:
12+
- 'playbooks/**'
13+
14+
jobs:
15+
build:
16+
name: Ansible Lint
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v4
20+
- name: Run ansible-lint
21+
uses: ansible/ansible-lint@v24

‎.github/workflows/check-yaml-syntax.yml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
name: Check yaml syntax
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
paths:
9+
- '**.yaml'
10+
- '**.yml'
11+
pull_request:
12+
paths:
13+
- '**.yaml'
14+
- '**.yml'
15+
16+
jobs:
17+
check-yaml-syntax:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v4
21+
- uses: actions/setup-python@v5
22+
with:
23+
python-version: '3.x'
24+
- run: pip3 install yamllint
25+
- run: yamllint .

‎.yamllint.yml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
extends: default
3+
4+
rules:
5+
comments: enable
6+
line-length: disable
7+
# accept both key:
8+
# - item
9+
# and key:
10+
# - item
11+
# (the latter is very common in k8s land)
12+
indentation:
13+
indent-sequences: whatever

‎.zuul.yaml

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
- job:
3+
name: openstack-e2e-abstract
4+
abstract: true
5+
parent: openstack-access-base
6+
description: |
7+
An abstract job for e2e testing of cluster stacks project.
8+
This job is not intended to be run directly, but instead
9+
must be inherited from it.
10+
pre-run: playbooks/dependencies.yaml
11+
run: playbooks/openstack/e2e.yaml
12+
cleanup-run: playbooks/openstack/cleanup.yaml # executed also when the job is canceled
13+
vars:
14+
wait_for_cluster_stack_resource: 120 # 2min
15+
wait_for_clusteraddons: 120 # 2min
16+
wait_for_cluster_stack: 1440 # 24min
17+
wait_for_cluster: 600 # 10min
18+
sonobouy:
19+
enabled: false
20+
scs_compliance:
21+
enabled: false
22+
23+
- job:
24+
name: e2e-openstack-conformance
25+
parent: openstack-e2e-abstract
26+
description: |
27+
Run e2e tests of cluster-stacks project using
28+
[sonobuoy](https://sonobuoy.io/) with mode conformance and
29+
SCS compliance checks meaning it will test if the Kubernetes
30+
cluster is conformant to the CNCF and to the SCS.
31+
timeout: 10800 # 3h
32+
vars:
33+
wait_for_cluster: 1200 # 20min
34+
sonobouy:
35+
enabled: true
36+
mode: conformance
37+
scs_compliance:
38+
enabled: true
39+
40+
- job:
41+
name: e2e-openstack-quick
42+
parent: openstack-e2e-abstract
43+
description: |
44+
Run e2e tests of cluster-stacks project using
45+
[sonobuoy](https://sonobuoy.io/) with mode quick and
46+
SCS compliance checks.
47+
timeout: 7200 # 2h
48+
vars:
49+
wait_for_cluster: 1200 # 20min
50+
sonobouy:
51+
enabled: true
52+
mode: quick
53+
scs_compliance:
54+
enabled: true
55+
56+
57+
- project:
58+
name: SovereignCloudStack/cluster-stacks
59+
default-branch: main
60+
merge-mode: "squash-merge"
61+
e2e-test:
62+
jobs:
63+
- e2e-openstack-conformance
64+
unlabel-on-update-e2e-test:
65+
jobs:
66+
- noop
67+
e2e-quick-test:
68+
jobs:
69+
- e2e-openstack-quick
70+
unlabel-on-update-e2e-quick-test:
71+
jobs:
72+
- noop

‎playbooks/dependencies.yaml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
- name: Ensure cluster stacks dependencies
3+
hosts: all
4+
vars:
5+
kind_version: "0.22.0"
6+
kubectl_version: "1.29.3"
7+
clusterctl_version: "1.7.2"
8+
helm_version: "3.14.4"
9+
yq_version: "4.44.1"
10+
envsubst_version: "1.4.2"
11+
install_dir: "{{ ansible_user_dir }}/.local/bin"
12+
roles: # https://opendev.org/zuul/zuul-jobs
13+
- role: ensure-docker
14+
- role: ensure-go
15+
vars:
16+
go_version: 1.21.6
17+
environment:
18+
PATH: "{{ install_dir }}:{{ ansible_env.PATH }}"
19+
tasks:
20+
- name: Make sure installation directory exists
21+
ansible.builtin.file:
22+
path: "{{ install_dir }}"
23+
state: directory
24+
mode: 0755
25+
- name: Install clusterctl
26+
ansible.builtin.get_url:
27+
url: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v{{ clusterctl_version }}/clusterctl-linux-amd64"
28+
dest: "{{ install_dir }}/clusterctl"
29+
mode: "+x"
30+
- name: Install envsubst
31+
ansible.builtin.get_url:
32+
url: "https://github.com/a8m/envsubst/releases/download/v{{ envsubst_version }}/envsubst-Linux-x86_64"
33+
dest: "{{ install_dir }}/envsubst"
34+
mode: "+x"
35+
- name: Install yq
36+
ansible.builtin.get_url:
37+
url: "https://github.com/mikefarah/yq/releases/download/v{{ yq_version }}/yq_linux_amd64"
38+
dest: "{{ install_dir }}/yq"
39+
mode: "+x"
40+
- name: Install KinD
41+
ansible.builtin.get_url:
42+
url: "https://kind.sigs.k8s.io/dl/v{{ kind_version }}/kind-linux-amd64"
43+
dest: "{{ install_dir }}/kind"
44+
mode: "+x"
45+
- name: Install kubectl
46+
ansible.builtin.get_url:
47+
url: "https://dl.k8s.io/release/v{{ kubectl_version }}/bin/linux/amd64/kubectl"
48+
dest: "{{ install_dir }}/kubectl"
49+
mode: "+x"
50+
# TODO: Install csctl and csctl-openstack from the release when it will be available
51+
- name: Install csctl and csctl-openstack
52+
ansible.builtin.import_tasks: tasks/csctl.yaml
53+
- name: Install helm
54+
ansible.builtin.unarchive:
55+
src: "https://get.helm.sh/helm-v{{ helm_version }}-linux-amd64.tar.gz"
56+
dest: "{{ install_dir }}"
57+
extra_opts: "--strip-components=1"
58+
mode: "+x"
59+
remote_src: true
60+
args:
61+
creates: "{{ install_dir }}/helm"

‎playbooks/openstack/cleanup.yaml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
- name: Cleanup
3+
hosts: all
4+
vars:
5+
cloud_name: "{{ cloud }}" # inherited from the parent job
6+
environment:
7+
PATH: "{{ ansible_user_dir }}/.local/bin:{{ ansible_env.PATH }}"
8+
tasks:
9+
- name: Delete server groups
10+
when: scs_compliance.enabled
11+
block:
12+
- name: List existing server groups
13+
ansible.builtin.command: "openstack server group list -f value -c Name -c ID"
14+
register: server_groups
15+
environment:
16+
OS_CLOUD: "{{ cloud_name }}"
17+
changed_when: true
18+
- name: Parse test-cluster-controller srvgrp and assign ID to srvgrp_controller
19+
ansible.builtin.set_fact:
20+
srvgrp_controller: "{{ item.split(' ')[0] }}"
21+
loop: "{{ server_groups.stdout_lines }}"
22+
when: "server_groups is defined and server_groups.stdout_lines | length > 0 and 'test-cluster-controller' in item.split(' ')"
23+
- name: Parse test-cluster-worker srvgrp and assign ID to srvgrp_worker
24+
ansible.builtin.set_fact:
25+
srvgrp_worker: "{{ item.split(' ')[0] }}"
26+
loop: "{{ server_groups.stdout_lines }}"
27+
when: "server_groups is defined and server_groups.stdout_lines | length > 0 and 'test-cluster-worker' in item.split(' ')"
28+
- name: Delete Server Group for worker nodes
29+
ansible.builtin.command: "openstack server group delete {{ srvgrp_worker }}"
30+
environment:
31+
OS_CLOUD: "{{ cloud_name }}"
32+
when: srvgrp_worker is defined
33+
changed_when: true
34+
- name: Delete Server Group for control-plane nodes
35+
ansible.builtin.command: "openstack server group delete {{ srvgrp_controller }}"
36+
environment:
37+
OS_CLOUD: "{{ cloud_name }}"
38+
when: srvgrp_controller is defined
39+
changed_when: true
40+
- name: Check if test-cluster exists
41+
ansible.builtin.command: "kubectl get cluster test-cluster"
42+
register: cluster_check
43+
ignore_errors: true
44+
changed_when: true
45+
- name: Cleanup workload cluster
46+
ansible.builtin.command: "kubectl delete -f {{ ansible_user_dir }}/cluster.yaml"
47+
when: cluster_check.rc == 0
48+
changed_when: true

0 commit comments

Comments
 (0)