Create kubeconfig for GitHub Actions

[veaceslavdoina]

We decided to run tests inside Kubernetes via Job and it means that GitHub Actions will create just a Job inside the cluster.

We can do it via kubectl and manifest with bash variables. As it was discovered during Manual run inside Kubernetes via Job #7, Service Account used for that should have permissions to create

1. Namespace
2. Job

For now, we will run all test from a default namespace, because

• Network policies permit connection to the pods in the tests namespace only from a default one
• Before implementing Add a way to configure log files location #3, Option to set kubeConfigFile #8 and Option to set number of parallel tests #13 we will use a config file from configMap in a default namespace
• Before implementing Option to authenticate using Service Account #18, we will use a kubeconfig from the Secrets

[veaceslavdoina]

List API Resources

# kubectl api-resources --api-group=batch -o wide
kubectl api-resources -o wide | grep -e 'namespace\|NAME' -e job
NAME                               SHORTNAMES          APIVERSION                             NAMESPACED   KIND                             VERBS                                                        CATEGORIES
namespaces                         ns                  v1                                     false        Namespace                        create,delete,get,list,patch,update,watch
cronjobs                           cj                  batch/v1                               true         CronJob                          create,delete,deletecollection,get,list,patch,update,watch   all
jobs                                                   batch/v1                               true         Job                              create,delete,deletecollection,get,list,patch,update,watch   all

We may use the following manifest

sa-ci.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: cs-codex-dist-tests

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cs-codex-dist-tests-ci-sa
  namespace: cs-codex-dist-tests

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cs-codex-dist-tests-ci-role
  namespace: cs-codex-dist-tests
rules:
  - apiGroups:
      - "*"
    resources:
      - namespaces
    verbs:
      - create
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - get
      - create

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cs-codex-dist-tests-ci-role-binding
  namespace: cs-codex-dist-tests
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cs-codex-dist-tests-ci-role
subjects:
  - kind: ServiceAccount
    name: cs-codex-dist-tests-ci-sa
    namespace: cs-codex-dist-tests

---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: cs-codex-dist-tests-ci-token
  namespace: cs-codex-dist-tests
  labels:
    name: cs-codex-dist-tests
  annotations:
    kubernetes.io/service-account.name: cs-codex-dist-tests-ci-sa

kubectl apply -f sa-ci.yaml

Which will create

Resource	Name
Namespace	cs-codex-dist-tests
ServiceAccount	cs-codex-dist-tests-ci-sa
ClusterRole	cs-codex-dist-tests-ci-role
ClusterRoleBinding	cs-codex-dist-tests-ci-role-binding
Secret	cs-codex-dist-tests-ci-token

And we may use the following script to generate a kubeconfig file for created Service Account

generate-kubeconfig.sh

# Generate KubeConfig for Service Account

# Variables
cluster=cs-codex-dist-tests
server=$(TERM=dumb kubectl cluster-info | awk '/control plane/ {print $NF}')
namespace=cs-codex-dist-tests
name=cs-codex-dist-tests-ci
sa="${name}-sa"
secret="${name}-token"
config="${cluster}-ci-sa.yaml"

ca=$(kubectl get secret "${secret}" -n "${namespace}" -o jsonpath='{.data.ca\.crt}')
token=$(kubectl get secret "${secret}" -n "${namespace}" -o jsonpath='{.data.token}' | base64 -d)

# Output
echo "apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${ca}
    server: ${server}
  name: ${cluster}
contexts:
- context:
    cluster: ${cluster}
    user: ${sa}
  name: ${cluster}
current-context: ${cluster}
kind: Config
preferences: {}
users:
- name: ${sa}
  user:
    token: ${token}" | tee "${config}"

And a permissions test for created kubeconfig

# Create namespace - Ok
kubectl --kubeconfig cs-codex-dist-tests-ci-sa.yaml create namespace dist-tests-run-0001
namespace/dist-tests-run-0001 created

# Create job - Ok
kubectl --kubeconfig cs-codex-dist-tests-ci-sa.yaml create job dist-tests-run-0001 --image nginx -n dist-tests-run-0001
job.batch/dist-tests-run-0001 created

# List namespaces - Failed
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:cs-codex-dist-tests:cs-codex-dist-tests-ci-sa" cannot list resource "namespaces" in API group "" at the cluster scope

# List jobs - Failed
kubectl --kubeconfig cs-codex-dist-tests-ci-sa.yaml get job -A
Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cs-codex-dist-tests:cs-codex-dist-tests-ci-sa" cannot list resource "jobs" in API group "batch" at the cluster scope

# List pods - Failed
kubectl --kubeconfig cs-codex-dist-tests-ci-sa.yaml get pod -A
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:cs-codex-dist-tests:cs-codex-dist-tests-ci-sa" cannot list resource "pods" in API group "" at the cluster scope

Now, we can use that kubeconfig with GitHub Actions, which will run the dist-tests by creating Kubernetes Namespaces and Jobs, no more permissions are required.

[veaceslavdoina]

kubeconfig added to the repository secrets

cat cs-codex-dist-tests-ci-sa.yaml | base64

[veaceslavdoina]

Job creation via manifest failed until we added get verb for jobs resources in the ClusterRole

rules:
  - verbs:
      - create
      - get
    apiGroups:
      - batch
    resources:
      - jobs

[veaceslavdoina]

We are considering to show Pods logs in GitHub Actions and for that we need some additional permissions. Now it looks the following

rules:
  - verbs:
      - create
    apiGroups:
      - '*'
    resources:
      - namespaces
  - verbs:
      - get
      - create
    apiGroups:
      - batch
    resources:
      - jobs
  - verbs:
      - list
      - get
    apiGroups:
      - '*'
    resources:
      - pods
  - verbs:
      - get
    apiGroups:
      - '*'
    resources:
      - pods/log