Manual run inside Kubernetes via Job

[veaceslavdoina]

This is a result of the short investigation how we can run dist-tests inside the Kubernetes via Job.

It helps us to understand what we should improve for automatic run inside the Kubernetes via Job and CLI app.

Now we can do it in the following way

1. Use a Docker image which we may run via Kubernetes Job
2. Create a separate namespace
3. Create a secret with kubeconfig
4. Create a config with the custom settings for the kubeConfigFile and Logging
5. Create a Job which will
   • mount secret inside a container
   • mount config inside a container
   • set a variable which will trigger a config owerride from cloned repository with a mounted one
   • mounted config use mounted kubeconfig and logs location

Deploy

1. Create the following files

   Note: Please, make sure to update at least secret.yaml with the required data

   namespace.yaml

   apiVersion: v1
   kind: Namespace
   metadata:
     name: cs-codex-dist-tests

   secret.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: cs-codex-dist-tests
     namespace: cs-codex-dist-tests
     labels:
       name: cs-codex-dist-tests
   stringData:
     kubeconfig.yaml: |
       apiVersion: v1
       clusters:
       - cluster:
           certificate-authority-data: ****
           server: ****
         name: ****
       contexts:
       - context:
           cluster: ****
           user: ****
         name: ****
       current-context: ****
       kind: Config
       preferences: {}
       users:
       - name: ****
         user:
           token: ****

   config.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cs-codex-dist-tests
     namespace: cs-codex-dist-tests
     labels:
       name: cs-codex-dist-tests
   data:
     Configuration.cs: |
       using KubernetesWorkflow;
   
       namespace DistTestCore
       {
           public class Configuration
           {
               public KubernetesWorkflow.Configuration GetK8sConfiguration(ITimeSet timeSet)
               {
                   return new KubernetesWorkflow.Configuration(
                       k8sNamespacePrefix: "ct-",
                       kubeConfigFile: "/opt/kubeconfig.yaml",
                       operationTimeout: timeSet.K8sOperationTimeout(),
                       retryDelay: timeSet.WaitForK8sServiceDelay(),
                       locationMap: new[]
                       {
                           new ConfigurationLocationEntry(Location.BensOldGamingMachine, "worker01"),
                           new ConfigurationLocationEntry(Location.BensLaptop, "worker02"),
                       } 
                   );
               }
   
               public Logging.LogConfig GetLogConfig()
               {
                   return new Logging.LogConfig("/var/log/cs-codex-dist-tests", debugEnabled: false);
               }
   
               public string GetFileManagerFolder()
               {
                   return "TestDataFiles";
               }
           }
       }

   job.yaml

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: cs-codex-dist-tests
     namespace: cs-codex-dist-tests
     labels:
       name: cs-codex-dist-tests
   spec:
     backoffLimit: 0
     template:
       metadata:
         name: cs-codex-dist-tests
       spec:
         containers:
         - name: cs-codex-dist-tests
           image: codexstorage/cs-codex-dist-tests:sha-300b91e
           env:
           - name: CONFIG
             value: "/opt/Configuration.cs"
           - name: CONFIG_SHOW
             value: "true"
           volumeMounts:
           - name: kubeconfig
             mountPath: /opt/kubeconfig.yaml
             subPath: kubeconfig.yaml
           - name: config
             mountPath: /opt/Configuration.cs
             subPath: Configuration.cs
           - name: logs
             mountPath: /var/log/cs-codex-dist-tests
           # command:
           # - "dotnet"
           # - "test"
           # - "Tests"
         restartPolicy: Never
         volumes:
           - name: kubeconfig
             secret:
               secretName: cs-codex-dist-tests
           - name: config
             configMap:
               name: cs-codex-dist-tests
           - name: logs
             hostPath:
               path: /var/log/cs-codex-dist-tests

2. Create resources

   kybectl apply -f namespace.yaml
   kybectl apply -f secret.yaml
   kybectl apply -f config.yaml
   kybectl apply -f job.yaml

Results

As a result, Kubernetes will run the Job, which will a run a Pod with app, which will run the dist-tests

# Job
k get jobs -A
NAMESPACE             NAME                  COMPLETIONS   DURATION   AGE
cs-codex-dist-tests   cs-codex-dist-tests   0/1           11m        11m

# Runner pod
k get pods -n cs-codex-dist-tests
NAME                        READY   STATUS   RESTARTS   AGE
cs-codex-dist-tests-hmxbm   0/1     Error    0          12m

# Dist-Tests pods
k get pods -n ct-00009
NAME                        READY   STATUS    RESTARTS   AGE
deploy-0-bfd474648-blnff    2/2     Running   0          8m44s
deploy-1-54557b8686-m2psh   1/1     Running   0          8m41s

screenshots

Improvements proposal

• Add a way to configure log files location #3 - now we workaround that with a custom config
• Option to set kubeConfigFile #8 - now we workaround that with a custom config
• Option to set an unique k8sNamespacePrefix per tests run #9
• Option to set a custom config #10 - now we workaround that with a custom entrypoint in the Docker image
• Option to set number of parallel tests #13
• Option to authenticate using Service Account #18

[veaceslavdoina]

And to handle dynamically Job name

apiVersion: batch/v1
kind: Job
metadata:
  name: cs-codex-dist-tests-${NOW}
  namespace: cs-codex-dist-tests
  labels:
    name: cs-codex-dist-tests-${NOW}
    run-id: ${NOW}

export NOW=$(date +%Y%m%d-%H%M%S) && envsubst < job.yaml | kubectl apply -f -
job.batch/cs-codex-dist-tests-20230602-155944 created

kubectl get jobs -n cs-codex-dist-tests
NAME                                  COMPLETIONS   DURATION   AGE
cs-codex-dist-tests-20230602-155944   0/1           67s        67s

[benbierens]

Since branch feature/configurability we have environment variable support:

"KUBECONFIG" - Path to Kubeconfig YAML file, or null to use system default.
LOGPATH - Path where logging will be saved.
LOGDEBUG - set to "true" for debug-logging of the test app.
DATAFILEPATH - Path where data files will be generated (and deleted).
LOGLEVEL - Codex log level. Default: "TRACE"
RUNNERLOCATION - Use "ExternalToCluster" (default) for docker-desktop runs. Use "InternalToCluster" when running from inside a pod in the cluster.

I have actually been using this with a little setup script in the ben-pod! :D

apt-get update
apt-get install vim -y
git clone https://github.com/codex-storage/cs-codex-dist-tests.git
mv /opt/codex-tests-ams3-dev-kubeconfig.yaml /opt/kubeconfig.yaml
export KUBECONFIG=/opt/kubeconfig.yaml
export RUNNERLOCATION=InternalToCluster

[veaceslavdoina]

Thank you for the update! Based on your latest changes bc4035e I was able to run test via Job in Kubernetes.

Now, we have a workflow with just manual run. Later we will consider to run it on marge and maybe on PR.

All parameters are optional and will override default values