Feat/add cluster stacks remote api plugin

[tonifinger]

This PR implement plugin for cluster-stacks-remote-api.

It provides a PoC for the basic functionality afor the usage of the moin cluster.
Currently supports only one cluster.yaml.

[tonifinger]

To test it against the scs-test-runner.py use the following command to provision a cluster:

./scs-test-runner.py --config ./config.toml --debug provision --preset="kaas-dev"

And the following command to unprovision a cluster:

./scs-test-runner.py --config ./config.toml --debug unprovision --preset="kaas-dev"

[michal-gubricky]

I've updated the implementation of the cluster stacks remote API plugin. The deletion and creation of workload clusters should work when the management cluster already has the necessary CAPI/CAPO resources, and the given namespace contains the required secret and ClusterResourceSet resource.

[tonifinger]

DRAFT:

I've updated the implementation of the cluster stacks remote API plugin. The deletion and creation of workload clusters should work when the management cluster already has the necessary CAPI/CAPO resources, and the given namespace contains the required secret and ClusterResourceSet resource.

I did not get the provisioning process working it crashes with the following error:

(venv) tf@tf-ThinkPad:~/repos/ci/scs/01_ISSUES/standards/Tests$ ./scs-test-runner.py --config ./config.toml --debug provision --preset="kaas-dev"
DEBUG: running provision for subject(s) moin-cluster-current-2, num_workers: 1
/bin/sh: 1: /tmp/envsubst: not found
error: unknown command "oidc-login" for "kubectl"
Unable to connect to the server: getting credentials: exec: executable kubectl failed with exit code 1
ERROR: Error applying clusterstack.yaml: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 203, in _apply_yaml
    self._run_subprocess(command, error_msg, shell=True, kubeconfig=kubeconfig)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 312, in _run_subprocess
    result = subprocess.run(command, shell=shell, capture_output=capture_output, text=text, check=True, env=env)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/run_plugin.py", line 65, in <module>
    cli()
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/run_plugin.py", line 48, in create
    plugin.create_cluster(cluster_id, clusterinfo['branch'], os.path.abspath(clusterinfo['kubeconfig']))
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 140, in create_cluster
    self._apply_yaml(self.clusterstack, "Error applying clusterstack.yaml", kubeconfig=self.kubeconfig_mgmnt)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 206, in _apply_yaml
    raise RuntimeError(f"{error_msg}: {error}")
RuntimeError: Error applying clusterstack.yaml: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.

In the config.yaml I enabled only one moin cluster tests subject:

[subjects.moin-cluster-current-2.kubernetes_setup]
kube_plugin         = "cluster-stacks-api"
kube_plugin_config  = "../playbooks/k8s_configs/moin_cluster_config.yaml"
clusterspec_cluster = "current-k8s-release-2"

[michal-gubricky]

DRAFT:

I've updated the implementation of the cluster stacks remote API plugin. The deletion and creation of workload clusters should work when the management cluster already has the necessary CAPI/CAPO resources, and the given namespace contains the required secret and ClusterResourceSet resource.

I did not get the provisioning process working it crashes with the following error:

(venv) tf@tf-ThinkPad:~/repos/ci/scs/01_ISSUES/standards/Tests$ ./scs-test-runner.py --config ./config.toml --debug provision --preset="kaas-dev"
DEBUG: running provision for subject(s) moin-cluster-current-2, num_workers: 1
/bin/sh: 1: /tmp/envsubst: not found
error: unknown command "oidc-login" for "kubectl"
Unable to connect to the server: getting credentials: exec: executable kubectl failed with exit code 1
ERROR: Error applying clusterstack.yaml: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 203, in _apply_yaml
    self._run_subprocess(command, error_msg, shell=True, kubeconfig=kubeconfig)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 312, in _run_subprocess
    result = subprocess.run(command, shell=shell, capture_output=capture_output, text=text, check=True, env=env)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/run_plugin.py", line 65, in <module>
    cli()
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tf/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/run_plugin.py", line 48, in create
    plugin.create_cluster(cluster_id, clusterinfo['branch'], os.path.abspath(clusterinfo['kubeconfig']))
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 140, in create_cluster
    self._apply_yaml(self.clusterstack, "Error applying clusterstack.yaml", kubeconfig=self.kubeconfig_mgmnt)
  File "/home/tf/repos/ci/scs/01_ISSUES/standards/Tests/kaas/plugin/plugin_cluster_stacks_remote_api.py", line 206, in _apply_yaml
    raise RuntimeError(f"{error_msg}: {error}")
RuntimeError: Error applying clusterstack.yaml: Command '/tmp/envsubst < /home/tf/repos/ci/scs/01_ISSUES/standards/Tests/../playbooks/k8s_configs/moin_cluster_clusterstack.yaml | kubectl apply -f -' returned non-zero exit status 1.

In the config.yaml I enabled only one moin cluster tests subject:

[subjects.moin-cluster-current-2.kubernetes_setup]
kube_plugin         = "cluster-stacks-api"
kube_plugin_config  = "../playbooks/k8s_configs/moin_cluster_config.yaml"
clusterspec_cluster = "current-k8s-release-2"

You must install envsubst. This step is described in README under the core requirements for now.

[garloff]

If envsubst becomes to much of a hurdle, we could replace it with a few lines of python ...
How can we get this finalized?

[tonifinger]

If envsubst becomes to much of a hurdle, we could replace it with a few lines of python ... How can we get this finalized?

The envsubst part Is working I added the package to the python requirements.in.
So the generation of the clusterstack and cluster resource itself is working.

What needs to be changed, however, is the process of waiting for the k8s cluster to be ready. This is currently handled by waiting for resources that are not accessible to user accounts with minimal RBAC setup.

To resolve this problem, I suggest that we only use information provided by the “Cluster” resource.

[mbuechse]

I'm seeing three non-negligible problems here:

• it uses sudo, which requires privileges and user interaction
• it uses clusterctl, which is documented nowhere
• it uses oidc login, which requires user interaction

Fortunately, I think we will have none of these problems with Syself clusterstacks, because they provide a service account with headless login, and the kubeconfig can be retrieved using kubectl get secrets.

[mbuechse]

superseded by #916