Audit policy as managed spec of cluster

[danielskowronski]

Is your feature request related to a problem? Please describe.

Currently, in order to add k8s audit policy configuration, user must utilize raw spec.api.extraArgs and manually define at least audit-policy-file + audit-log-path.

This is problematic because those paths must have appropriate permissions so that kube-apiserver can read config and store logs. However, k0s runs kube-apiserver as a regular process launched by a dedicated user, which normally is created by k0s install, therefore externally adding files with relevant permissions (e.g. using clound-init) or using before hook will fail. This was originally raised in #3738

Describe the solution you would like

I think it could be beneficial to have something similar to the proposed in #1420 (comment) for handling audit log.

At minimum, it could be something like having all --audit-* flags replicated into a separate section of cluster spec with the addition of a process that upon bootstrap would set appropriate permissions to path from --audit-log-path (either via normal chown process, which can be tricky not to break other permissions, or via setfacl). In this option, it'd be up to user to ensure that the policy file and log directory exist.

At best, we could have the audit config stored in cluster spec as YAML. That way, we could have this important safety feature handled entirely from k0s.yaml. This would help users to ensure each machine has the same policy file and allow dynamic re-configuration (right now it requires manual handling of service restart).

Describe alternatives you've considered

• cloud-init is enough to bootstrap audit with users and write_files
• further adjustments can be achieved with Terraform (e.g. null_resource that deploys updated file and restarts k0s controller triggered by template changes)

Additional context

Cluster auditing is important for security and debugging. Given the non-standard nature of kube-apiserver in k0s and k0s ability to use dynamic cluster configuration, it's a good idea to have it handled better than with raw extraArgs.