Implement Policy Lifecycle RFC

[fabriziosestito]

This epic tracks the implementation of the Policy Lifecycle RFC.

Summary

This change addresses a critical issue with the Policy Server.
Currently, a policy can fail to load for a variety of reasons. For example, an inability to pull from the registry or incorrect settings. This causes the Policy Server to exit with an error, causing the pod to enter a CrashLoopBackOff state.
Resolving this requires inspecting the error messages from the Policy Server pod and fixing the underlying issue.
However, a user might have the permissions to create a policy but lack the necessary permissions to check the logs of the Policy Server. This presents challenges to diagnosis and resolution of problems.
Additionally, when updating a policy, there is no status reported to indicate the failure. The policy remains in the Active state even while the Policy Server is stuck in a crash loop.
This creates a significant risk, as new Policy Server pods might fail to start, and the old ones running the previous functional configuration could be lost if their node becomes unavailable.
This situation can disrupt the cluster, as all incoming admission requests will be denied in the absence of operational Policy Server instances.

The proposed changes aim to address these issues by introducing a new policy lifecycle that includes the following features:

• Hot reload of policies: The Policy Server will be able to load a new policy without requiring a restart.
• Policy validation: The Policy Server will validate the policy before loading it, ensuring correct pre-compilation and loading.
• Policy status monitoring: The Policy Server will report the status of the policy, indicating whether it was loaded successfully or if an error occurred.
• Policy versioning: The Policy Server will keep running with a previous version of the policy in case of issues with the latest policy.
• Policy rollback: The user will be able to rollback to a previous policy version in case of issues with the latest policy.

[jvanz]

I just reread the RFC and this epic. It looks good overall. I just have a question about the PolicyRevision CRDs. In the RFC, we state that we will use a single CRD for both cluster and namespaced policies. However, in the epic, we have a sub-issue to create two PolicyRevision CRDs: one for cluster-wide policies and another for namespaced ones. This approach is listed as an "alternative" in the RFC document. I cannot remember if we decided to adopt the alternative solution. If so, the epic looks fine to me. Otherwise, we need to update the sub-issue.

[fabriziosestito]

I just reread the RFC and this epic. It looks good overall. I just have a question about the PolicyRevision CRDs. In the RFC, we state that we will use a single CRD for both cluster and namespaced policies. However, in the epic, we have a sub-issue to create two PolicyRevision CRDs: one for cluster-wide policies and another for namespaced ones. This approach is listed as an "alternative" in the RFC document. I cannot remember if we decided to adopt the alternative solution. If so, the epic looks fine to me. Otherwise, we need to update the sub-issue.

We decided to go with the Namespaced and cluster-scoped alternative, but the RFC was not updated. @flavio do you remember the decision?

[fabriziosestito]

We are thinking about using a feature branch as all the changes will be needed for the architectural change to work, so it will be hard to use feature flags, as this will require feature flags in all components. Also, we are sure that we will introduce breaking changes in the controller's reconcilers, which are not possible to hide under a feature flag.