Refactor events and introduce v1beta3 API for Alert and Provider

[darkowlzz]

Refer #626 for more details.

Based on a lot of observations from v1beta2 API discussion in #435, it became apparent that Alert and Provider can be static data objects without reconcilers and status subresource.

These changes are based on the experiments from last November, refer https://github.com/fluxcd/notification-controller/commits/refactor-event-handler, updated to include all the latest changes.

• Add Alert and Provider v1beta3 APIs which make them static API objects removing the need for any reconciler.
• Remove the old Alert and Provider reconcilers.
• Refactor event handler, splitting it into smaller functions with unit test and integration test of the event server. Also, the event handlers emit k8s events on the respective Alert object for the events.
• Add new Alert and Provider reconcilers to help in migration to static API objects. This is needed to remove the finalizer set by the previous reconcilers. Stale finalizers would interfere with object deletion. Since the new API don't have status, the status from the old objects are automatically dropped when the new API is used. These are temporary migration reconcilers and will be removed in a subsequent release. (NOTE: These new reconcilers are added separately, not on top of the previous reconcilers. Reading commit-by-commit would make it easier to understand the changes.)
• Add new v1beta3 spec docs.

[matheuscscp]

first pass LGTM

[darkowlzz]

These changes depended on the custom metrics that was release in Flux v2.1.0 for the static Alert and Provider metrics. It's no longer blocked now.

I've rebased and tried my best to go through all the recent changes to make sure I haven't missed anything that was added as these changes completely change the alert and provider related code. But still, it'd be great if others look out for things I may have missed. I'll also continue looking for anything missing. As it has been a few months since these changes were written, I'll be testing more and reevaluating if everything is working as they are supposed to be. Marking the PR ready for review.

❔ ❓ I need some help in deciding what to do with the deprecated caFile field support for TLS Secret data in Provider. It was deprecated in v1.1.0 (Flux v2.1.0) in v1beta2 API. This PR introduces v1beta3 API and we can drop the support for it. In the current code, I've kept the support with all the tests. If we decide to drop the support with v1beta3 API, I can remove all the related code.

[stefanprodan]

@darkowlzz I'm for dropping caFile support in v1beta3, please go ahead and delete the code and related docs if any.

[darkowlzz]

I need some help in deciding what to do with the deprecated caFile field support for TLS Secret data in Provider. It was #597 in v1.1.0 (Flux v2.1.0) in v1beta2 API. This PR introduces v1beta3 API and we can drop the support for it. In the current code, I've kept the support with all the tests. If we decide to drop the support with v1beta3 API, I can remove all the related code.

We had a discussion about this with all the maintainers and it was decided to keep this support for now and drop it in v1 API of the Provider. I'll add a TODO in the code to remove it in v1.

[stefanprodan]

Please mark the v1beta2 APIs as deprecated.

From docs:

A beta version API becomes deprecated once a subsequent beta or stable API version is released. A deprecated beta version is subject to removal after a six-months period.

[darkowlzz]

Please mark the v1beta2 APIs as deprecated.

Marked both v1beta1 and v1beta2 Alert and Provider APIs as deprecated.

[stefanprodan]

LGTM

Thanks @darkowlzz 🥇

PS. Please create a PR in website to move the API docs to v1beta3 and update all docs to the new version.

[stefanprodan]

@darkowlzz please integrate #639 and make the correction in docs to

[Bitbucket Server/Data Center](#bitbucket-serverdata-center)

[darkowlzz]

Pushed a fix for log in the new event handler. A post request failure was logged without any context before:

{
  "level": "error",
  "ts": "2023-11-21T16:50:24.582Z",
  "msg": "failed to send notification",
  "error": "postMessage failed: request failed with status code 403, invalid_token"
}

This was because of using the wrong context in the logger. Using the proper context fixes it with rich contextual log:

{
  "level": "error",
  "ts": "2023-11-21T16:58:07.483Z",
  "logger": "event-server",
  "msg": "failed to send notification",
  "eventInvolvedObject": {
    "kind": "HelmRepository",
    "namespace": "default",
    "name": "podinfo",
    "uid": "4892228c-eed5-4dc7-8969-94e51d0df5e7",
    "apiVersion": "source.toolkit.fluxcd.io/v1beta2",
    "resourceVersion": "991119"
  },
  "Alert": {
    "name": "default-alerts",
    "namespace": "default"
  },
  "error": "postMessage failed: request failed with status code 403, invalid_token"
}

[darkowlzz]

please integrate #639 and make the correction in docs to

Rebased and updated the v1beta3 docs with Bitbucket server changes and correction.