Add Helm hook to upgrade CRDs

[ChenYi015]

Purpose of this PR

Proposed changes:

• Add a Helm pre-install and pre-upgrade hook job to upgrade CRDs. (Will not rollback CRDs when helm rollback.)
• Add a new values hook.upgradeCrd (defaults to true).
• The hook job and RBAC resources will be created only if hook.upgradeCrd is true
• The RBAC resources related to hook will be deleted immediately when hook job completes.

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from chenyi015. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

/hold for review

[ChenYi015]

/assign @yuchaoran2011 @jacobsalway @ImpSy

[jacobsalway]

The fact that Helm doesn't upgrade CRDs during an upgrade has bitten me in the past, so I'm definitely in favour of including some way to do so. It goes without saying that we should never delete CRDs. My only concern would be maintaining Go code to do this versus some other options I've seen in the wild:

• cert-manager adds the CRDs into the chart templates but annotates them with a keep resource policy so they're not uninstalled.
  • https://github.com/cert-manager/cert-manager/blob/master/deploy/crds/crd-certificaterequests.yaml
  • https://cert-manager.io/docs/installation/upgrade/
• gatekeeper uses some Helm templating along with a kubectl image in a Helm hook
  • https://github.com/mesosphere/charts/blob/master/staging/gatekeeper/templates/crds.yaml
• Another suggestion I've seen is to manage the CRDs in a separate Helm chart and add it as a dependency to the main chart, but I don't have any examples for this off the top of my head.

If one of these options can work for our use case, there'd be less code/surface area to maintain. Curious what you think as well?

[ChenYi015]

The fact that Helm doesn't upgrade CRDs during an upgrade has bitten me in the past, so I'm definitely in favour of including some way to do so. It goes without saying that we should never delete CRDs. My only concern would be maintaining Go code to do this versus some other options I've seen in the wild:

• cert-manager adds the CRDs into the chart templates but annotates them with a keep resource policy so they're not uninstalled.

  • https://github.com/cert-manager/cert-manager/blob/master/deploy/crds/crd-certificaterequests.yaml
  • https://cert-manager.io/docs/installation/upgrade/

• gatekeeper uses some Helm templating along with a kubectl image in a Helm hook

  • https://github.com/mesosphere/charts/blob/master/staging/gatekeeper/templates/crds.yaml

• Another suggestion I've seen is to manage the CRDs in a separate Helm chart and add it as a dependency to the main chart, but I don't have any examples for this off the top of my head.

If one of these options can work for our use case, there'd be less code/surface area to maintain. Curious what you think as well?

1. Helm creates a secret for each revision of the chart release, containing the base64 encoding of all rendered templates. By default, the size of a secret is limited to 1MB. Therefore, in our case, placing the CRD files under the chart templates would cause the secret to exceed the size limit.
2. It is just my personal bias that I prefer a self-contained Spark operator docker image rather than using a non-official kubectl image to update the CRDs. BTW, if we choose to use the kubectl way, we need to kubectl replace -f instead of kubectl apply -f due to the size limit of annotations.
3. I believe maintaining the CRDs in another helm chart will definitely increase the complexity of installing/upgrading spark operator.