Enable the override of MemoryLimit through webhook

[danielrsfreitas]

Purpose of this PR

Proposed changes:

Add the ability to setup memoryLimits definition through webhook.

The changes were originally part of the development of #2383 and #2389.

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

This changes addresses the community most reacted issue #1489.

A few users have created their own custom version of either SparkOperator or Kubernetes Admission Controller to enable memoryLimits with a different value than memory requested.

It was first raised here #1990. The pull request 1990 contained two features: memoryLimits and queue boost. Since the second part was covered in another PR, the 1990 was closed and the memoryLimits was not merged in.

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[danielrsfreitas]

Hi @ChenYi015 , I am tagging you here as you were the reviewer of #2383 and I am addressing your comments in here. Hope you like it!

[ChenYi015]

@danielrsfreitas Thanks for your contribution, I will review it later today.
/assign @vara-bonthu @jacobsalway @ImpSy Tagging other maintainers for review too.

[ChenYi015]

/lgtm

[danielrsfreitas]

Hi @ChenYi015 , thanks for approving the code. I see it is not ready for merge yet. Is there anything left on my side? What would be the next steps?

[ChenYi015]

Hi @ChenYi015 , thanks for approving the code. I see it is not ready for merge yet. Is there anything left on my side? What would be the next steps?

Let us wait for approve from another maintainer and we will release this feature in the next minor version (i.e. v2.2.0).

[ChenYi015]

/assign @vara-bonthu @jacobsalway @ImpSy

[danielrsfreitas]

Any updates? Was anybody else able to check this?

[vara-bonthu]

Thank for your contribution @danielrsfreitas !

[bnetzi]

@danielrsfreitas just saw this, thanks for taking the effort and make this ready for merge, much appreciated 🙏

[ChenYi015]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[forma-wolf]

Did anyone test this? These new parameters are not working when I am using a kubectl manifest of such on v2.2.0:

Error from server (BadRequest): error when creating "xxxxx.yaml": SparkApplication in version "v1beta2" cannot be handled as a SparkApplication: strict decoding error: unknown field "spec.executor.memoryLimit"

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: ...
namespace: ...
spec:
type: Python
pythonVersion: "3"
mode: cluster
image: ...
imagePullPolicy: IfNotPresent
mainApplicationFile: ...
sparkVersion: 3.5.5
sparkConf:
...
driver:
cores: 14
coreLimit: "14"
memory: "108g"
executor:
instances: 1
cores: 14
coreLimit: "14"
memory: "108g"
memoryLimit: "108g"