Change the method for calculating the memory usage for spark apps

[zaultooz]

Currently the pipelines projects provides the memoryOverhead as a pre-calculated fixed value for the spark apps it starts.

I think it would be better if we rely on the spark property "spark.executor.memoryOverheadFactor" to let spark do it itself. So instead of sending multiple values, pipelines should send the total amount of memory needed to process the dataset.

[timrobertson100]

I like the idea - does it get passed up the K8s stack to ensure the machines don't overcommit?

[zaultooz]

Regarding memory we are always setting request and limit the same so a pod shouldn't be allocated to a node if there aren't space for it.

Currently we allow over-commitment for CPU as some application rarely use their full limit, maybe it make sense to set request and limit for spark jobs.