Add resource limits when providing a spark cluster to Openshift

[wolli2710]

When I create a new spark cluster the resource limits for the master and the worker nodes are not set by default.
How is the flow for creating the spark cluster out of oshinko-webui or tutorial-sparkpi-python-flask and where can I set the resource limits for the pod?

[elmiko]

hi @wolli2710 , thanks for opening this issue.

currently, i do not think there is a way to specify the resource limits for the pods deployed by the oshinko-s2i. i do have a couple thoughts though.

1. if we added a variable that you could specify during deployment of the s2i, would that be helpful?
   i'm thinking something like this:

oc new-app --template oshinko-python-spark-build-dc  \
>     -p APPLICATION_NAME=sparkpi \
>     -p GIT_URI=https://github.com/radanalyticsio/tutorial-sparkpi-python-flask.git
>     -p POD_MEMORY=1024

2. you might want to check out the spark-operator project. it is an operator that can perform many similar functions to the oshinko tooling and it does have options for resource limit specification.

[wolli2710]

@elmiko thanks for the reply, I think that the spark-operator project is the way to go. thx!

[elmiko]

np, good luck @wolli2710

i am going to keep this issue open as i feel it's relevant to the project.