Struggling to run spark on flyte

[shubzp]

I am new to flyte. I have been trying to orchestrate my spark application. I have set things up on GKE.

Every time I run a workflow for spark, I get the same error (even for the examples listed in the doc)
[1/1] currentAttempt done. Last Error: USER::Spark Job Submission Failed with Error: failed to run spark-submit: failed to run spark-submit: Exception in thread "main" org.apache.spark.SparkException: --deploy-mode must be either "client" or "cluster"

I see that the sparkapplication that flyte creates does not have mode: cluster in it. When I edit it manually the application gets submitted.

I have tried a lot of settings to default deployMode to cluster in flyte-propeller-config but none works. Moreover, I am also trying to edit the spark_conf with
"spark.submit.deployMode": "cluster"

but this doesn't work either. Any help will be appreciated!

[davidmirror-ops]

We identified the above to be an issue anytime you're using spark-operator>=2.0.0 where a new validator was introduced that would require the deploy mode to be specified.
We'll be testing to determine the course of action to fix this from Flyte