How to get the plugin class to be visible to executors

[brandonvin]

Hi, I tried using an ExecutorPlugin as you showed in this repo, with Spark 2.4.4. My setup is to define the plugin class in my application code, and build an uberjar of the application.

I am passing the application jar as --jars to spark-submit. However, executors fail to launch with a ClassNotFoundException. I can only workaround this by passing the application JAR explicitly as spark.executor.extraClassPath. Do you have any clue why?

I have more details in this post: https://stackoverflow.com/questions/58586671/executorplugin-plugin-class-bundled-in-app-jar-class-not-found-on-executor

Thanks, I'd appreciate any insights. Knowledge about ExecutorPlugin seems scarce, sorry if this is the wrong place to ask.

[LucaCanali]

Hi,
Good point. I believe that using --jars for executor plugins works only for YARN (tested). It does not work for K8S either (tested). See also the comment on this matter related to recent work to extend plugin functionality for Spark 3.0: apache/spark#26170 (comment)

[brandonvin]

Hi @LucaCanali, thanks! That's great to know. I'm not going crazy.