Debugging Spark

It is possible to connect a remote debugger to a spark process.

To connect a debugger to the driver append the following to your spark submit (or gatk-launch) options:

replace 5005 with a different available port if necessary

--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005```

This will suspend the driver until it gets a remote connection from intellij.

Configure a new intellij remote debugging configuration as follows:

• Run -> Edit Configurations will pop up a dialog
• Hit the + to add a new configuration.
• Choose Remote
  • Set Mode to Attach
  • Set Host to your driver node name i.e. dataflow01.broadinstitute.org

--sparkRunner SPARK --sparkMaster yarn-client --num-executors 1 --executor-cores 1 --conf "spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=n,address=wm1b0-8ab.broadinstitute.org:5455,suspend=n"