[SPARK-51149][CORE] Log classpath in SparkSubmit on ClassNotFoundException

[vrozov]

What changes were proposed in this pull request?

The PR adds logging of the classpath in SparkSubmit when SparkApp throws ClassNotFoundException.

Why are the changes needed?

When class is submitted to Spark for execution using SparkSubmit, it may depend on other classes for which libraries are missing on the classpath (for example due to incorrect --jar option or bad install). Logging classpath helps to troubleshoot a missing dependency.

Does this PR introduce any user-facing change?

Yes, classpath is logged when ClassNotFoundException is raised while running application

How was this patch tested?

The patch only impacts logging. Logging was manually verified.

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

Seems you can use spark-submit --verbose ... to rerun the failed cases to know the classpath.

I think verbose error messages should help administrators diagnose the issue, but they might also scare users. Keeping the default error message clear and short, and providing a --verbose might be a good balance.

[vrozov]

I don't think that --verbose outputs the full classpath and between enhancing --verbose to log classpath and logging classpath on the specific error that relates to the classpath issue, IMO the better option is to log the classpath on the error.

[vrozov]

@dongjoon-hyun Please review

[vrozov]

@dongjoon-hyun Please review