Microsoft.Spark.JvmException has no JVM details

[dbeavon]

I've been encountering what I believe to be an out-of-memory error in the JVM driver on databricks. (Not yet proven.)

The class Microsoft.Spark.JvmException lacks a lot of information, and makes it hard to troubleshoot.

Would it be possible to expose the underlying JVM exception details? Eg. Here is a typical example of a JvmException (output from ToString method).'

[At 01:18:43.133 on 0426-005735-hires536-10-129-253-6 for file datarail/bb53d9cf-fb49-4af9-be73-3e92ec1fbd62.xml] Error is : System.Exception: JVM method execution failed: Static method 'builder' failed for class 'org.apache.spark.sql.SparkSession' when called with no arguments
 ---> Microsoft.Spark.JvmException: Exception of type 'Microsoft.Spark.JvmException' was thrown.
   --- End of inner exception stack trace ---
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String methodName, Object[] args)
   at Microsoft.Spark.Sql.Builder..ctor()
   at UFP.DataRail.Spark.Driver.Program.Main(String[] args) in D:\a\1\s\DotNetWorkspaces\DataRailWorkspace\DataRail\Spark\Driver \Program.cs:line 170.
Unhandled exception. System.Exception: JVM method execution failed: Static method 'builder' failed for class 'org.apache.spark.sql.SparkSession' when called with no arguments
 ---> Microsoft.Spark.JvmException: Exception of type 'Microsoft.Spark.JvmException' was thrown.
   --- End of inner exception stack trace ---
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args)
   at Microsoft.Spark.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String methodName, Object[] args)
   at Microsoft.Spark.Sql.Builder..ctor()
   at UFP.DataRail.Spark.Driver.Program.Main(String[] args) in D:\a\1\s\DotNetWorkspaces\DataRailWorkspace\DataRail\Spark\Driver \Program.cs:line 170

Notice that there are no details exposed from the JVM side of things. It says that the "builder" method failed but doesn't expose the underlying reason.

Any tips would be appreciated. Let me know if I should be using another approach to get the JVM details. Ideally the ToString() method should be overridden in this class, so that it exposes all the relevant information about the error, and the stack frames where it happened.

[dbeavon]

It appears that the underlying details of the JVM exceptions can be found in the log4j output of the cluster.

This is sufficient for now. However it is kind of a pain that we don't get the details in the ToString() of the .Net exception as well.

Currently I have to go to 2 or 3 different places to gather up all the details about an error that just happened in an ad-hoc (run-once) job on azure-databricks. I'd prefer to get everything sent back to the client application. There is a client application that is interfacing with the azure-databricks REST API. But there is almost no information that makes it back across that API to the client application (... except for a URL that I can click, to open the cluster in question and research problems in their portal).

In the case of a databricks applications which fails as a result of routine business validation rules, it seems quite inconvenient to have to spend so much time hunting down the root cause of errors.... Alternatively I may have to move my error-handling logic into the databricks cluster itself, rather than trying to bubble the error details out to the client...