Custom Spark 3.5.3 Kernel

[fatihmete]

Hello everyone. I am using Jupyter Enterprise Gateway with PySpark sessions on Kubernetes. The elyra/kernel-spark-py:3.2.3 image works as expected.

I modified the image and rebuilt it to upgrade the Spark version to 3.5.3. When I start this kernel through JEG, the Spark driver and executor pods are created and run as expected. However, within the notebook, the spark variable is stuck in the WaitingForSparkSessionToBeInitialized value. If I redefine it with spark=SparkSession.builder.getOrCreate(), it doesn't give an error and works.

[fatihmete]

I did some changes in the launch_ipykernel.py file. I removed all codes related with background thread in initialize_spark_session function. Now it works, but the kernel waits about 2 minutes in unknown status after driver and executor pods started.
Also this issue exist in 3.4.4 version of spark.

[fatihmete]

Latency in starting spark notebook is relating to web socket timeout. I found the error shown below in EnterpriseGateway logs:

[W 2024-12-13 .... EnterpriseGatewayApp] Websocket ping timeout after 90537ms. 
[W 2024-12-13 .... EnterpriseGatewayApp] Websocket closed <kernel_id>
...

The error doesn't affect spark notebook, notebook works as expected. So I tried to find how to decrease timeout seconds. I found a environment variable described as below in documentation:

--EnterpriseGatewayApp.ws_ping_interval=<Int>
    Specifies the ping interval(in seconds) that should be used by zmq port
     associated withspawned kernels.Set this variable to 0 to disable ping mechanism.
    (EG_WS_PING_INTERVAL_SECS env var)
    Default: 30

I set EG_WS_PING_INTERVAL_SECS to 1, then timeout seconds decreased to 30 seconds. When I tried 0 second, It waited 90 seconds again. The variable configures tornado web application parameters and doesn't work like described in documentation. Finally I changed related parameters directly on enterprisegatewayapp.py file. I succeeded to disable ping mechanism. Now starting of spark 3.5.3 notebooks take about 20 seconds.

Parameters related to websocket timeout:

• websocket_ping_interval
• websocket_ping_timeout

enterprisegatewayapp.py related line:

enterprise_gateway/enterprise_gateway/enterprisegatewayapp.py

Line 252 in ffebbaf

ws_ping_interval=self.ws_ping_interval * 1000,

[paf91]

This is related to this discussion
Can you please describe which Java version and libraries you used in the custom image (or post a Dockerfile if it has lot of changes) as I had similar problems using 3.5, I succesfully upgraded to 3.3.2 though.
Also I must say that I use lazy spark initialization as I believe each user want to run it with different options

[patilsanket48]

hi @kevin-bates @lresende @fatihmete @ , i am relative new to the spark in JEG, could you please share the dockerfile that you used for the spark update ? or the steps that worked for you to update spark ?

[fatihmete]

@paf91 I guess the link is broken.

@patilsanket48 you can use this docker file.

[lresende]

@paf91 @patilsanket48 I haven't used 3.5.3 yet, but you should look into the spark driver logs and see what is causing the spark session creation to fail, probably some additional changes required due to new major version of Spark.

[patilsanket48]

@fatihmete thanks...i have upgraded spark to 3.4.1 and i see the exact problem that is raised by you, however i have not deleted anything from launch_ipykernel.py . i see the similar error as shown below.

did you just set ws_ping_interval as 1 second ? what worked with you ? or completely disabled it ? would it affect other kernels ?

it just stucks at WaitingForSparkSessionToBeInitialized. but manual invocation works like spark = SparkSession.builder.getOrCreate() i am trying to figure out why spark variable is not getting set automatically.

@lresende I see no logs that helps from spark driver

[lresende]

@patilsanket48 this is what controls the session initialization:

enterprise_gateway/etc/kernelspecs/spark_python_kubernetes/kernel.json

Line 30 in ffebbaf

"--RemoteProcessProxy.spark-context-initialization-mode",

And this parameter is described in https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/system-architecture.html

Other options supported by launchers include:

--RemoteProcessProxy.port-range {port_range} - passes configured port-range to launcher where launcher applies that range to kernel ports. The port-range may be configured globally or on a per-kernel specification basis, as previously described.

--RemoteProcessProxy.spark-context-initialization-mode [lazy|eager|none] - indicates the timeframe in which the spark context will be created.

lazy (default) attempts to defer initialization as late as possible - although this can vary depending on the underlying kernel and launcher implementation.

eager attempts to create the spark context as soon as possible.

none skips spark context creation altogether.

Note that some launchers may not be able to support all modes. For example, the scala launcher uses the Apache Toree kernel - which currently assumes a spark context will exist. As a result, a mode of none doesn’t apply. Similarly, the lazy and eager modes in the Python launcher are essentially the same, with the spark context creation occurring immediately, but in the background thereby minimizing the kernel’s startup time.