[SPARK-52640][SDP] Propagate Python Source Code Location #51344
Conversation
There was a problem hiding this comment.
i left a few comments, but this looks close to ready to merge to me.
CC @gengliangwang @hvanhovell in case either of you are also interested in taking a look.
python/pyspark/pipelines/spark_connect_graph_element_registry.py
Outdated
Show resolved
Hide resolved
| @@ -49,35 +49,54 @@ class GraphRegistrationContext( | |||
| flows += flowDef.copy(sqlConf = defaultSqlConf ++ flowDef.sqlConf) | |||
| } | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
These changes look independent from what the rest of this PR is doing?
There was a problem hiding this comment.
I just added the scaladoc for the sake of future readers, but the changes to toDataflowGraph are relevant.
toDataflowGraph is where we ensure all identifiers are fully qualified, and qualify if not. It's intentional that once we fully qualify (or verify that the identifier is already fully qualified), we also update the associated query origin with the fully qualified identifier.
| @@ -156,6 +156,10 @@ private[connect] object PipelinesHandler extends Logging { | |||
| .filter(_.nonEmpty), | |||
| properties = dataset.getTablePropertiesMap.asScala.toMap, | |||
| baseOrigin = QueryOrigin( | |||
| filePath = Option.when(dataset.getSourceCodeLocation.hasFileName)( | |||
There was a problem hiding this comment.
nit: we can store filePath and line to variables or a method to avoid duplicated code
|
LGTM too. |
What changes were proposed in this pull request?
Propagate source code location details (line number and file path) E2E for declarative pipelines. That is, collect this information from the python REPL that registers SDP datasets/flows, propagate it through the appropriate spark connect handlers, and associate it to the appropriate datasets/flows in pipeline events/exceptions.
Why are the changes needed?
Better observability and debugging experience for users. Allows users to identify the exact lines that cause a particular exception.
Does this PR introduce any user-facing change?
Yes, we are populating source code information in the origin for pipeline events, which is user-facing. Currently SDP is not released in any spark version however.
How was this patch tested?
Added tests to
org.apache.spark.sql.connect.pipelines.PythonPipelineSuiteWas this patch authored or co-authored using generative AI tooling?
No