[SPARK-51152][Python][SQL] Add usage examples for the get_json_object function #49875

fusheng9399 · 2025-02-11T03:51:21Z

What changes were proposed in this pull request?

The pr aims to add some usage examples for function get_json_object, including: get_json_object('[{"a":"b"},{"a":"c"}]', '$[0].a')，get_json_object('[{"a":"b"},{"a":"c"}]', '$[*].a').

Why are the changes needed?

Most users are unaware of how to retrieve the corresponding JSON object using get_json_object when the JSON object is of a JSON array type.
Before this PR, the example:

SELECT get_json_object('{"a":"b"}', '$.a');
+-------------------------------+
|get_json_object({"a":"b"}, $.a)|
+-------------------------------+
|                              b|
+-------------------------------+

After this PR, the example of get_json_object has been changed to look like this:

SELECT get_json_object('{"a":"b"}', '$.a');
+-------------------------------+
|get_json_object({"a":"b"}, $.a)|
+-------------------------------+
|                              b|
+-------------------------------+

SELECT get_json_object('[{"a":"b"},{"a":"c"}]', '$[0].a');
+----------------------------------------------+
|get_json_object([{"a":"b"},{"a":"c"}], $[0].a)|
+----------------------------------------------+
|                                             b|
+----------------------------------------------+

SELECT get_json_object('[{"a":"b"},{"a":"c"}]', '$[*].a');
+----------------------------------------------+
|get_json_object([{"a":"b"},{"a":"c"}], $[*].a)|
+----------------------------------------------+
|                                     ["b","c"]|
+----------------------------------------------+

Does this PR introduce any user-facing change?

Yes, added a more intuitive example to the get_json_object function when users visit the https://spark.apache.org/docs/latest/api/sql/#get_json_object webpage and https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.get_json_object.html webpage.

How was this patch tested?

Pass GA & Manually Test

Was this patch authored or co-authored using generative AI tooling?

No

…ray type

fusheng9399 · 2025-02-11T05:55:38Z

Please help review it when you have free time, thanks! @panbingkun

panbingkun · 2025-02-11T06:35:44Z

Are there any other examples of wide characters that need to be shown?
Additionally, the PySpark example also needs to be updated.

fusheng9399 · 2025-02-11T13:24:53Z

No, I have added a more comprehensive example. The PySpark example has been updated.

HyukjinKwon · 2025-02-12T00:51:19Z

python/pyspark/sql/functions/builtin.py

+
+    Example2: Get json object from json array object
+
+    data = [("1", '''[{"f1": "value1", "f2": "value2"},{"f1": "value3", "f2": "value4"}]'''), ("2", '''[{"f1": "value12"},{"f1": "value13"}]''')]


can we also fix the indentation

panbingkun · 2025-02-12T01:44:43Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala

@@ -42,6 +42,10 @@ import org.apache.spark.unsafe.types.UTF8String
    Examples:
      > SELECT _FUNC_('{"a":"b"}', '$.a');
       b
+      > SELECT _FUNC_('[{"a":"b"},{"a":"c"}]', '$[0].a');


Please format the JSON of the example to make it look more pretty and formal, eg:

SELECT FUNC('[{"a": "b"}, {"a": "c"}]', '$[0].a');

I referred to the native syntax of get_json_object.

panbingkun · 2025-02-12T01:49:00Z

python/pyspark/sql/functions/builtin.py

@@ -20115,11 +20115,24 @@ def get_json_object(col: "ColumnOrName", path: str) -> Column:

    Examples
    --------
+    Example1: Get json object from json object


Please format it, eg:
Example 1: ...

Also, the title Get json object from json object looks weird.
Can we give it a more suitable name?

panbingkun · 2025-02-12T01:49:15Z

python/pyspark/sql/functions/builtin.py

    >>> data = [("1", '''{"f1": "value1", "f2": "value2"}'''), ("2", '''{"f1": "value12"}''')]
    >>> df = spark.createDataFrame(data, ("key", "jstring"))
    >>> df.select(df.key, get_json_object(df.jstring, '$.f1').alias("c0"), \\
    ...                   get_json_object(df.jstring, '$.f2').alias("c1") ).collect()
    [Row(key='1', c0='value1', c1='value2'), Row(key='2', c0='value12', c1=None)]
+
+    Example2: Get json object from json array object


Add an example for get_json_object when the JSON object is of JSON ar…

5fb862f

…ray type

github-actions bot added the SQL label Feb 11, 2025

fusheng9399 changed the title ~~Add an example for get_json_object when the JSON object is of JSON array type~~ [SPARK-51152][SQL]Add an example for get_json_object when the JSON object is of JSON array type Feb 11, 2025

add more comprehensive examples

76045ed

github-actions bot added the PYTHON label Feb 11, 2025

Update python/pyspark/sql/functions/builtin.py

25fb4f8

HyukjinKwon reviewed Feb 12, 2025

View reviewed changes

panbingkun reviewed Feb 12, 2025

View reviewed changes

fusheng9399 changed the title ~~[SPARK-51152][SQL]Add an example for get_json_object when the JSON object is of JSON array type~~ [SPARK-51152][SQL]Add a more intuitive example for get_json_object Feb 12, 2025

fusheng9399 changed the title ~~[SPARK-51152][SQL]Add a more intuitive example for get_json_object~~ [SPARK-51152][SQL] Add richer examples for the get_json_object function Feb 12, 2025

fusheng added 5 commits February 12, 2025 14:40

fix the indentation

6b9d029

fix the indentation

0b28e7b

fix the indentation

922104b

fix the indentation

3b663dd

fix the indentation

287d917

fusheng9399 changed the title ~~[SPARK-51152][SQL] Add richer examples for the get_json_object function~~ [SPARK-51152][Python][SQL] Add richer examples for the get_json_object function Feb 14, 2025

fusheng9399 changed the title ~~[SPARK-51152][Python][SQL] Add richer examples for the get_json_object function~~ [SPARK-51152][Python][SQL] Add usage examples for the get_json_object function Feb 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-51152][Python][SQL] Add usage examples for the get_json_object function #49875

[SPARK-51152][Python][SQL] Add usage examples for the get_json_object function #49875

fusheng9399 commented Feb 11, 2025 •

edited

Loading

fusheng9399 commented Feb 11, 2025

panbingkun commented Feb 11, 2025

fusheng9399 commented Feb 11, 2025 •

edited

Loading

HyukjinKwon Feb 12, 2025

panbingkun Feb 12, 2025

fusheng9399 Feb 12, 2025

panbingkun Feb 12, 2025

fusheng9399 Feb 12, 2025 •

edited

Loading

panbingkun Feb 12, 2025

fusheng9399 Feb 12, 2025

panbingkun Feb 12, 2025

fusheng9399 Feb 12, 2025


		Example2: Get json object from json array object

		data = [("1", '''[{"f1": "value1", "f2": "value2"},{"f1": "value3", "f2": "value4"}]'''), ("2", '''[{"f1": "value12"},{"f1": "value13"}]''')]

[SPARK-51152][Python][SQL] Add usage examples for the get_json_object function #49875

Are you sure you want to change the base?

[SPARK-51152][Python][SQL] Add usage examples for the get_json_object function #49875

Conversation

fusheng9399 commented Feb 11, 2025 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

fusheng9399 commented Feb 11, 2025

panbingkun commented Feb 11, 2025

fusheng9399 commented Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fusheng9399 Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fusheng9399 commented Feb 11, 2025 •

edited

Loading

fusheng9399 commented Feb 11, 2025 •

edited

Loading

fusheng9399 Feb 12, 2025 •

edited

Loading