[SPARK-53841][PYTHON][CONNECT] Implement transform() in Column API

[Yicong-Huang]

What changes were proposed in this pull request?

To align with scala side, this PR adds transform() API in Columns API, similar to Dataset.transform():

def transform(self, f: Callable[["Column"], "Column"]) -> "Column":

Why are the changes needed?

We want to give users a way to chain their methods, such as

 >>> from pyspark.sql.functions import trim, upper
 >>> df = spark.createDataFrame([("  hello  ",), ("  world  ",)], ["text"])
 >>> df.select(df.text.transform(trim).transform(upper)).show()

This pattern is also easier for AI agents to learn and write.

Does this PR introduce any user-facing change?

Yes. New API is introduced.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Tests generated by Copilot.

[akashadsare]

I would like to work on this issue.Please assign this to me

[Yicong-Huang]

I would like to work on this issue.Please assign this to me

Hi @akashadsare, This is a pull request rather than an issue. Do you want to work on the review instead?
However, I don't have the privilege to assign this to anyone.

[akashadsare]

ok can you tell me how to contribute to this project

[sarutak]

@akashadsare
Please read this guide to contribute to Apache Spark.
Thanks.

[akashadsare]

Thanks

[HyukjinKwon]

would have to be listed in python/docs/source/reference/pyspark.sql

[Yicong-Huang]

done!

[HyukjinKwon]

LGTM otherwise

[zhengruifeng]

LGTM, pending adding it to the column.rst

[Yicong-Huang]

@zhengruifeng @HyukjinKwon I've added it to column.rst.