Expose delta version in python as delta.__version__

[khakhlyuk]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

This PR exposes delta-spark version in python as delta.__version__. The user will be able to get it using from delta import __version__. The variable will contain the package version in the x.y.z format. This is a common (although optional) pattern in python packages and is provided in numpy, pandas, pyspark, and many other packages.

Delta version is stored in version.sbt. This is a single source of truth for getting the package version and is used by the build.sbt file to add the version to the Scala package and by setup.py when building a delta-spark whl.

This PR adds a new generatePythonVersion sbt task. This task will read the version from version.sbt file and will autogenerate the python/delta/version.py file. The autogeneration code can be run manually build/sbt sparkV1/generatePythonVersion and is run automatically on each build/sbt sparkV1/compile or build/sbt compile run.

PR also contains premerge tests that compare if the version in version.sbt and version.py match. This means that if the version.sbt version is changed and version.py isn't (although the latter should be updated automatically during sbt compile), CI will block the PR and tell the developer to run build/sbt sparkV1/generatePythonVersion.

Additionally, this PR fixes some failing linter checks:

# false positive when checking f-strings, fixed by ignoring E231
/home/alex.khakhlyuk/oss/delta/python/delta/tests/test_pip_utils.py:64:66: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:32: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/pip_utils.py:97:48: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:45:41: E231 missing whitespace after ','
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:22: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/run-tests.py:118:63: E231 missing whitespace after ':'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E231 missing whitespace after ';'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E231 missing whitespace after ';'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E231 missing whitespace after ';'

# false positive detecting backticks ` in sql queries, fixed by ignoring W604
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:360:49: W604 backticks are deprecated, use 'repr()'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:371:49: W604 backticks are deprecated, use 'repr()'
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:382:49: W604 backticks are deprecated, use 'repr()'

# removed ; at the end of the SQL queries
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:101:57: E702 multiple statements on one line (semicolon)
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:194:73: E702 multiple statements on one line (semicolon)
/home/alex.khakhlyuk/oss/delta/python/delta/integration_tests/unity-catalog-commit-coordinator-integration-tests.py:204:77: E702 multiple statements on one line (semicolon)

How was this patch tested?

test_version.py verifies if delta.__version__ is valid and if it's up to date with version.sbt.

Does this PR introduce any user-facing changes?

Python users will be able to read delta version via from delta import __version__.