[Feat] Upgrade spark-operator to v2.3.0

[machichima]

Tracking issue

Related to: #6348

Why are the changes needed?

To be able to set PriorityClassName and ephermal-storage, we need to first upgrade our spark plugin

What changes were proposed in this pull request?

Update spark operator to version 2.3.0 and upgrade other packages to support this.

How was this patch tested?

Ensure build and spark test passed

Labels

Please add one or more of the following labels to categorize your PR:

• added: For new features.
• changed: For changes in existing functionality.
• deprecated: For soon-to-be-removed features.
• removed: For features being removed.
• fixed: For any bug fixed.
• security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[flyte-bot]

Bito Automatic Review Skipped - Large PR

Bito didn't auto-review this change because the pull request exceeded the line limit. No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.95%. Comparing base (74d9e86) to head (2b343ee).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6686      +/-   ##
==========================================
+ Coverage   58.57%   62.95%   +4.37%     
==========================================
  Files         929      164     -765     
  Lines       70879     9903   -60976     
==========================================
- Hits        41520     6234   -35286     
+ Misses      26206     3240   -22966     
+ Partials     3153      429    -2724     

Flag	Coverage Δ
unittests-datacatalog	59.03% <ø> (ø)
unittests-flyteadmin	?
unittests-flytecopilot	41.36% <ø> (ø)
unittests-flytectl	?
unittests-flyteidl	76.12% <ø> (ø)
unittests-flyteplugins	?
unittests-flytepropeller	?
unittests-flytestdlib	63.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kevinliao852]

It seems the new spark-operator version requires Go >= 1.24.1.
We might need to wait until PR #6603
(Go 1.24 upgrade) gets merged before moving forward here.