Bump Delta 4.0.0 and enable Delta tests for Spark 4.0

[pan3793]

Why are the changes needed?

https://github.com/delta-io/delta/releases/tag/v4.0.0

How was this patch tested?

GHA.

Was this patch authored or co-authored using generative AI tooling?

No

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (7fbeea6) to head (febaa11).
Report is 1 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7103   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         697     697           
  Lines       43214   43214           
  Branches     5855    5855           
======================================
  Misses      43214   43214           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pan3793]

CI passed, merged to master