[Spark] include runtime filters and snapshot version for SparkScan/SparkTable equality check

[yyanyy]

• Also improved thread safety with filter() method for updating state mutations

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

• This is a follow up of [Spark] Implements equals and hashcode for SparkTable and SparkScan #5827 , to add the missing snapshot info to the equality check of SparkTable.
• Separately, it was noticed that the current filter() method in SparkScan mutates states, but such mutation was not thread safe and not reflected in equality checks.
  • for equality check, only the effective filter is tracked. This is aligned with the Iceberg approach in SparkBatchQueryScan.
  • However, a difference is that Iceberg uses a list (i.e. order dependent) for the incoming filters for equality check here, where we are using a set in this change. From the original PR Spark: Support dynamic partition filtering in 3.2 apache/iceberg#3400 it seems that it was not discussed, so I doubt if it is intentional.

How was this patch tested?

Additional unit test

Does this PR introduce any user-facing changes?

No

[gengliangwang]

nit: let's break these test cases into differnt test methods