ExpectColumnValueLengthsToBeBetween is raising exception in DBR 15.4 LTS

[suchintakp5]

Describe the bug
ExpectColumnValueLengthsToBeBetween is failing and raising exception in DBR 15.4 LTS and Unity Catalog enabled cluster.
To Reproduce

import great_expectations as gx
import great_expectations.expectations as gxe

# Retrieve your Data Context
data_context = gx.get_context(mode="ephemeral")
# Define the Data Source name
data_source_name = "source_system_name_spark_dataframe"
# Add the Data Source to the Data Context
data_source = data_context.data_sources.add_spark(name=data_source_name)
# Define the Data Asset name
data_asset_name = "dataset_name"
# Add a Data Asset to the Data Source
data_asset = data_source.add_dataframe_asset(name=data_asset_name)
# Define the Batch Definition name
batch_definition_name = "dataset_batch_definition"

# Add a Batch Definition to the Data Asset
batch_definition = data_asset.add_batch_definition_whole_dataframe(
    batch_definition_name
)

df = <A pyspark dataframe>

batch_parameters = {"dataframe": df}
# Get the dataframe as a Batch
batch = batch_definition.get_batch(batch_parameters=batch_parameters)

test = gxe.ExpectColumnValueLengthsToBeBetween(column=<column_name>, min_value=1, max_value=5, catch_exceptions=True)
# Test the Expectation
validation_results = batch.validate(test, result_format="COMPLETE")

Expected behavior
No exception should be raised

Environment (please complete the following information):

• Operating System: [Azure Databricks cluster with DBR 15.4 LTS and Unity Catalog enabled]
• Great Expectations Version: [1.3.6]
• Data Source: [Pyspark dataframe]
• Cloud environment: [Azure Databricks]

[suchintakp5]

Any update on this? @adeola-ak @tyler-hoffman @joshua-stauffer The same behavior is observed for ExpectColumnValueLengthsToEqual as well

[adeola-ak]

Spark on Azure is not a fully tested/supported environment and we state in our documentation that we can't guarantee ongoing compatibility. was this working before?

[suchintakp5]

@adeola-ak Can you please share the doc link please mentioning Azure not supported?

[adeola-ak]

Sure, Support policy

[adeola-ak]

it would be helpful to know if this worked for you in the past but because your environment is not supported, I will go ahead and close this issue. I do want to share that it has been raised to the team but unfortunately further action is not guaranteed. Sorry for the inconvenience.

[suchintakp5]

@adeola-ak This is working fine in DBR 12.2 version