Expectation on Snowflake starts with schema scan

[razido1]

Discussed in #5343

^{Originally posted by razido1 June 19, 2022}
When running great expectation on Snowflake with SqlAlchemyExecutionEngine, before the validations starts, there is a scan of all the schemas in the given DB:
It starts with
SHOW /* sqlalchemy:get_schema_names */ SCHEMAS
and then for each schema:

SHOW /* sqlalchemy:get_table_names */ TABLES IN <schema>
SHOW /* sqlalchemy:get_view_names */ VIEWS IN <schema>

How come it scans all schemas if my datasource is to a specific schema?
Is there a way to avoid this? our snowflake schemas are dynamic and can be deleted between the time it reads them and the time it checks for their tables/views. In addition, it is quite costly and not needed since our expectations are on a specific schema.

We are working with version 0.15.10
This is the datasource configuration:
snowflake_db4:
execution_engine:
class_name: SqlAlchemyExecutionEngine
credentials:
host: [host]
username: [username]
database: [db]
query:
schema: [schema]
warehouse: WH_SMALL_POOL
role: [role]
password: [pass]
drivername: snowflake
data_connectors:
default_runtime_data_connector_name:
class_name: RuntimeDataConnector
batch_identifiers:
- default_identifier_name
default_configured_data_connector_name:
class_name: ConfiguredAssetSqlDataConnector
include_schema_name: True

[AFineDayFor]

Howdy @razido1 👋

I'm moving this to discussions so the community can weigh in, and have discourse about this; but most of all so that can be a resource going forward. 🙇

[nikhiltjoy]

I have a PR for this. #5403

The way you can bypass this is by initializing InferredAssetSqlDataConnector appropriately.

datasource_config = {
        "name": table_name,
        "class_name": "Datasource",
        "execution_engine": {
            "class_name": "SqlAlchemyExecutionEngine",
            "connection_string": f"snowflake://{user}:{password}@{account_name}/{database}/{schema}?warehouse={warehouse}&role={role}"
        },
        "data_connectors": {
            "default_runtime_data_connector_name": {
                "class_name": "RuntimeDataConnector",
                "batch_identifiers": batch_identifiers,
            },
            "default_inferred_data_connector_name": {
                "class_name": "InferredAssetSqlDataConnector",
                "include_schema_name": True,
                "included_tables": [f"{schema}.{table_name}"],
                "introspection_directives": {
                    "schema_name": schema,
                    "include_views": False
                }
            },
        },
    }

[razido1]

This is great. waiting for it to be merged