Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to retrieve validation definition name='XXX' id='XXX' from store (type=value_error) #10936

Open
sarvesh1911 opened this issue Feb 13, 2025 · 0 comments

Comments

@sarvesh1911
Copy link

sarvesh1911 commented Feb 13, 2025

Describe the bug
Cannot convert an ephemeral data context to a file context
(This is the duplicate to bug - #10271 which was closed)

To Reproduce

great_expectations.yml

# Welcome to Great Expectations! Always know what to expect from your data.
#
# Here you can define datasources, batch kwargs generators, integrations and
# more. This file is intended to be committed to your repo. For help with
# configuration please:
#   - Read our docs: https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/connect_to_data_overview/#2-configure-your-datasource
#   - Join our slack channel: http://greatexpectations.io/slack

# config_version refers to the syntactic version of this config file, and is used in maintaining backwards compatibility
# It is auto-generated and usually does not need to be changed.
config_version: 4

# This config file supports variable substitution which enables: 1) keeping
# secrets out of source control & 2) environment-based configuration changes
# such as staging vs prod.
#
# When GX encounters substitution syntax (like `my_key: ${my_value}` or
# `my_key: $my_value`) in the great_expectations.yml file, it will attempt
# to replace the value of `my_key` with the value from an environment
# variable `my_value` or a corresponding key read from this config file,
# which is defined through the `config_variables_file_path`.
# Environment variables take precedence over variables defined here.
#
# Substitution values defined here can be a simple (non-nested) value,
# nested value such as a dictionary, or an environment variable (i.e. ${ENV_VAR})
#
#
# https://docs.greatexpectations.io/docs/guides/setup/configuring_data_contexts/how_to_configure_credentials


config_variables_file_path: uncommitted/config_variables.yml

# The plugins_directory will be added to your python path for custom modules
# used to override and extend Great Expectations.
plugins_directory: plugins/

stores:
# Stores are configurable places to store things like Expectations, Validations
# Data Docs, and more. These are for advanced users only - most users can simply
# leave this section alone.
  expectations_store:
    class_name: ExpectationsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: expectations/

  validation_results_store:
    class_name: ValidationResultsStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/validations/

  checkpoint_store:
    class_name: CheckpointStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      suppress_store_backend_id: true
      base_directory: checkpoints/

  validation_definition_store:
    class_name: ValidationDefinitionStore
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: validation_definitions/

expectations_store_name: expectations_store
validation_results_store_name: validation_results_store
checkpoint_store_name: checkpoint_store

data_docs_sites:
  # Data Docs make it simple to visualize data quality in your project. These
  # include Expectations, Validations & Profiles. The are built for all
  # Datasources from JSON artifacts in the local repo including validations &
  # profiles from the uncommitted directory. Read more at https://docs.greatexpectations.io/docs/terms/data_docs
  local_site:
    class_name: SiteBuilder
    # set to false to hide how-to buttons in Data Docs
    show_how_to_buttons: true
    store_backend:
        class_name: TupleFilesystemStoreBackend
        base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
        class_name: DefaultSiteIndexBuilder

analytics_enabled: True 

The code is a copy of the tutorial available in the GX Core docs here, with context.convert_to_file_context() appended to the end.

# Import required modules from GX library.
import great_expectations as gx

# Create Data Context.
context = gx.get_context()

# Connect to data.
# Create Data Source, Data Asset, Batch Definition, and Batch.
connection_string = "postgresql+psycopg2://try_gx:[email protected]/gx_example_db"

data_source = context.data_sources.add_postgres(
    "postgres db", connection_string=connection_string
)
data_asset = data_source.add_table_asset(name="taxi data", table_name="nyc_taxi_data")

batch_definition = data_asset.add_batch_definition_whole_table("batch definition")
batch = batch_definition.get_batch()

# Create Expectation Suite containing two Expectations.
suite = context.suites.add(
    gx.core.expectation_suite.ExpectationSuite(name="expectations")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="passenger_count", min_value=1, max_value=6
    )
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(column="fare_amount", min_value=0)
)

# Create Validation Definition.
validation_definition = context.validation_definitions.add(
    gx.core.validation_definition.ValidationDefinition(
        name="validation definition",
        data=batch_definition,
        suite=suite,
    )
)

# Create Checkpoint, run Checkpoint, and capture result.
checkpoint = context.checkpoints.add(
    gx.checkpoint.checkpoint.Checkpoint(
        name="checkpoint", validation_definitions=[validation_definition]
    )
)

checkpoint_result = checkpoint.run()
print(checkpoint_result.describe())

context.convert_to_file_context()

Traceback

Traceback (most recent call last):
  File "/Users/[email protected]/Documents/PDF_Statement_Extraction/PDF_Statement_Extraction/great_expectation_test/great_expectations_test.py", line 53, in <module>
    context.convert_to_file_context()
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/data_context/ephemeral_data_context.py", line 95, in convert_to_file_context
    return migrator.migrate()
           ^^^^^^^^^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/migrator/file_migrator.py", line 63, in migrate
    self._migrate_primary_stores(
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/migrator/file_migrator.py", line 89, in _migrate_primary_stores
    self._migrate_store(
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/migrator/file_migrator.py", line 110, in _migrate_store
    source_obj = source_store.get(key)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/store/store.py", line 246, in get
    return self.deserialize(value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/great_expectations/data_context/store/checkpoint_store.py", line 96, in deserialize
    return Checkpoint.parse_raw(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/pydantic/v1/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/pydantic/v1/main.py", line 526, in parse_obj
    return cls(**obj)
           ^^^^^^^^^^
  File "/Applications/anaconda3/envs/great_expectation_test/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Checkpoint
validation_definitions
  Unable to retrieve validation definition name='validation definition' id='7d48ca95-0f7a-4d91-8a64-c0b6e740bdbf' from store (type=value_error)

Expected behavior
Successfully export ephemeral context to file

Environment (please complete the following information):

Operating System: MacOS
Great Expectations Version: 1.3.5
Data Source: PostgreSQL
Cloud environment: NA

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant