Skip to content

[Bug] create_test Fails to Materialize Test Data with NULL in TIMESTAMP Column Due to NaT Conversion #4881

Open
@Char-Noir

Description

@Char-Noir

The sqlmesh create_test command fails when an upstream model dependency contains NULL values in a TIMESTAMP column.

During the test generation process, SQLMesh appears to use pandas to fetch data for test fixtures. In this process, a NULL value in a TIMESTAMP column is converted to a pandas.NaT object.

Subsequently, when attempting to load this DataFrame into the in-memory DuckDB instance for the test, the process fails because DuckDB cannot parse the NaT value. This silent failure in data loading prevents the test from being generated correctly.

Probably related to #2436

Steps to Reproduce

  1. Clone the repository:https://github.com/[Char-Noir/sqlmesh_nat_test](https://github.com/Char-Noir/sqlmesh_nat_test)
  2. sqlmesh plan
  3. sqlmesh create_test example.full_model

Expected Behavior

The create_test command should correctly handle NULL values in TIMESTAMP columns, possibly by converting pandas.NaT back to a database-compatible NULL before loading it into the test database. The test should be generated successfully

Actual Behavior

The command fails during the test generation process due to the unhandled NaT value.

Traceback (most recent call last):
  File "/usr/app/sqlmesh_nat_test/.venv/bin/sqlmesh", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 29, in wrapper
    return handler(sqlmesh_context, lambda: func(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 51, in _debug_exception_handler
    return func()
           ^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 29, in <lambda>
    return handler(sqlmesh_context, lambda: func(*args, **kwargs))
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/analytics/__init__.py", line 82, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/main.py", line 710, in create_test
    obj.create_test(
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/analytics/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/context.py", line 2026, in create_test
    generate_test(
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/test/definition.py", line 916, in generate_test
    output = test._execute(model_query)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/test/definition.py", line 610, in _execute
    return self.engine_adapter.fetchdf(query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2055, in fetchdf
    df = self._fetch_native_df(query, quote_identifiers=quote_identifiers)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2031, in _fetch_native_df
    self.execute(query, quote_identifiers=quote_identifiers)
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2183, in execute
    self._execute(sql, **kwargs)
  File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2204, in _execute
    self.cursor.execute(sql, **kwargs)
duckdb.duckdb.ConversionException: Conversion Error: invalid timestamp field format: "NaT", expected format is (YYYY-MM-DD HH:MM:SS[.US][±HH:MM| ZONE])

Environment

SQLMesh version: 0.199.0
Duckdb version: 1.3.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions