Description
The sqlmesh create_test command fails when an upstream model dependency contains NULL values in a TIMESTAMP column.
During the test generation process, SQLMesh appears to use pandas to fetch data for test fixtures. In this process, a NULL value in a TIMESTAMP column is converted to a pandas.NaT object.
Subsequently, when attempting to load this DataFrame into the in-memory DuckDB instance for the test, the process fails because DuckDB cannot parse the NaT value. This silent failure in data loading prevents the test from being generated correctly.
Probably related to #2436
Steps to Reproduce
- Clone the repository:https://github.com/[Char-Noir/sqlmesh_nat_test](https://github.com/Char-Noir/sqlmesh_nat_test)
- sqlmesh plan
- sqlmesh create_test example.full_model
Expected Behavior
The create_test command should correctly handle NULL values in TIMESTAMP columns, possibly by converting pandas.NaT back to a database-compatible NULL before loading it into the test database. The test should be generated successfully
Actual Behavior
The command fails during the test generation process due to the unhandled NaT value.
Traceback (most recent call last):
File "/usr/app/sqlmesh_nat_test/.venv/bin/sqlmesh", line 10, in <module>
sys.exit(cli())
^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 29, in wrapper
return handler(sqlmesh_context, lambda: func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 51, in _debug_exception_handler
return func()
^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/__init__.py", line 29, in <lambda>
return handler(sqlmesh_context, lambda: func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/analytics/__init__.py", line 82, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/cli/main.py", line 710, in create_test
obj.create_test(
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/analytics/__init__.py", line 110, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/context.py", line 2026, in create_test
generate_test(
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/test/definition.py", line 916, in generate_test
output = test._execute(model_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/test/definition.py", line 610, in _execute
return self.engine_adapter.fetchdf(query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2055, in fetchdf
df = self._fetch_native_df(query, quote_identifiers=quote_identifiers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2031, in _fetch_native_df
self.execute(query, quote_identifiers=quote_identifiers)
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2183, in execute
self._execute(sql, **kwargs)
File "/usr/app/sqlmesh_nat_test/.venv/lib/python3.12/site-packages/sqlmesh/core/engine_adapter/base.py", line 2204, in _execute
self.cursor.execute(sql, **kwargs)
duckdb.duckdb.ConversionException: Conversion Error: invalid timestamp field format: "NaT", expected format is (YYYY-MM-DD HH:MM:SS[.US][±HH:MM| ZONE])
Environment
SQLMesh version: 0.199.0
Duckdb version: 1.3.1