[BUG]: Validating Snowflake DBT to Databricks DBT leads to not transpiling code

[matthewmoorcroft]

Is there an existing issue for this?

• I have searched the existing issues

Category of Bug / Issue

Converter bug

Current Behavior

Root Cause Analysis

The issue appears to be that the transpiler is attempting to execute dbt Jinja templating directly against a data warehouse instead of going through the dbt compilation process. This causes the warehouse to encounter raw Jinja syntax ({{ }}) which is not valid SQL.

Impact

• Misleading success messages that hide actual transpilation failures
• Generated SQL files that cannot be executed due to syntax errors
• Confusion about whether the transpilation actually succeeded

Suggested Solutions

1. Proper Jinja Processing: Implement proper Jinja templating processing during transpilation
2. Accurate Status Reporting: Only report success if the final SQL is syntactically valid
3. Clear Error Handling: Provide clear error messages when Jinja syntax cannot be processed
4. Validation Flag Behavior: Ensure --skip-validations false actually validates the final SQL syntax

Environment

• Transpiler: remorph
• Transpiler version: 1
• Source: Snowflake dbt demo project
• Target: Databricks DBT
• Validation mode: --skip-validations false

Additional Context

This issue affects multiple files in the project, as evidenced by the transpile_errors file which shows similar Jinja syntax errors across various dbt models.

Example Output:

Expected Behavior

Suggested Solutions

1. Proper Jinja Processing: Implement proper Jinja templating processing during transpilation
2. Accurate Status Reporting: Only report success if the final SQL is syntactically valid
3. Clear Error Handling: Provide clear error messages when Jinja syntax cannot be processed
4. Validation Flag Behavior: Ensure --skip-validations false actually validates DBT code through DBT instead of directly through Warehouse

Steps To Reproduce

1. Transpile Snowflake DBT code to Databricks DBT code with --skip-validations false

Sample CLI code
databricks labs lakebridge transpile --input-source /Users/.../dbt-cloud-snowflake-demo-template --source-dialect snowflake --output-folder /Users/.../dbt-cloud-databricks-demo-template --skip-validation False --catalog-name users --schema-name user

Relevant log output or Exception details

12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/dbt_project.yml: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/analysis/show_incremental_model.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_orders.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_part_suppliers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_suppliers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_regions.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_customers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_nations.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_line_items.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_parts.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/dim_suppliers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/incremental_dim_orders.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/dim_parts.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/fct_order_items.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/fct_orders.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/non_transient_dim_customers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/aggregates/agg_ship_modes_hardcoded_pivot.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/aggregates/agg_ship_modes_dynamic_pivot.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/intermediate/part_suppliers.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/intermediate/order_items.sql: 1  found
12:12:23  WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/macros/grant_all_on_schemas.sql: 1  found

Sample Query

{{
    config(
        materialized = 'table'
    )
}}
with part as (

    select * from {{ref('stg_tpch_parts')}}

),

final as (
    select 
        part_key,
        manufacturer,
        name,
        brand,
        type,
        size,
        container,
        retail_price
    from
        part
)
select *
from final  
order by part_key

Operating System

macOS

Version

latest via Databricks CLI

[sundarshankar89]

@matthewmoorcroft currently that is supported only for SQL. so you have to run with --skip-validations ture basically do not validate DBT outputs.

[sundarshankar89]

@matthewmoorcroft can you post the results with setting --skip-validations true

[matthewmoorcroft]

Hi Sundar, I posted in the ticket what it returned from a CLI perspective. Attached is the transpile errors file. I thought as much that it isn't supported. Docs should make it clearer with either a warning when DBT is detected or disabling validations when it is detected.

transpile_errors.txt

matthew.moorcroft@FXW760KWNP ~ % databricks labs lakebridge transpile --input-source /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template --source-dialect snowflake --output-folder /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-databricks-demo-template --skip-validation False --catalog-name users --schema-name matthew_moorcroft 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/dbt_project.yml: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/analysis/show_incremental_model.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_orders.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_part_suppliers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_suppliers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_regions.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_customers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_nations.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_line_items.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/staging/tpch/stg_tpch_parts.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/dim_suppliers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/incremental_dim_orders.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/dim_parts.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/fct_order_items.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/fct_orders.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/core/non_transient_dim_customers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/aggregates/agg_ship_modes_hardcoded_pivot.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/aggregates/agg_ship_modes_dynamic_pivot.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/intermediate/part_suppliers.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/models/marts/intermediate/order_items.sql: 1 found 12:12:23 WARNING [d.labs.lakebridge] /Users/matthew.moorcroft/Documents/GitHub/dbt-cloud-snowflake-demo-template/macros/grant_all_on_schemas.sql: 1 found