Building or running a model with dbt cloud for databricks fails with datavault4dbt is undefined

[kafkaguru]

Expected behavior

Given:
dbt Power User installed and configured to work with dbt cloud and
datavault4dbt package defined in packages.yml and
a dbt model created using a macro from the datavault4dbt package
When:
using dbt Power User to build or run the model
Then:
dbt builds the model successfully

Actual behavior

Given:
dbt Power User installed and configured to work with dbt cloud and
datavault4dbt package defined in packages.yml and
a dbt model created using a macro from the datavault4dbt package
When:
using dbt Power User to build or run the model
Then:
dbt fails with:

'datavault4dbt' is undefined. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps"

The dbt command that is ran when using dbt Power User is: dbt build --select my_model --source dbt-power-user --no-defer

Running this command from the cli results in the same error.

However, running the command: dbt build --select my_model --no-defer without the --source option succeeds

Steps To Reproduce

1. Install and configure dbt Power User to work with dbt cloud on databricks and validate setup
2. Add the following to your packages.yml

  - package: ScalefreeCOM/datavault4dbt 
    version: 1.8.1

2. Add the following to your vars in the dbt_project.yml file:

vars:
# datavault4dbt
  #Column Aliases
  datavault4dbt.ldts_alias: 'LOAD_DATETIME'
  datavault4dbt.rsrc_alias: 'SOURCE'
  datavault4dbt.ledts_alias: 'LOAD_END_DATETIME'
  datavault4dbt.snapshot_trigger_column: 'IS_ACTIVE'
  datavault4dbt.sdts_alias: 'SOURCE_DATETIME'
  datavault4dbt.is_current_col_alias: 'IS_CURRENT'
  datavault4dbt.deleted_flag_alias: 'DELETED_FLAG'
  escape_char_left: '`'
  escape_char_right: '`'

  #Hash Configuration
  datavault4dbt.hash: SHA1
  datavault4dbt.hash_datatype: STRING
  datavault4dbt.hashkey_input_case_sensitive: FALSE
  datavault4dbt.hashdiff_input_case_sensitive: TRUE
  
  # Stage Configuration
  datavault4dbt.copy_rsrc_ldts_input_columns: FALSE

  # General Configuration
  datavault4dbt.include_business_objects_before_appearance: false

  #Ghost Record Configuration
  datavault4dbt.beginning_of_all_times: {"databricks":"0001-01-01 00:00:00.001"}
  datavault4dbt.end_of_all_times: {"databricks":"9999-12-31 23:59:59.999"}
  datavault4dbt.timestamp_format: {"databricks":"yyyy-MM-dd HH:mm:ss.SSS"}
  
  datavault4dbt.beginning_of_all_times_date: {"databricks":"0001-01-01"}
  datavault4dbt.end_of_all_times_date: {"databricks":"9999-12-31"}
  datavault4dbt.date_format: {"databricks":"yyyy-MM-dd"}

  datavault4dbt.default_unknown_rsrc: 'SYSTEM'
  datavault4dbt.default_error_rsrc: 'ERROR'

  datavault4dbt.rsrc_default_dtype: {"databricks":"STRING"}
  datavault4dbt.timestamp_default_dtype: {"databricks":"TIMESTAMP"}
  datavault4dbt.stg_default_dtype: {"databricks":"STRING"}
  datavault4dbt.derived_columns_default_dtype: {"databricks":"STRING"}


  #Datatype specific default values
  datavault4dbt.error_value__STRING: '(error)'
  datavault4dbt.error_value_alt__STRING: 'e'
  datavault4dbt.unknown_value__STRING: '(unknown)'
  datavault4dbt.unknown_value_alt__STRING: 'u'
  datavault4dbt.unknown_value__numeric: '-1'
  datavault4dbt.error_value__numeric: '-2'

2. Create a table with columns: id: int, data: string
3. Insert a row into the table
4. Create a sources.yml file referencing the schema name and table you just created.
5. Create my_model.sql file containing:

{{ config(materialized='view') }}

{%- set yaml_metadata -%}
source_model:
    'schema': 'table'
hashed_columns: 
    HK_ID_H:
        - id
    HD_MODEL:
        is_hashdiff: true
        columns:
            - id
            - data

            
ldts: "CURRENT_TIMESTAMP()"
rsrc: '!schema.table
{%- endset -%}

{%- set metadata_dict = fromyaml(yaml_metadata) -%}


{{ datavault4dbt.stage(source_model=metadata_dict['source_model'],
                    ldts=metadata_dict['ldts'],
                    rsrc=metadata_dict['rsrc'],
                    hashed_columns=metadata_dict['hashed_columns'],
                    derived_columns=none,
                    missing_columns=none,
                    prejoined_columns=none,
                    enable_ghost_records=false,
                    include_source_columns=true) }}

6. Right click on model (or select build or run from drop down icon) and select build or run model
7. Watch for failure of resulting run. e.g.

Executing task: dbt build --select my_model --source dbt-power-user --no-defer
Sending project to dbt cloud
Created invocation id ##############
Waiting for response
Streaming invocation logs
20:59:39 Running dbt...
20:59:41 Found 39 models, 4 data tests, 1 snapshot, 95 sources, 1015 macros
20:59:41
20:59:41 Concurrency: 4 threads (target='dev')
20:59:41
20:59:44 1 of 1 START sql view model schema.my_model ........... [RUN]
20:59:45 1 of 1 ERROR creating sql view model schema.my_model .. [ERROR in 1.45s]
20:59:46
20:59:46 Finished running 1 view model in 0 hours 0 minutes and 4.91 seconds (4.91s).
20:59:46
20:59:46 Completed with 1 error, 0 partial successes, and 0 warnings:
20:59:46
20:59:46
20:59:46 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 TOTAL=1
20:59:46 Compilation Error in model my_model (models/my_model.sql)
'datavault4dbt' is undefined. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".
Downloading artifacts
Invocation has finished

Log output/Screenshots

No response

Operating System

MacOS

dbt version

dbt Cloud CLI - 0.38.23 (1ab8d5b0dd92df4e0f13549b04ab88abadcd12e3 2025-01-07T20:26:26Z)

dbt Adapter

databricks

dbt Power User version

0.50.5

Are you willing to submit PR?

• Yes I am willing to submit a PR!