[Feature] Allow a new MergeBehavior for config.meta

[yidawang-shopify]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Currently, the field config.meta will be directly updated according to the hierarchically order of where this field is specified.

For example, at dbt_project.yml level, I have the definition:

models:
  +meta:
    sub_lvl_1_key_1:
      sub_lvl_2_key_1: 11
      sub_lvl_2_key_2: 22

At the same time, at schema.yml level for the specific model, I have the definition:

models:
  config:
    meta:
      sub_lvl_1_key_1:
        sub_lvl_2_key_3: 33

The merged config that I hope to be is:

models:
  config:
    meta:
      sub_lvl_1_key_1:
        sub_lvl_2_key_1: 11
        sub_lvl_2_key_2: 22
        sub_lvl_2_key_3: 33

But the actual config is the config at schema.yml level:

models:
  config:
    meta:
      sub_lvl_1_key_1:
        sub_lvl_2_key_3: 33

Describe alternatives you've considered

Our use case is that:

• at project level, we want every model comes with a default set of values on the config.meta
• at model level, we want to enrich the config.meta with additional values depends on the model

Without this approach, we will need to resolve to:

• at model level, fully specify the config.meta values, which would result duplicating lines of configurations across models

Who will this benefit?

For teams that manage large amount of models within a single dbt project, this enhancement will provide better management of configs.

Are you interested in contributing this feature?

Yes

Anything else?

I looked into this issue a little bit and it seems the behavior comes from here:

• https://github.com/dbt-labs/dbt-common/blob/main/dbt_common/contracts/config/base.py#L228
• https://github.com/dbt-labs/dbt-common/blob/main/dbt_common/contracts/config/base.py#L284

To be more specific, it is the default behavior of python.dict.update() that does not gracefully handle merge of nested dictionary:

• https://github.com/dbt-labs/dbt-common/blob/main/dbt_common/contracts/config/base.py#L244

Using the default update() method will override nested dictionary values instead of merging the two dictionary.

If we consider a new MergeBehavior or change MergeBehavior.Update behavior (instead of update(), possibly a deep merge), then this could help us address the requirements.