Support adapter specific config

[ShaneMazur]

What is the best approach to allow for adapter specific configuration. I've encountered issues trying to use the YAML validation on a dbt_project.yml on dbt-databricks with their custom configuration.

I noticed some other issues in the repo for similar issues with other adapters. For example #176.

I imagine adding all possible options (of every adapter) to a single config defeats the purpose of the validation schema. What is the suggested way to support adapter specific config i.e. dbt-core yaml schema + <my-adapter(s) yaml schema>

[joellabes]

@ShaneMazur good question! I've gone back and forth on this - I originally thought we might wind up with a schema per adapter and version, but the combinatorial explosions sounded like a phenomenal headache. JSON Schema theoretically supports composition of multiple schemas but their examples are all toys and don't really get into the idea of "patching" an adapter's extensions into the base stuff. To say nothing of the inheritance chain of adapters like Redshift, which themselves extend Postgres!

The way I'm thinking about it now is that the JSON Schema represents things that are legal in a dbt project, not necessarily your dbt project. Adapter-specific configs should have that called out in their description (like this) but it's fine for them to coexist. So feel free to add the missing dbt-databricks configs 🙏

[kumpirusha]

Hi, we are also facing some adapter specific issues when validating dbt_project.yml file (we are using Athena adapter), especially with missing support for parquet format and iceberg table types. Is this also something that we should look to change on our side by expanding/modifying the existing jsonschema config file? Thanks

[joellabes]

@kumpirusha yep!