[Bug] dbt-core pydantic 2 support; package constraints do not match comments

[cburroughs]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

• https://pypi.org/project/dbt-core/1.10.0b3/#files
• wget https://files.pythonhosted.org/packages/64/4b/9131fd8b5f02a345e30a3df766af7f964c36c98a5a3c3c552d839b9d22ee/dbt_core-1.10.0b3-py3-none-any.whl

$ grep -i pydantic dbt_core-1.10.0b3.dist-info/METADATA
Requires-Dist: pydantic<2

Expected Behavior

Per https://github.com/dbt-labs/dbt-core/blame/1efad4e68ed44e53a2175b29484399029919d9ac/core/setup.py#L77 this section says " Expect compatibility with all new versions of these packages, so lower bounds only." but pydantic<2 is an upper bound.

Steps To Reproduce

See "Current Behavior".

Relevant log output

Environment

- OS: This version of dbt releases a `py3-none-any` wheel
- Python: `Requires-Python: >=3.9`
- dbt: `1.10.0b3`

Which database adapter are you using with dbt?

No response

Additional Context

• Pydantic version last changed in Support artifact upload #11419
• Prior Pydantic v2 discussion: `dbt-core` Python dependency audit #6915
• Pydantic v2 was released in 2023: https://docs.pydantic.dev/2.0/blog/pydantic-v2-final/

I understand this could be described as a bug since it 'regressed' in 1.10.x and doesn't match the comment, or a feature request for pydantic v2 support. I am not pedantic about pydantic and accept either triage.
The motivation is to use Python models with code that depends on pydantic>2

[cdchan]

This is affecting us too, since we do use pydantic > 2 (which has been out since 2023)

[JavierLopezT]

We are also affected by this

[menzenski]

Also impacted by this

[jtmcn]

Expect compatibility with all new versions of these packages, so lower bounds only.

@jtcohen6 it looks like you added this comment. Any thoughts or updates on this issue? We are unable to upgrade to 1.10 with this constraint in place

[miseyu]

Also impacted by this

[afranzi]

I'm quite surprised dbt-core forces pydantic<2 given how old that package is and how well adopted pydantic is in the Python community.

Do you have any plan to upgrade the version?

[taborsarah]

My team is also impacted by this

[afranzi]

@dataders do you know if the dbt team is aware of this dependency issue/constraint?

[lambda]

I believe that the following fix would work, but I don't really feel like dealing with the CLA process for such a small fix. This fix is de-minimis, I am releasing it under the public domain, anyone who feels like who has signed the CLA and is set up for contribution, feel free to submit a PR incorporating this.

Note that this would have the effect of transitively adding a dependency on pydantic 2, rather than pydantic 1. In our use case, all of the other dependencies in our dependency tree currently depend on pydantic 2, so this would be preferable to the current situation where we have to pin dbt-core to less than 1.10 to work around this.

modified   core/dbt/utils/artifact_upload.py
@@ -3,7 +3,7 @@ import uuid
 import zipfile
 
 import requests
-from pydantic import BaseSettings
+from pydantic-settings import BaseSettings
 
 import dbt.tracking
 from dbt.config.runtime import UnsetProfile, load_project
modified   core/setup.py
@@ -61,6 +61,7 @@ setup(
         "protobuf>=5.0,<6.0",
         "requests<3.0.0",  # should match dbt-common
         "snowplow-tracker>=1.0.2,<2.0",
+        "pydantic-settings>=2.0,<3.0",
         # ----
         # These packages are major-version-0. Keep upper bounds on upcoming minor versions (which could have breaking changes)
         # and check compatibility / bump in each new minor version of dbt-core.
@@ -82,7 +83,6 @@ setup(
         "pyyaml>=6.0",
         "daff>=1.3.46",
         "typing-extensions>=4.4",
-        "pydantic<2",
         # ----
     ],
     zip_safe=False,

[VincentChong123]

Impacted by this for dbt core 1.10.1 after installing airflow

[QMalcolm]

This appears to be an issue for both 1.10.1 and 1.10.0 (and all pre-release versions). We're working on a solution ASAP. We unfortunately need to support both pydantic v1 and pydantic v2 😞 So we'll unfortunately need a more robust solution than has been suggested.