Pydantic v2

[yaugenst-flex]

This PR completes the migration of our codebase from Pydantic V1 to Pydantic V2.

1. Imports & Dependency Updates:

• Standardized all Pydantic imports to use the pydantic namespace (and pydantic_settings for BaseSettings), removing pydantic.v1 compatibility imports.
• Switched to direct component imports (e.g., from pydantic import Field, model_validator).
• Modernized typing imports (e.g., Optional, Union, Self, TypeAlias) and other general Python type hinting.
• Updated project-specific imports, including removal of the skip_if_fields_missing decorator and replacing annotate_type with discriminated_union.
• Incorporated schema utilities from pydantic_core and pydantic.json_schema.

2. Field Definitions & Type-Hinting:

• Migrated pydantic.v1.Field usage to the new pydantic.Field syntax.
• Made field optionality explicit using Optional[T] and appropriate Field defaults (e.g., default=None, default_factory).
• Updated constrained types (e.g., PositiveFloat, NonNegativeInt) and replaced confloat with Annotated and Field.
• Standardized on default_factory for mutable defaults (e.g., dict, list, custom classes).
• Converted list defaults/assignments to tuples for immutability where appropriate.
• Defined fields with Unions of 'Traced' types for autograd compatibility.

3. Validation Logic:

• Replaced @pydantic.v1.validator with @pydantic.field_validator, adjusting signatures and return values.
• Migrated @pydantic.v1.root_validator to @pydantic.model_validator, specifying mode='before' or mode='after' and updating logic for field access (e.g., self.field in mode='after').
• Adjusted validator signatures, utilized the info object, and used direct attribute access/setting (e.g. object.__setattr__). Added @classmethod where needed.
• Removed the custom skip_if_fields_missing decorator, integrating its logic into model_validator or relying on V2's improved validation flow.
• Consolidated custom _post_init_validators logic into @model_validator(mode='after').

4. Configuration & Settings:

• Replaced the inner Config class with model_config = ConfigDict(...).
• Migrated pydantic.v1.BaseSettings to pydantic_settings.BaseSettings.
• Removed obsolete V1-specific configurations like allow_population_by_field_name.

5. Serialization & Parsing:

• Replaced .dict() with .model_dump() and .json() (or ._json()) with .model_dump_json().
• Updated data parsing from cls.parse_obj() to cls.model_validate().
• Streamlined serialization logic, removing custom encoders (e.g., ndarray_encoder) in favor of V2's built-in capabilities or Annotated serializers.
• Standardized exception handling to pydantic.ValidationError and PydanticSerializationError.

6. Schema & Custom Types:

• Replaced __get_validators__ with __get_pydantic_core_schema__ and __modify_schema__ with __get_pydantic_json_schema__ for custom type definitions.
• Refactored custom types using Pydantic V2's Annotated type along with BeforeValidator/AfterValidator and PlainSerializer/WrapSerializer.
• Replaced custom constrained_array with a new array_alias factory function using Annotated.
• Updated custom complex number types to Pydantic V2-compliant versions.
• Updated autograd utilities (e.g., TracedDict, get_static, hasbox).
• Utilized pydantic.PrivateAttr for non-schema model attributes.

7. Model Methods & Internal Refactors:

• Refactored copy() and updated_copy() methods to use model_copy(update=...).
• Adapted __init_subclass__ for V2 and updated docstring generation to use model_fields.
• Introduced new model traversal utilities (find_paths, find_submodels).
• Reworked __eq__ methods for V2 compatibility and updated/added __str__, __repr__, and __rich_repr__ for improved representations.
• Replaced __fields__ access with model_fields for introspection.

[momchil-flex]

It seems like fundamentally there's just a few things that change? Maybe I'm missing something important/tricky. Could you explain if there's anything like that, and what changes going forward?

Also very important - we will certainly need to update the backend python code too and might need to update denormalizer, metadata api, and who knows what else.

[yaugenst-flex]

Yes the biggest changes are related to custom types and serialization, everything else is relatively straightforward. I'll write something up.

[frederikschubertflex]

I wrote a little script for some deprecated symbols but was not sure whether you have already removed them or whether they are relevant. If not, you can ignore this comment @yaugenst-flex, otherwise I think its faster if you fix them?

parse_obj_as – deprecated; prefer TypeAdapter

./tidy3d/web/core/task_core.py
54: parse_obj_as(
127: parse_obj_as(
313: return parse_obj_as(list[SimulationTask], resp)

./tidy3d/web/api/material_libray.py
44: return parse_obj_as(list[MaterialLibray], resp) if resp else None

./tidy3d/web/api/material_library.py
61: return parse_obj_as(list[MaterialLibrary], resp) if resp else None

json_encoders – replaced by serializers

./tidy3d/plugins/adjoint/components/base.py
45: json_encoders={

parse_raw – removed; use model_validate_json()

./tidy3d/plugins/dispersion/web.py
297: best_medium = PoleResidue.parse_raw(run_result["message"])

[yaugenst-flex]

thanks @frederikschubertflex that's great. the adjoint plugin one is left intentionally but i missed those parse_ methods!

[daquinteroflex]

Thanks for migrating this! I've triggered the docs to see how it goes.

Did a small before and after check on the terminal:

Rectangular prism.
       Also base class for :class:`Simulation`, :class:`Monitor`, and :class:`Source`.

    Parameters
    ----------
    attrs : dict = {}
        Dictionary storing arbitrary metadata for a Tidy3D object. This dictionary can be freely used by the user for storing data without affecting the operation of Tidy3D as it is not used internally. Note that, unlike regular Tidy3D fields, ``attrs`` are mutable. For example, the following is allowed for setting an ``attr`` ``obj.attrs['foo'] = bar``. Also note that `Tidy3D`` will raise a ``TypeError`` if ``attrs`` contain objects that can not be serialized. One can check if ``attrs`` are serializable by calling ``obj.json()``.
    center : Union[tuple[Union[float, autograd.tracer.Box], Union[float, autograd.tracer.Box], Union[float, autograd.tracer.Box]], Box] = (0.0, 0.0, 0.0)
        [units = um].  Center of object in x, y, and z.
    size : Union[tuple[Union[pydantic.v1.types.NonNegativeFloat, autograd.tracer.Box], Union[pydantic.v1.types.NonNegativeFloat, autograd.tracer.Box], Union[pydantic.v1.types.NonNegativeFloat, autograd.tracer.Box]], Box]
        [units = um].  Size in x, y, and z directions.

    Example
    -------
    >>> b = Box(center=(1,2,3), size=(2,2,2))

after

>>> print(td.Box(size=(1,1,1)).__doc__)
Rectangular prism.
       Also base class for :class:`Simulation`, :class:`Monitor`, and :class:`Source`.

    Parameters
    ----------
    attrs : dict = dict()
        Dictionary storing arbitrary metadata for a Tidy3D object. This dictionary can be freely used by the user for storing data without affecting the operation of Tidy3D as it is not used internally. Note that, unlike regular Tidy3D fields, ``attrs`` are mutable. For example, the following is allowed for setting an ``attr`` ``obj.attrs['foo'] = bar``. Also note that `Tidy3D`` will raise a ``TypeError`` if ``attrs`` contain objects that can not be serialized. One can check if ``attrs`` are serializable by calling ``obj.json()``.
    center : Optional[tuple[Union[float, Annotated[autograd.tracer.Box, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined), PlainSerializer(func=<function traced_alias.<locals>.<lambda> at 0x734e1f509b20>, return_type=PydanticUndefined, when_used='json')], Annotated[object, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined)]], Union[float, Annotated[autograd.tracer.Box, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined), PlainSerializer(func=<function traced_alias.<locals>.<lambda> at 0x734e1f509b20>, return_type=PydanticUndefined, when_used='json')], Annotated[object, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined)]], Union[float, Annotated[autograd.tracer.Box, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined), PlainSerializer(func=<function traced_alias.<locals>.<lambda> at 0x734e1f509b20>, return_type=PydanticUndefined, when_used='json')], Annotated[object, BeforeValidator(func=<function traced_alias.<locals>._validate_box_or_container at 0x734e1f5099e0>, json_schema_input_type=PydanticUndefined)]]]] = None
        [units = um].  Center of object in x, y, and z.
    size : tuple
        [units = um].  Size in x, y, and z directions.

    Example
    -------
    >>> b = Box(center=(1,2,3), size=(2,2,2))

The parameter table reference change is kind of expected given the nature of the size of the changes. Just got to see how it renders, but I suspect there might be issues. I'm actually keen to try to straight up upgrade our docs APIs to pydantic v2 extension based on this branch so trying that now separately. This table was actually a bit of a hack so it's time to sort it out properly.

[yaugenst-flex]

oooh yeah the type annotations are now much more extensive and probably need to be trimmed down significantly. a lot of the validation is attached to the type annotations

[tylerflex]

I went through a bunch of files, but it's becoming clear I'm definitely not going to be able to go through all of this line by line. One question for now: what if any are the changes that a user would experience from this PR? A few that come to mind right now are:
a. I guess if the ArrayLike field is initialized as a list it will become np.ndarray instead of a tuple?
b. users will need to also use the pydantic v2 methods instead of the v1 methods, so their scripts could break if they rely on those?

are there any others? and have you run notebooks from this branch yet (out of curiosity...)?

[tylerflex]

not sure if previous comment got through , are we planning to ignore these warnings indefinitely?

[yaugenst-flex]

no this needs to go before 2.9, it's just for the rc's currently

[tylerflex]

what is this doing?