✨ Automatically map dictionaries to JSON

[patrick91]

No description provided.

[YuriiMotov]

Would be nice to have such native support for dict!

As an improvement idea, we can support automatic mapping to JSON for pydantic.Json type, BaseModel, and also for list (we can map it to JSON by default, but it will still be possible to change mapping to sqlalchemy.dialects.postgresql.ARRAY using the sa_type parameter).

One potential problem we should probably think about:
If we define our field as dict[str, int] it will not set any DB constraints and it will be possible to have invalid data in the DB (e.g. {"param": "text"}).

Code example in the details

from sqlmodel import Field, Session, SQLModel, create_engine, text


class MyObject(SQLModel, table=True):
    id: int = Field(primary_key=True)
    params: dict[str, int]

engine = create_engine("sqlite:///")
SQLModel.metadata.create_all(engine)


# Imagine this was added before, probably by external
with Session(engine) as session:
    session.exec(
        text('INSERT INTO myobject (id, params) VALUES (1, \'{"param": "text"}\')')
    )
    session.commit()

with Session(engine) as session:
    obj = session.get(MyObject, 1)
    assert isinstance(obj.params, dict)

    assert isinstance(obj.params["param"], int), type(obj.params["param"])
    # AssertionError: <class 'str'>

When we fetch such row, SQLModel will not complain that type is wrong, but users may have errors in their code if they assume that fetched data has valid type.
Such errors might be tricky to debug - static type checkers will not show any warnings.

As a solution, we can probably go extra mile and implement a new type (using [TypeDecorator](https://docs.sqlalchemy.org/en/20/core/custom_types.html)) that will not just convert data to JSON, but also ensure data has valid type.

PS: ready to help with it

[YuriiMotov]

This will still fail with Pydantic V1.
I suggest we keep with test, but just mark it with needs_pydanticv2

[YuriiMotov]

Suggested change

	This fixes the original error:
	ValueError: <class 'app.models.NeonMetadata'> has no matching SQLAlchemy type

I think we don't need this part here

[YuriiMotov]

As I understand, the idea of this test module is to test all type mapping in one module.
Then we should probably add tests for:

• Enum
• ipaddress.IPv4Address
• ipaddress.IPv4Network
• ipaddress.IPv6Address
• ipaddress.IPv6Network
• Path
• EmailStr
• timedelta
• Decimal
• uuid.UUID

[patrick91]

@YuriiMotov happy for you to add support for TypeDecorator, maybe we can ask @tiangolo if he has any opinions first? 😊

[patrick91]

@YuriiMotov also doesn't seems like SQLModel does validation for any incoming data? it would be a different behaviour from the rest?

[YuriiMotov]

@YuriiMotov also doesn't seems like SQLModel does validation for any incoming data? it would be a different behaviour from the rest?

It's true, but..
Currently, in most cases we will have validation on DB side (since we specify the types of columns and other constraints). So, assuming DB schema is in line with models, we don't need to validate it on SQLModel side (assuming we are using not SQLite, but more advanced DB).
Having Dict -> JSON mapper by default will increase the number of cases when data is not validated by any side..

[YuriiMotov]

As an idea: we can add such validation for dict - JSON mapping, but document how it can be disabled.
So, it will be enabled by default, but if it's not needed in particular project, developers will be able to disable it and avoid overheads.

For example

class MyModel(SQLModel, table=True):
    f1: dict[str, int]  # DB data will be validated
    f2: dict[str, int] = Field(sa_type=JSON)  # DB data will not be validated

[nkrishnaswami]

convert data to JSON, but also ensure data has valid type

I've had good luck using Pydantic's TypeAdapter in type decorators for this; it would let you support Pydantic types as well as builtins like Dict[str, int].