ObjectType instantiation and pydantic datamodel

[CagtayFabry]

I started playing around with this package and found it really great, especially using pydantic in the background makes some things very smooth and clear. 👍

However, I also stumbled upon a case that was quite confusing for me: What happens when I instantiate an ObjectType(or more specifically one of its derived classes like ExperimentalStep )

I will try to explain my understanding and source of confusion:

Looking at bam_masterdata.datamodel_object_types.ExperimentalStep:

class ExperimentalStep(ObjectType):
    defs = ObjectTypeDef(
        code="EXPERIMENTAL_STEP",
        description="""Experimental Step (generic)//Experimenteller Schritt (allgemein)""",
        generated_code_prefix="EXP",
    )

    name = PropertyTypeAssignment(
        code="$NAME",
        data_type="VARCHAR",
        property_label="Name",
        ...
    )

    show_in_project_overview = PropertyTypeAssignment(
        code="$SHOW_IN_PROJECT_OVERVIEW",
        data_type="BOOLEAN",
        property_label="Show in project overview",
        ...
    )

    finished_flag = PropertyTypeAssignment(
        code="FINISHED_FLAG",
        data_type="BOOLEAN",
        property_label="Experiment completed",
        ...
    )
...

we can see that the class attributes are assigned to describe the ExperimentalStep object type, which mostly concerns the PropertyTypeAssignments alongside some metadata.

My problem started when I wanted to use ExperimentalStep to instantiate an instance (duh..) that could be uploadedto openBIS later. Since I didn't remember all the possible attributes to define an experimental step, I looked up the init docstring/signature. However all I could find was

Init signature:
ExperimentalStep(
    *,
    properties: list[bam_masterdata.metadata.definitions.PropertyTypeAssignment] = [],
) -> None

Slightly confusing at first, but it seems to come from bam_masterdata.datamodel.metadata.ObjectType which lists propertiesas a field:

class ObjectType(BaseEntity):
    properties: list[PropertyTypeAssignment] = Field(
        default=[],
        description="""
        List of properties assigned to an object type. This is useful for internal representation of the model.
        """,
    )

I initally suspected some issues with docstring inheritance, but looking into it it became clear that these properties never get assigned as pydantic Fields. It took me a while to understand the parsing that ultimately assigns (and validates) the user supplied kwargs for class instantiation into class attributes in ObjectType.__setattr__

A few caveats I noticed with the current approach:

• when supplied with a value, ObjectType.__setattr__ will basically override the previously defined class value:

  ExperimentalStep().name
  >>> PropertyTypeAssignment(code='$NAME', description='Name' ....
  print(ExperimentalStep(name="experiment name").name)
  >>> experiment name

  So the attributes are either the PropertyAssignments or the value. Normally I would expect something like None for optional values.

• The type validation has to be done by the library in ObjectType.__setattr__, which I imagine could be quite difficult sometimes.

• if the properties would be actual fields, we could also leverage pydantics conversion methods, e.g. creating datetime.datetimefrom string like this would work:

  class MyExperimentalStep(BaseModel):
      name: str 
      start_date: datetime
  MyExperimentalStep(name="d", start_date="2025-01-01")
  >>> MyExperimentalStep(name='d', start_date=datetime.datetime(2025, 1, 1, 0, 0))

  currently it fails because we have to supply a datetime directly

  ExperimentalStep(name="a new name", start_date="2025")
  >>> TypeError: Invalid type for 'start_date': Expected datetime, got str

• understanding and inspecting the code (I guess also for IDEs) get's a bit more complicated due to the custom logic handling the properties and assignments. E.g. one cannot really use ExperimentalStep.__pydantic_fields__ to get information about the class.

So after all this, I am wondering would it be worth it to try and refactor the code to convert the proprties into actual pydantic Fields ?
(disclaimer: I haven't looked into the code that actually generates the python class definitions yet)

[JosePizarro3]

So after all this, I am wondering would it be worth it to try and refactor the code to convert the proprties into actual pydantic Fields ?

This is indeed a very good point. As stupid as it sounds, when I started working on the abstraction of the data structures (i.e., the definitions you can find in bam_masterdata/metadata/entities.py and bam_masterdata/metadata/definitions.py), I honestly didn't know how similar to a pydantic-like model this would look. What confused me most at the time were the data_type a property assigned to an object can have, especially when coming to the equivalents of python types. I was also expecting to deviate a bit from Fields when (if eventually) data types in openBIS become arrays (i.e., they have a finite shape).

But like I said, you raised a very good point. I will take a look later this year, as for now we can keep it like this and have it in mind.

---

Regarding your other points, now there is a couple of (very very similar, I would like to get rid off one of them) private variables for an ObjectType: _properties_metadata and _properties, similar to __pydantic_fields__. The first is a list of dictionaries containing the metadata of each property assigned, like:

            {
                "name": PropertyTypeAssignment(
                    code="$NAME",
                    data_type=VARCHAR,
                    mandatory=True,
                    property_label="Name"
                ),
                "age": PropertyTypeAssignment(
                    code="AGE",
                    data_type=INTEGER,
                    mandatory=False,
                    property_label="Age"
                ),
                ...
            }

The other is a dictionary only with the data_type (yeah, I want to get rid off that one...).

The idea is to keep the info of the PropertyTypeAssignment metadata to validate when usign __setattr__. I think something similar is done in pydantic, so I think if you would do print(ExperimentalStep(name="experiment name").name) you will still get "experiment name".

---

@LucasZimm also worked a lot on the parsing infrastructure (on the backend, this means __setattr__ and the abstract classes for a parser). He wrote some documentation who will be soon on the docu page, and hopefully this clarifies how to use this from an end user perspective when writing a parser.

[CagtayFabry]

Thanks for the detailed response!

Just a follow-up thought: Let's leave out the "metadata" part of describing the ObjectType-Class and only use the classe to instanciate new objects.

If for example "ExperimentalStep" was a pure pydantic class like

class ExperimentalStep(BaseModel):
    name: str 
    start_date: datetime

we can skip the custom __setattr__ implementation?

Maybe putting both the ObjectType-Class definitions and the instanciation into one class is too complex?
I have some ideas we can discuss soon if you are interested 😃

[JosePizarro3]

Indeed, only problem I have with that is that we would need more metadata for each field (mandatory, property_label, etc)