DeviceType_Count on Manufacturer

[ZionDials]

Deployment Type

Self-hosted

Triage priority

I volunteer to perform this work (if approved)

NetBox Version

v4.1.6

Python Version

3.10

Steps to Reproduce

GET /api/dcim/device-types/

{
    "count": 2424,
    "next": "https://netbox.com/api/dcim/device-types/?limit=50&offset=50",
    "previous": null,
    "results": [
        {
            "id": 1112,
            "url": "https://netbox.com/api/dcim/device-types/1112/",
            "display_url": "https://netbox.com/dcim/device-types/1112/",
            "display": "Thunder 4440 ADC",
            "manufacturer": {
                "id": 47,
                "url": "https://netbox.com/api/dcim/manufacturers/47/",
                "display": "A10",
                "name": "A10",
                "slug": "a10",
                "description": ""
            },
            "default_platform": null,
            "model": "Thunder 4440 ADC",
            "slug": "a10-thunder-4440-adc",
            "part_number": "TH4440",
            "u_height": 1.0,
            "exclude_from_utilization": false,
            "is_full_depth": true,
            "subdevice_role": null,
            "airflow": null,
            "weight": null,
            "weight_unit": null,
            "front_image": null,
            "rear_image": null,
            "description": "",
            "comments": "[A10 Thunder 4440 Applcation Delivery Controller Datasheet](https://www.a10networks.com/wp-content/uploads/A10-DS-Thunder-ADC.pdf)",
            "tags": [],
            "custom_fields": {},
            "created": "2024-10-28T06:49:53.123905-04:00",
            "last_updated": "2024-10-28T06:49:53.123918-04:00",
            "device_count": 0,
            "console_port_template_count": 0,
            "console_server_port_template_count": 0,
            "power_port_template_count": 0,
            "power_outlet_template_count": 0,
            "interface_template_count": 8,
            "front_port_template_count": 0,
            "rear_port_template_count": 0,
            "device_bay_template_count": 0,
            "module_bay_template_count": 0,
            "inventory_item_template_count": 0
        }
    ]
}

Note "devicetype_count" is missing.

Expected Behavior

The expected response, according to the OpenAPI specification for Netbox:

GET /api/dcim/device-types/

{
    "count": 2424,
    "next": "https://netbox.com/api/dcim/device-types/?limit=50&offset=50",
    "previous": null,
    "results": [
        {
            "id": 1112,
            "url": "https://netbox.com/api/dcim/device-types/1112/",
            "display_url": "https://netbox.com/dcim/device-types/1112/",
            "display": "Thunder 4440 ADC",
            "manufacturer": {
                "id": 47,
                "url": "https://netbox.com/api/dcim/manufacturers/47/",
                "display": "A10",
                "name": "A10",
                "slug": "a10",
                "description": "",
                "devicetype_count": 1
            },
            "default_platform": null,
            "model": "Thunder 4440 ADC",
            "slug": "a10-thunder-4440-adc",
            "part_number": "TH4440",
            "u_height": 1.0,
            "exclude_from_utilization": false,
            "is_full_depth": true,
            "subdevice_role": null,
            "airflow": null,
            "weight": null,
            "weight_unit": null,
            "front_image": null,
            "rear_image": null,
            "description": "",
            "comments": "[A10 Thunder 4440 Applcation Delivery Controller Datasheet](https://www.a10networks.com/wp-content/uploads/A10-DS-Thunder-ADC.pdf)",
            "tags": [],
            "custom_fields": {},
            "created": "2024-10-28T06:49:53.123905-04:00",
            "last_updated": "2024-10-28T06:49:53.123918-04:00",
            "device_count": 0,
            "console_port_template_count": 0,
            "console_server_port_template_count": 0,
            "power_port_template_count": 0,
            "power_outlet_template_count": 0,
            "interface_template_count": 8,
            "front_port_template_count": 0,
            "rear_port_template_count": 0,
            "device_bay_template_count": 0,
            "module_bay_template_count": 0,
            "inventory_item_template_count": 0
        }
    ]
}

Observed Behavior

According to the OpenAPI specification BriefManufacturer has a required field of "devicetype_count". However, when querying DeviceTypes and ModuleTypes, the field is not populated and causes SDKs using the OpenAPI generator, to fail.

Go-Netbox Creating Manufacture errors out because of missing required property devicetype_count #165

[jeremystretch]

The devicetype_count field is being omitted from the nested representation of manufacturers because the dynamic annotation logic does not recurse for nested serializers. (The field is populated as expected when querying the manufacturers endpoint with ?brief=True.)

While it's feasible to fix this by explicitly attaching the annotation to DeviceTypeViewSet, we should instead extend the dynamic logic to account for nested fields, as it's very likely there are similar instances in other nested serializers that also need to be addressed.

[jeremystretch]

@ZionDials just realized that you volunteered for this. Assigning to you, thanks!

[jeremystretch]

@ZionDials are you still planning to work on this?

[ZionDials]

I've made several efforts to analyze the situation and identify a solution; however, I realize that I don't have the necessary expertise to resolve this issue effectively.

[arthanson]

Updated get_annotations_for_serializer which returns the correct annotations, but they aren't displayed.

def get_annotations_for_serializer(serializer_class, fields_to_include=None, annotations=None, base_name=''):
    """
    Return a mapping of field names to annotations to be applied to the queryset for a serializer.
    """
    if annotations is None:
        annotations = {}

    # If specific fields are not specified, default to all
    if not fields_to_include:
        fields_to_include = serializer_class.Meta.fields

    model = serializer_class.Meta.model

    for field_name, field in serializer_class._declared_fields.items():
        if field_name in fields_to_include:
            serializer_field = serializer_class._declared_fields.get(field_name)

            if type(field) is RelatedObjectCountField:
                related_field = model._meta.get_field(field.relation).field
                annotations[base_name + field_name] = count_related(related_field.model, related_field.name)
            elif issubclass(type(serializer_field), Serializer):
                subfields = serializer_field.Meta.brief_fields if serializer_field.nested else None
                annotations = get_annotations_for_serializer(serializer_field, subfields, annotations, f'{field_name}__')

    return annotations

might need to add the annotation directly on the prefetch, along the lines of:

queryset = Post.objects.all()\
    .prefetch_related(
        Prefetch(
            'comments',
             queryset=Comment.objects\
                 .filter(parent__isnull=True)\
                 .annotate(likes=Count('votes__choice'))\
                 .prefetch_related(
                    'children',
                    queryset=Comments.objects.all()\
                    .annotate(likes=Count('votes__choice'))
                 )
        )
    )

[Lucas-Eriksson]

Hi there! I am not sure if I should be commenting on this thread but I am experiencing a similar issue and thought it better to comment on here rather than opening a new thread.

When calling GET /api/virtualization/virtual-machines/{id} the OpenAPI specification and the actual response body does not match.

The request returns a 'VirtualMachineWithConfigContext' but is missing the nested properties 'virtualmachine_count' and/or 'device_count' for some of its properties. In the picture below, I have highlighted the affected properties.

The objects of the actual response looks like this.

"cluster": {
    "id": 1,
    "url": "",
    "display": "",
    "name": "",
    "description": ""
},
"role": {
    "id": 1,
    "url": "",
    "display": "",
    "name": "",
    "slug": "",
    "description": ""
},
"platform": {
    "id": 1,
    "url": "",
    "display": "",
    "name": "",
    "slug": "",
    "description": ""
}

Am I correct in assuming the fix you are working on in #18583 will resolve this as well? I am on version 4.2.2 of Netbox.

[renatoalmeidaoliveira]

After exploring this issue at #18583, I didn't find a proper solution for this issue, Here are my findings:

1 - There're some models' serializers that doesn't use brief_field in the nested fields, example, and the current implementation always use the brief_fields, code
2 - There're some models like PowerPort that defines a Serializer field inside the nested fields, like device and module_bay
3 - Some of the nested Serializer fields, have fields that requires annotations

Challenges:

When the nested Serializer field doesn't have Serializer fields, adding a Prefetch(field_name, qs=model.objects.all().annotate(**nested_annotations)) with nested_annotations as a result of get_annotations_for_serializer works fine.
In scenarios where the nested Serializer have Serializer fields, that approach fails when executed in a recursive mode, that fails happens by returning an empty queryset, resulting in a not found result in the API, it seens that it happens when the prefetch array have a Prefetch along with an inner nested field.