Add DINOv3ViTForImageClassification support by dimidagd · Pull Request #41224 · huggingface/transformers

dimidagd · 2025-09-30T13:00:02Z

@yonigozlan @molbap

molbap

Thanks! Left two comments

src/transformers/models/dinov3_vit/modular_dinov3_vit.py

tests/models/dinov3_vit/test_modeling_dinov3_vit.py

src/transformers/models/dinov3_vit/modeling_dinov3_vit.py

dimidagd · 2025-10-01T10:37:49Z

Discussed reviewers comments, no immediate points of action. Left for future

Refactoring to remove redundant code between Dinov2 and DINOv3
Convert meta checkpoint with classifier and add a regression test

@molbap would you like to approve this PR?

molbap

Sure, just need to check get_input_embeddings, as in most cases it's not needed to add it and we prefer to add as little code as possible. It's not needed in Dinov2, why here?
Otherwise LGTM and ok to merge once this is sorted out!

dimidagd · 2025-10-07T11:44:41Z

@molbap what is the process for merging PRs after they have been approved? Trying to understand better the contribution workflow

Best
DD

molbap

I have a pending question on which test was failing for the embeddings, please let me know! apart from that for the review process when reviewers have approved we ping the core maintainers to approve and merge your PR!

This one in particular launched an internal discussion since we're trying to minimize the maintenance surface, see here #41450 but it'll wait a follow-up PR.

src/transformers/models/dinov3_vit/modeling_dinov3_vit.py

molbap

Added a missed head_mask, other tham that ping me when you know for the failing test and we can merge this! With #41276 it's nice additions to dinov3

src/transformers/models/dinov3_vit/modular_dinov3_vit.py

dimidagd · 2025-10-16T19:42:41Z

Added a missed head_mask, other tham that ping me when you know for the failing test and we can merge this! With #41276 it's nice additions to dinov3

Here is the failing test

https://app.circleci.com/jobs/github/huggingface/transformers/1982122

dimidagd · 2025-10-16T20:03:49Z

Added a missed head_mask, other tham that ping me when you know for the failing test and we can merge this! With #41276 it's nice additions to dinov3

Here is the failing test

https://app.circleci.com/jobs/github/huggingface/transformers/1982122

on a second thought, I removed the associated test in 1fb6000

Looking forward to your feedback

dimidagd · 2025-10-27T14:46:00Z

Added a missed head_mask, other tham that ping me when you know for the failing test and we can merge this! With #41276 it's nice additions to dinov3

@molbap upon rereading your comments, I think you implied that adding the get_input_embeddings should not be necessary.

Here is the failing test, which was copied over from dinov2.

https://app.circleci.com/jobs/github/huggingface/transformers/1993707

molbap · 2025-10-31T15:03:11Z

Hi @dimidagd , circling back to this, this is to support the classification head released by Meta for dinov3, right? In that case, it would also be needed to have a small test to load that head and test it works to avoid future regressions. I'm referring to this https://github.com/facebookresearch/dinov3?tab=readme-ov-file#pretrained-heads---image-classification

That way there'd be more ground to support the implementation of a new head!

dimidagd · 2025-10-31T15:07:51Z

@molbap I will see if I can find the time for this, what are your thoughts on the `get_input_embeddings` test?

…

On Fri, 31 Oct 2025 at 16:03, Pablo Montalvo ***@***.***> wrote: *molbap* left a comment (huggingface/transformers#41224) <#41224 (comment)> Hi @dimidagd <https://github.com/dimidagd> , circling back to this, this is to support the classification head released by Meta for dinov3, right? In that case, it would also be needed to have a small test to load that head and test it works to avoid future regressions. I'm referring to this https://github.com/facebookresearch/dinov3?tab=readme-ov-file#pretrained-heads---image-classification That way there'd be more ground to support the implementation of a new head! — Reply to this email directly, view it on GitHub <#41224 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALECAUJ5KWWT4LDURHPDCB332N24LAVCNFSM6AAAAACH4V2IVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINZTGQ4DGOJWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- [image: photo] *Dagdilelis Dimitris | PhD* m:+45 52 60 18 25 <+45+52+60+18+25> | ***@***.*** | a:Gullandsgade 18, 2800 Lyngby <http://us.linkedin.com/in/dimidagd>

merveenoyan · 2025-10-31T16:05:02Z

hey @dimidagd thanks a lot for working on this!

Meta has three checkpoints for image classification (with linear layer), object detection (DINO3+DETR) and depth estimation (DINO3+DPT). Would you like to load image classification weights and push to Hub for people's convenience?

We have DETR and DPT officially supported in transformers. Would you be down to convert the depth and detector checkpoints too?

dimidagd · 2025-11-06T15:14:30Z

hey @dimidagd thanks a lot for working on this!

Meta has three checkpoints for image classification (with linear layer), object detection (DINO3+DETR) and depth estimation (DINO3+DPT). Would you like to load image classification weights and push to Hub for people's convenience?

We have DETR and DPT officially supported in transformers. Would you be down to convert the depth and detector checkpoints too?

Hi @merveenoyan! My dev machine is on codespaces, with limited memory, therefore I can't even load their checkpoints. Do you have access to resources I could use?

merveenoyan · 2025-11-07T14:04:06Z

@dimidagd can you use a Colab? 👀 using small checkpoints to validate implementation is ok!

dimidagd · 2025-11-07T15:39:41Z

@dimidagd can you use a Colab? 👀 using small checkpoints to validate implementation is ok!

@merveenoyan hi again, meta released the classification head only for the 7B model. I can't fit that on colab resources either (GPU or CPU). I can't imagine myself implementing such tests from a colab env tbh.

molbap · 2025-11-07T17:10:30Z

@dimidagd Just in case, we have a run-slow CI which we can run, which runs tests with a @slow marker and fits a 7B. If you write a corresponding ...IntegrationTest, we can run it on our large github runner and test out the result, just ping me and I'll run it

- Slow test checking logits and predicted class based on COCO cat sample - Migrated tests from Dinov2

…dinov3`. When the backbone is stored under the wrong attribute, both `AutoModelForImageClassification.from_pretrained` and `DINOv3ViTForImageClassification` created from a headless checkpoint fail to load weights correctly because the state dict cannot map the backbone parameters. This change aligns the class with the expected naming used by the loader utilities and restores correct weight loading behavior.

…quirement

molbap · 2025-11-18T17:02:28Z

run-slow: dinov3_vit

github-actions · 2025-11-18T17:03:44Z

This comment contains run-slow, running the specified jobs:

models: ["models/dinov3_vit"]
quantizations: []

github-actions · 2025-11-18T17:25:03Z

CI Results

Workflow Run ⚙️

Model CI Report

❌ Failed tests

dinov3_vit:
tests/models/dinov3_vit/test_modeling_dinov3_vit.py::DINOv3ViTModelIntegrationTest::test_inference_lc_head_imagenet

dimidagd · 2025-11-21T12:17:04Z

@molbap I moved the tests on cpu using the large runner, had a GPU OOM error before, hopefully it has enough RAM to run the test. And by the way, tests based on facebook/dinov3* checkpoints are failing on that runner, because the repo is gated.

dimidagd · 2025-11-27T17:01:37Z

@molbap let me know if any further improvements are needed on this :)

dimidagd · 2026-01-21T09:11:32Z

@molbap pinging you to hear more abt this PR :)

github-actions · 2026-01-21T09:12:51Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, dinov3_vit

github-actions · 2026-01-21T09:20:26Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41224&sha=466177

molbap

Hello! Now that this is merged with main, some things have changed a bit in terms of code quality. The rest is ok

molbap · 2026-01-21T10:04:18Z

src/transformers/models/dinov3_vit/modeling_dinov3_vit.py

+        pixel_values: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        **kwargs: Unpack[TransformersKwargs],


with v5 around the corner:
now you should use make style && make fix-repo before pushing to make CI happy(happier). in this particular occurence we don't use Optional anymore as you can see on the failed tests (just default to None + use | is enough)

molbap · 2026-01-21T10:07:20Z

tests/models/dinov3_vit/test_modeling_dinov3_vit.py

        self.assertIsNotNone(model)

+    @slow
+    def test_model_for_image_classification_from_pretrained(self):


will also require the large accelerator no? Let's remove it, it's tested in the Integration test below

molbap · 2026-01-21T10:09:27Z

docs/source/en/model_doc/dinov3.md

Let's add to the doc in usage snippet an example with the usage now possible with the image classification head, with the example of the one converted from Meta

dimidagd force-pushed the feat/dinov3forimageclassification branch 6 times, most recently from ebb610b to 0462fb0 Compare September 30, 2025 13:53

molbap reviewed Sep 30, 2025

View reviewed changes

src/transformers/models/dinov3_vit/modular_dinov3_vit.py Show resolved Hide resolved

tests/models/dinov3_vit/test_modeling_dinov3_vit.py Show resolved Hide resolved

molbap reviewed Sep 30, 2025

View reviewed changes

src/transformers/models/dinov3_vit/modeling_dinov3_vit.py Outdated Show resolved Hide resolved

molbap approved these changes Oct 1, 2025

View reviewed changes

molbap reviewed Oct 8, 2025

View reviewed changes

src/transformers/models/dinov3_vit/modeling_dinov3_vit.py Outdated Show resolved Hide resolved

molbap reviewed Oct 9, 2025

View reviewed changes

src/transformers/models/dinov3_vit/modular_dinov3_vit.py Outdated Show resolved Hide resolved

dimidagd force-pushed the feat/dinov3forimageclassification branch from fce9f88 to 548597e Compare October 16, 2025 16:03

dimidagd force-pushed the feat/dinov3forimageclassification branch from 1fb6000 to 416d0b2 Compare October 16, 2025 20:05

dimidagd force-pushed the feat/dinov3forimageclassification branch 5 times, most recently from 63639c5 to 2c0786b Compare November 14, 2025 13:49

dimidagd added 5 commits November 18, 2025 12:59

Add Dinov3ViT support for image-classification task

30d0486

update auto-modeling configuration

eb38559

write tests

3cb3931

- Slow test checking logits and predicted class based on COCO cat sample - Migrated tests from Dinov2

docs: Update docs to reflect support for classification task

c1eca08

dimidagd force-pushed the feat/dinov3forimageclassification branch from ea1c1fb to fb8d4c4 Compare November 18, 2025 12:59

Uses @require_torch_large_accelerator to run test with high VRAM re…

bcdd666

…quirement

Merge branch 'huggingface:main' into feat/dinov3forimageclassification

170d639

dimidagd force-pushed the feat/dinov3forimageclassification branch from 73c9fca to 83ca9ca Compare November 21, 2025 12:15

dimidagd force-pushed the feat/dinov3forimageclassification branch from 83ca9ca to 6914a43 Compare November 24, 2025 12:31

fix: moves tests to cpu because not enough VRAM avail

2e9a1c2

dimidagd force-pushed the feat/dinov3forimageclassification branch from 6914a43 to 2e9a1c2 Compare November 24, 2025 13:49

dimidagd added 7 commits November 27, 2025 18:01

Merge branch 'main' into feat/dinov3forimageclassification

345c931

Merge branch 'main' into feat/dinov3forimageclassification

860b438

Merge branch 'main' into feat/dinov3forimageclassification

0ec8f13

Merge branch 'main' into feat/dinov3forimageclassification

56acbc3

Merge branch 'main' into feat/dinov3forimageclassification

c168733

Merge branch 'main' into feat/dinov3forimageclassification

5fb2b74

Merge branch 'main' into feat/dinov3forimageclassification

563fb43

Merge branch 'main' into feat/dinov3forimageclassification

466177a

molbap reviewed Jan 21, 2026

View reviewed changes

ArthurZucker removed their request for review February 11, 2026 12:32

Comments

Conversation

dimidagd commented Sep 30, 2025

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimidagd commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

dimidagd commented Oct 7, 2025

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimidagd commented Oct 16, 2025

Uh oh!

dimidagd commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimidagd commented Oct 27, 2025

Uh oh!

molbap commented Oct 31, 2025

Uh oh!

dimidagd commented Oct 31, 2025 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merveenoyan commented Oct 31, 2025

Uh oh!

dimidagd commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merveenoyan commented Nov 7, 2025

Uh oh!

dimidagd commented Nov 7, 2025

Uh oh!

molbap commented Nov 7, 2025

Uh oh!

molbap commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025

CI Results

Model CI Report

❌ Failed tests

Uh oh!

dimidagd commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimidagd commented Nov 27, 2025

Uh oh!

dimidagd commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

github-actions bot commented Jan 21, 2026

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

molbap Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

molbap Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

molbap Jan 21, 2026

Choose a reason for hiding this comment

dimidagd commented Oct 1, 2025 •

edited

Loading

dimidagd commented Oct 16, 2025 •

edited

Loading

dimidagd commented Oct 31, 2025 via email •

edited

Loading

dimidagd commented Nov 6, 2025 •

edited

Loading

dimidagd commented Nov 21, 2025 •

edited

Loading