On the effectiveness of the NaFlex variant for high-resolution images

[hm123450]

Dear Developers,

Your work on SigLIP2 is truly impressive. As a beginner, I would like to ask: can the NaFlex version of SigLIP handle images larger than 512×512, for example 896×896 (assuming I have already set max_num_patches to 1024)? Will the performance degrade significantly in such cases? If my future task involves training an LLM+ViT model, would it be possible to mitigate this degradation during training? I am very much looking forward to your reply.

Best regards,

[mitscha]

It sounds like you are referring to the HF implementation, so you'd need to check how max_num_patches is handled there under the hood.

Generally if you apply a NaFlex model to resolutions larger than the maximal training resolution/sequence length you might observe a (mild) regression. If you fine-tune the model in a sensible way though, it should be possible to get improvements for resolution-sensitive tasks. One thing to keep in mind is to increase the the positional embedding depending on how much beyond the training sequence length/resolution you want to go.