Open-Vocabulary semantic segmentation

[zxqing01]

What parameters are fine-tuned for siglip2 using catseg's line for Open-Vocabulary semantic segmentation in this paper?

[ferjad]

Thanks for your interest in our work. The Cat-Seg experiment follows the protocol from the Cat-seg paper's v1 (https://arxiv.org/pdf/2303.11797v1). Quite briefly, on the vision side, the ViT is kept frozen and the attention layers are finetuned. On the text side, the text transformer is kept frozen. Additionally you add the segmentation related model components from Cat-Seg. We didn't change anything from Cat-seg's training protocol other than dropping in siglip2 as the contrastive model.

[zxqing01]

感谢您对我们的工作感兴趣。Cat-Seg 实验遵循 Cat-seg 论文 v1 （https://arxiv.org/pdf/2303.11797v1） 中的协议。简而言之，在视觉方面，ViT 保持冻结状态，注意力层被微调。在文本端，文本转换器保持冻结状态。此外，您还可以从 Cat-Seg 添加与分割相关的模型组件。我们没有改变 Cat-seg 的训练方案，除了将 siglip2 作为对比模型。

Thank you for your answer, I would like to ask, is the tokenizer and processor you use in the original catseg or the tokenizer and processor in siglip2? In addition, catseg made the image dense from 768 to 512 channels, did you use dense, or did you change 512 to 768 for subsequent operations? Fine-tuning is the [self_attn.q_proj,self_attn.k_proj,self_attn.v_proj,self_attn.out_proj] in the visual, right, does the learning rate of fine-tuning follow the learning rate of the original catseg fine-tuning clip?

• clip_features = self.sem_seg_head.predictor.clip_model.vision_model(clip_images_resized).last_hidden_state ，

• texts = self.tokenize(texts, padding="max_length", max_ length=64, return_tensors="pt")，

• texts = texts["input_ids"].cuda(),

• class_embeddings = clip_model.text_model(tokens).pooler_output，

Is the above code correct?

If you can answer my questions, I would be very grateful for your help.

[ferjad]

I ll have to check the code to see if we mapped from 768 to 512 but the goal is to stick to cat-seg as closely as possible so learning rate and all other hyper parameters including which layers are unfrozen are exactly the same. If the projection is an important detail for you, I can check it next week. The tokenizer for open vocabulary stage needs to match the one used for contrastive training so we use the same one as siglip2 for the experiments.

[zxqing01]

I ll have to check the code to see if we mapped from 768 to 512 but the goal is to stick to cat-seg as closely as possible so learning rate and all other hyper parameters including which layers are unfrozen are exactly the same. If the projection is an important detail for you, I can check it next week. The tokenizer for open vocabulary stage needs to match the one used for contrastive training so we use the same one as siglip2 for the experiments.

Thank you very much for your help! This project is very important to me. Currently, my open-vocabulary semantic segmentation model, built on a similar CatSeg model using CLIP, has achieved new SOTA results. However, when reproducing the CatSeg model using Siglip2, I noticed that the early checkpoint performance was suboptimal, and the final results were not as expected. I have been trying to find the root cause but have not yet found a clear solution.

If you could share some implementation details of Siglip2 or key code snippets from your use of Siglip2 with CatSeg, I would greatly appreciate it. I firmly believe that Siglip2 will provide a significant boost to my model, helping me achieve better performance.

Thank you again for your support! I look forward to your help. My email is: 2587721154@qq.com.

[ChaojunZ]

I ll have to check the code to see if we mapped from 768 to 512 but the goal is to stick to cat-seg as closely as possible so learning rate and all other hyper parameters including which layers are unfrozen are exactly the same. If the projection is an important detail for you, I can check it next week. The tokenizer for open vocabulary stage needs to match the one used for contrastive training so we use the same one as siglip2 for the experiments.

I have the same problem, how to solve this. @ferjad

[ferjad]

Did you folks change the image processing to be compatible with big vision checkpoints? Namely we do not normalize the image with mean and std but rather between -1 and 1.

[zxqing01]

Did you folks change the image processing to be compatible with big vision checkpoints? Namely we do not normalize the image with mean and std but rather between -1 and 1.

Thank you very much for your help, I found out that I was using

clip_images = [(x - self.clip_pixel_mean) / self.clip_pixel_std for x in images]

clip_images = ImageList.from_tensors(clip_images, self.size_divisibility)

clip_images_resized = F.interpolate(clip_images.tensor, size=self.clip_resolution, mode='bilinear', align_corners=False, )

I'll further replace it with

pp_img = pp_builder.get_preprocess_fn( f'resize({RES})|value_range(-1, 1)')
imgs = np.array([pp_img({'image': np.array(image)})['image'] for image in images])

and the clip used by catseg turns the number of channels 768 into 512 channels in the last layer through dense processing, have you been dense? The second question is: Does the loss function of catseg need to be modified? Do you have any plans to publish or share catseg_v1 code that uses siglip2?

[ChaojunZ]

Did you folks change the image processing to be compatible with big vision checkpoints? Namely we do not normalize the image with mean and std but rather between -1 and 1.

Thanks, I got catseg training done, but for testing catseg uses a sliding window, which produces floating-point values, but when I pass floating-point values to siglip2 processor, it throws an error. If I directly use image_norm = image / 255.0, the effect will be poor when passed to the processor. How should I deal with it? @ferjad