Question Regarding Ground-Truth Mask Usage During Training and Potential Information Leakage

[c1ircle]

Hello,

I have a question regarding the use of ground-truth masks during the training process. Specifically, I would like to understand whether using the ground-truth masks in the training phase can lead to information leakage.

From my understanding, using ground-truth masks directly during training could potentially allow the model to access the actual target information, which might cause overfitting and hinder the model's ability to generalize to unseen data. This seems to suggest that it may lead to data leakage during training, as the model is not truly learning to predict the masks, but rather directly relying on the target information.

Is this correct? Or, are there any specific scenarios where using ground-truth masks during training is acceptable without causing issues such as information leakage?

I'd appreciate any insights on this topic.

Thanks in advance!

[owl-10]

Firstly, regarding the concern about information leakage, for standard open-vocabulary segmentation, models are typically trained on datasets such as COCO panoptic or COCO stuff and then evaluated on different datasets like ADE20k or Pascal-VOC. In the context of our mask adapter training, we utilize both ground-truth masks from COCO and masks generated by Mask2Former. These masks are fed into the mask adapter to produce semantic activation maps and class scores, aiming to align the mask embeddings with text embeddings. During inference, performed on images not seen during training (like those in ADE20k), we first employ mask generators such as SAM or Mask2Former to predict masks. These predicted masks are then input into the mask adapter to extract their semantic activation maps. Since ground-truth masks are not used at all during the inference stage, there is no information leakage in this process.
Secondly, it's standard practice in both semantic and open-vocabulary segmentation to use ground-truth masks and corresponding labels during training, so I'm not entirely clear on the specific nature of the concern about using ground-truth masks in this context. Thirdly, concerning the point about causing overfitting and hindering generalization to unseen data, our mask adapter's output is limited to semantic activation maps. These maps are subsequently aggregated with CLIP features and matched against text embeddings to determine the class. We are not training the masks themselves; rather, we are training a mechanism that generates a better map than a simple mask to more effectively aggregate features within the mask region from the CLIP features. Crucially, the CLIP features themselves remain unchanged, and therefore, we do not compromise the inherent generalization capabilities of CLIP.