Questions about reproducibility

[soc12]

Hi,

Congratulations on the paper—it is truly interesting! I have a few questions regarding the implementation and the reproducibility of the results.

For the Cityscapes dataset, I downloaded the leftImg8bit images along with the gtFine annotations. Then, I used the CityscapesScripts to obtain the trainLabelIds. Is this the correct procedure to set up the data?

Additionally, I have some specific questions about the training script:

• In train.py, optimizer.zero_grad() appears to be missing. Is this intentional?
• The default learning rate in the code is 0.0001, whereas the paper mentions a learning rate of 0.004. Which one is correct?
• When enabling all the losses, the MAV loss becomes extremely large and the training becomes very unstable. Essentially, the model does not learn anything. I have tried using both learning rates mentioned above, as well as including and excluding optimizer.zero_grad(). Why is this happening? Are there specific hyperparameters required for the algorithm to function correctly?

Lastly, if possible, it would be incredibly helpful to have a set of instructions to reproduce the results.

Thanks!

[aggelos-michael-papadopoulos]

I am also encountering several issues with the current implementation:

The MAV loss consistently takes on a very large value, regardless of the learning rate used. Furthermore, the test function seems to incorrectly mark the unknown labels from the training set as known. If I am not mistaken, this is incorrect because these labels are used in the objectosphere loss as unknown and pushed to zero. Even when ignoring these mislabeled unknowns, the mIoU for the test set using only the contrastive decoder for knowns/unknowns yields poor results. The maximum mIoU for unknowns observed was only 0.07. This is in the case we training unknows as -1, knows as 0 and test unknowns as 1 with mIoU ignore_index = -1

Finally, there is a discrepancy between the objectosphere loss in the code and the one described in the paper. The paper specifies that the norm should be squared, but this is not reflected in the code. Additionally, in the code, the unknown component is multiplied by 10.

[Kimiarfaie]

Hi, I am facing the same issue with the loss (the feature loss) taking a very large value, has anyone found a solution to this problem?

[matteosodano]

Thank you for your feedback.

• The zero_grad instruction should indeed not be missing, as well as the squared norm in the objectosphere; this is likely an oversight from a previous code refactoring. I'll address this.
• Regarding the evaluation process, only the "unknown unknowns" are considered, while the "known unknowns" are discarded.
• The hyperparameters used in the experiments are detailed in the paper. The defaults provided in the code serve only as placeholders.
• The loss function is inherently unstable, and its behavior in TensorBoard does not resemble that of a smooth loss function. This is expected, as the target varies across epochs and is determined solely by a changing subset of pixels (the true positives of each class).

EDIT: I overlooked it when I commented, but yes it is intentional that the zero_grad is missing. param.grad = None for all model parameters is, in fact, more efficient, and used instead. I added a comment in the code about that.