Double checking the num_layer assignment in mds_ensemble_postprocessor

[Dengyu-Wu]

line 192 num_layer = len(feature_type_list), which only ranges from 1-3 according to config file line 6. Is it correct?

1. Should it be num_layer = the number of layers or features?
2. Or only first feature would be used for the evaluation?

[zjysteven]

@Dengyu-Wu Thank you for pointing it out. Let me answer in two steps.

1. In the official implementation of Mahalanobis, the authors only picked the output of residual blocks (instead of each and every individual layer) to compute feature statistics. So num_layer should be equal to the number of residual blocks.
2. There indeed seems to be a minor bug, where with the current config file only the first residual block's output will be used. To include more blocks one may need to manually change feature_type_list in the config, which obviously is not robust and efficient. We will look into this the following days.

[Dengyu-Wu]

Thanks for the fast reply. I can understand that only outputs of the residual blocks would be used, but there is not only one residual connection, isn't it? My understanding is that num_layer would be a value related to the number of the features, not process_feature_type.

Another issue is that I tried 'flat' in line 6. It ran out of 64 GB CPU memory on Cifar10 and crashed the workstation. A fix of this issue would also be helpful.

[zjysteven]

Yes your understanding is correct and that is what I tried to explain in my point 2 in the previous comment.

The memory issue is expected and there won't be a "fix" for it because with 'flat' you will be flattening out the feature map and dealing with a very high-dimensional vector, which is expected to blow your memory. The original Mahalanobis implementation always takes the mean if I remember correctly, so for example if a feature map is 256 x 16 x 16, it will take the mean over the last two dimensions and get a 256-d vector, which will be used later to compute feature statistics and perform OOD detection (this is also achieved by OpenOOD when using 'mean'). Yet with 'flat', you will be using the whole 256x16x16=65536-d vector as the feature.

[Dengyu-Wu]

I see, thanks a lot.