Don't optimize one thing but measure another #17
Description
I've seen this in a fair number of papers recently, where one quantity is optimized and another is measured.
Examples of this mismatch include:
- conducting targeted attacks but measuring model accuracy (rather than success rate)
- conducting untargeted attacks to change top-1 classification but measuring top-5 accuracy
A related issue includes comparing two attacks that are not directly comparable, for example:
- conducting targeted attacks with (target = least likely class) and measuring accuracy, and comparing with prior work's results with accuracy on attacks of the form (target = random class)
It might be nice to include some text on this topic. Thoughts?