Why are the images generated by using DeepFool to attack CIFAR-10 so strange?

[leo-bleuming]

Describe the bug
Why are the images generated by using DeepFool to attack CIFAR-10 so strange?

It seems that this is related to the cnn model I built. When I use DeepFool to attack the CNN model in adversarial_training_cifar10.py, the attack can be successful and the perturbation is invisible to the naked eye. However, if I use the CNN model built by myself, the perturbation generated by using DeepFool to attack the CNN model is visible to the naked eye, and it cannot be adjusted through parameters.

Screenshots

System information (please complete the following information):

• OS win11
• Python version 3.8.20
• ART version or commit number 1.18.2
• TensorFlow-gpu = 2.6.0 / Keras = 2.6.0

the cnn model of mine
cnn_cifar10.txt

adversarial_training_cifar10.txt

[beat-buesser]

Hi @leo-bleuming Same question here as in the #2542 , does your model return logits or probabilities?

[leo-bleuming]

I repeated the experiment and found that the problem may not lie in my Convolutional Neural Network (CNN) model. I used the code "adversarial-robustness-toolbox\examples\adversarial_training_cifar10.py" to conduct the experiment. In the original code "adversarial_training_cifar10.py", I only added the code for saving the model and adversarial examples, without making any changes to the other parts. I named the modified code "adversarial_training_cifar10_savemodeladv.py". I ran this code twice and saved the results according to the experiment time. The only difference between the two experiments was the parameter "nb_epochs", which was set to 10 in one experiment and 100 in the other. That is to say, I trained models with different TOP - 1 metrics using the same CNN architecture.

I found that when using the DeepFool attack on the model with lower accuracy ("nb_epochs = 10"), large perturbations were less likely to occur, as shown in "x_train_DF_adv_cnn_2503061120.png" and "x_test_DF_adv_cnn_2503061120.png". However, when attacking the model with higher accuracy ("nb_epochs = 100"), extremely abnormal adversarial examples were generated, as shown in "x_train_DF_adv_cnn_2503061144.png" and "x_test_DF_adv_cnn_2503061144.png". The parameters used in the DeepFool attack were the default parameters in the code.

Regarding "Does you model return logits or probabilities? #2542 ", when I was running the code, I found a prompt saying "[WARNING] It seems that the attacked model is predicting probabilities. DeepFool expects logits as model output to achieve its full attack strength." , as shown in "xxx_runing_log.txt". What's your opinion on this?

Experimental environment:
Operating system: Windows 11
Python version: 3.8.20
ART version or commit number: 1.19.1
TensorFlow - gpu = 2.10.1 / Keras = 2.10.0

Now, I attach the "requirement_env.txt", experimental code "adversarial_training_cifar10_savemodeladv.py", running logs "xxx_runing_log.txt", as well as the visualization code and images of the adversarial examples"x_train_DF_adv_cnn_2503061120.png, x_test_DF_adv_cnn_2503061120.png, x_train_DF_adv_cnn_2503061144.png, x_test_DF_adv_cnn_2503061144.png". Here, "2503061120" corresponds to the experimental results of "nb_epochs = 10", and "2503061144" corresponds to the experimental results of "nb_epochs = 100".
requirement_env.txt
adversarial_training_cifar10_savemodeladv.txt
show_adv_image_singlepic.py.txt
adversarial_training_cifar10_savemodeladv_2503061120_running_log.txt
adversarial_training_cifar10_savemodeladv_2503061144_running_log.txt

DeepFool parameter:
def init(
self,
classifier: "CLASSIFIER_CLASS_LOSS_GRADIENTS_TYPE",
max_iter: int = 100,
epsilon: float = 1e-6,
nb_grads: int = 10,
batch_size: int = 1,
verbose: bool = True,
) -> None:
"""
Create a DeepFool attack instance.

    :param classifier: A trained classifier.
    :param max_iter: The maximum number of iterations.
    :param epsilon: Overshoot parameter.
    :param nb_grads: The number of class gradients (top nb_grads w.r.t. prediction) to compute. This way only the
                     most likely classes are considered, speeding up the computation.
    :param batch_size: Batch size
    :param verbose: Show progress bars.
    """

when nb_epoch=10，the visualization of adv is:


when nb_epoch=100，the visualization of adv is: