Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for GREAT Score [NIPS-2024] #2527

Open
wants to merge 45 commits into
base: dev_1.20.0
Choose a base branch
from

Conversation

lizaitang
Copy link

Description

This pull request adds the support of the BEYOND Detection method proposed in [1].

[1] GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models. Paper

Type of change

Please check all relevant options.

  • Improvement (non-breaking)
  • Bug fix (non-breaking)
  • [✔️] New feature (non-breaking)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

  • [✔️] Unit Test

Test Configuration:

  • OS: Ubuntu 22.04
  • Python version: 3.9.18
  • ART version: 1.18.1
  • PyTorch: 2.4.0
  • cuDNN version: 90100

Checklist

  • [✔️] My code follows the style guidelines of this project
  • [✔️] I have performed a self-review of my own code
  • [✔️] I have commented my code
  • [✔️] I have made corresponding changes to the documentation
  • [✔️] My changes generate no new warnings
  • [✔️] I have added tests that prove my fix is effective or that my feature works
  • [✔️] New and existing unit tests pass locally with my changes
  • [✔️] My changes have been tested using both CPU and GPU devices

dependabot bot and others added 2 commits October 23, 2024 07:46
Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.3.0 to 11.0.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.3.0...11.0.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
@beat-buesser beat-buesser self-requested a review December 3, 2024 16:43
@beat-buesser beat-buesser self-assigned this Dec 3, 2024
@beat-buesser beat-buesser added the enhancement New feature or request label Dec 3, 2024
Copy link

codecov bot commented Dec 10, 2024

Codecov Report

Attention: Patch coverage is 0% with 60 lines in your changes missing coverage. Please review.

Project coverage is 53.17%. Comparing base (970c74a) to head (ae33fe1).
Report is 108 commits behind head on dev_1.20.0.

Files with missing lines Patch % Lines
art/evaluations/great_art_test.py 0.00% 33 Missing ⚠️
art/evaluations/great_art.py 0.00% 27 Missing ⚠️

❌ Your project check has failed because the head coverage (53.17%) is below the target coverage (83.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff               @@
##           dev_1.20.0    #2527      +/-   ##
==============================================
- Coverage       53.38%   53.17%   -0.21%     
==============================================
  Files             658      670      +12     
  Lines           60292    61978    +1686     
  Branches        10274    10523     +249     
==============================================
+ Hits            32186    32956     +770     
- Misses          26450    27325     +875     
- Partials         1656     1697      +41     
Files with missing lines Coverage Δ
art/evaluations/great_art.py 0.00% <0.00%> (ø)
art/evaluations/great_art_test.py 0.00% <0.00%> (ø)

... and 19 files with indirect coverage changes


import numpy as np
import torch
from torch.autograd import Variable

Check notice

Code scanning / CodeQL

Unused import

Import of 'Variable' is not used.
great_scores = []
for i in range(len(samples)):
if correct_predictions[i]:
predicted_label = predicted_labels[i]

Check notice

Code scanning / CodeQL

Unused local variable

Variable predicted_label is not used.
# SOFTWARE.

from art.defences.evaluation.evasion import great_score
import numpy as np

Check notice

Code scanning / CodeQL

Unused import

Import of 'np' is not used.
@beat-buesser beat-buesser added this to the ART 1.20.0 milestone Dec 12, 2024
beat-buesser and others added 11 commits December 13, 2024 16:55
Signed-off-by: Beat Buesser <[email protected]>
correct f-strings in parallel/non-parallel AutoAttack metadata
Signed-off-by: Beat Buesser <[email protected]>
Bumps [six](https://github.com/benjaminp/six) from 1.16.0 to 1.17.0.
- [Changelog](https://github.com/benjaminp/six/blob/main/CHANGES)
- [Commits](benjaminp/six@1.16.0...1.17.0)

---
updated-dependencies:
- dependency-name: six
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
@beat-buesser beat-buesser changed the base branch from dev_1.19.0 to dev_1.20.0 January 16, 2025 13:12
beat-buesser and others added 10 commits January 17, 2025 11:36
Bumps [pandas](https://github.com/pandas-dev/pandas) from 2.2.2 to 2.2.3.
- [Release notes](https://github.com/pandas-dev/pandas/releases)
- [Commits](pandas-dev/pandas@v2.2.2...v2.2.3)

---
updated-dependencies:
- dependency-name: pandas
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
…das-2.2.3

Bump pandas from 2.2.2 to 2.2.3
Bumps [cma](https://github.com/CMA-ES/pycma) from 3.3.0 to 4.0.0.
- [Release notes](https://github.com/CMA-ES/pycma/releases)
- [Commits](CMA-ES/pycma@r3.3.0...r4.0.0)

---
updated-dependencies:
- dependency-name: cma
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
…low-11.0.0

Bump pillow from 10.3.0 to 11.0.0
Signed-off-by: Beat Buesser <[email protected]>
…2553

Fix missing transfer to device in ProjectedGradientDescentPyTorch
beat-buesser and others added 22 commits January 20, 2025 17:35
Updates the requirements on [kornia](https://github.com/kornia/kornia) to permit the latest version.
- [Release notes](https://github.com/kornia/kornia/releases)
- [Changelog](https://github.com/kornia/kornia/blob/main/CHANGELOG.md)
- [Commits](kornia/kornia@v0.7.1...v0.8.0)

---
updated-dependencies:
- dependency-name: kornia
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
…nia-approx-eq-0.8.0

Update kornia requirement from ~=0.7.1 to ~=0.8.0
Bumps [pillow](https://github.com/python-pillow/Pillow) from 11.0.0 to 11.1.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@11.0.0...11.1.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
…low-11.1.0

Bump pillow from 11.0.0 to 11.1.0
…_docs

Improve typing and documentation of AdversarialPatchPyTorch
Bumps [gpy](https://github.com/SheffieldML/GPy) from 1.13.1 to 1.13.2.
- [Release notes](https://github.com/SheffieldML/GPy/releases)
- [Changelog](https://github.com/SheffieldML/GPy/blob/devel/CHANGELOG.md)
- [Commits](SheffieldML/GPy@v1.13.1...v1.13.2)

---
updated-dependencies:
- dependency-name: gpy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Beat Buesser <[email protected]>
Bumps [catboost](https://github.com/catboost/catboost) from 1.2.3 to 1.2.7.
- [Release notes](https://github.com/catboost/catboost/releases)
- [Changelog](https://github.com/catboost/catboost/blob/master/RELEASE.md)
- [Commits](catboost/catboost@v1.2.3...v1.2.7)

---
updated-dependencies:
- dependency-name: catboost
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
…boost-1.2.7

Bump catboost from 1.2.3 to 1.2.7
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.5.5 to 0.9.3.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.5.5...0.9.3)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Beat Buesser <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants