Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the nofuzz system to allow specific exclusions #2832

Open
wants to merge 11 commits into
base: fuzz-assignment
Choose a base branch
from

Conversation

MichaelChirico
Copy link
Collaborator

Continues the chain from #2818 to #2829 (the last of which this is currently based against).

I wasn't satisfied with the approach to just comment out & ignore large swaths of the test suite needed to implement # nofuzz through now, especially after the comment_injection_fuzzer turned out to require such large # nofuzz regions, but I was stuck on how to proceed with specific exclusions.

I think the approach taken here is pretty satisfactory: as a preprocessing step, we convert the # nofuzz markup into new code in the suite that signals activated/deactivated fuzzers for each execution of expect_lint().

The approach could be improved -- it can be a bit of a pain to debug, e.g. when expect_lint(file = .) was erroring due to encoding issues, the error causes the test to exit and thus fails to run the activate_fuzzers() step, leading to cascading confusing errors it's hard to pin down. And there's no simple workaround here, as e.g. using a withr::defer() approach to ensure the activate_fuzzers() step is run is not straightforward.

But I think there's enough good progress here (and it turned up a few more true positive fixes!) that it is ready for review.

Copy link

codecov bot commented Mar 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.64%. Comparing base (ad85636) to head (357f56f).

Additional details and impacted files
@@               Coverage Diff                @@
##           fuzz-assignment    #2832   +/-   ##
================================================
  Coverage            99.64%   99.64%           
================================================
  Files                  126      126           
  Lines                 7042     7043    +1     
================================================
+ Hits                  7017     7018    +1     
  Misses                  25       25           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant