Skip to content

Add counter validation against CUTEst native reporting #463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

arnavk23
Copy link
Contributor

@arnavk23 arnavk23 commented Jul 12, 2025

This PR implements comprehensive validation of evaluation counters by comparing CUTEst.jl's internal counter tracking with CUTEst's native reporting functions.

  • test/test_counters.jl: Complete test suite for counter validation
  • get_cutest_counters(): Utility function that extracts counters from CUTEst's native reporting functions
  • test_counter_validation(): Main test function with separate validation for unconstrained and constrained problems

Closes #278

arnavk23 and others added 5 commits July 11, 2025 11:35
…n permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Potential fix for code scanning alert no. 8: Workflow does not contain permissions
@arnavk23
Copy link
Contributor Author

arnavk23 commented Jul 12, 2025

@amontoison Please review the changes made here.

Copy link
Member

@tmigot tmigot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks ! I made a couple of comments.


Get the evaluation counters from CUTEst's native reporting functions.
"""
function get_cutest_counters(nlp::CUTEstModel{T}) where T
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it is doable to test all the fields of Counters https://github.com/JuliaSmoothOptimizers/NLPModels.jl/blob/d258d3550db5ebfb828a1edc049d1185a91587b7/src/nlp/counters.jl#L14 ?
If not it should be described here which fields you didn't manage to do, and we will create another issue after this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was able to test 17 out of the 20 counter fields, started an issue for the missing 3 at #465

Copy link

codecov bot commented Jul 14, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.95%. Comparing base (f2ad6a4) to head (00ef295).
Report is 157 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #463       +/-   ##
===========================================
- Coverage   89.11%   73.95%   -15.16%     
===========================================
  Files           5        7        +2     
  Lines         790     1678      +888     
===========================================
+ Hits          704     1241      +537     
- Misses         86      437      +351     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arnavk23 arnavk23 requested a review from tmigot July 15, 2025 03:01
@arnavk23
Copy link
Contributor Author

@tmigot If you can, Please also review this pr as I think this is very close to merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check eval counters against those from CUTEst
2 participants