Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lower the time tests take to run #1567

Open
another-rex opened this issue Feb 4, 2025 · 2 comments · May be fixed by #1603
Open

Lower the time tests take to run #1567

another-rex opened this issue Feb 4, 2025 · 2 comments · May be fixed by #1603
Labels
performance

Comments

@another-rex
Copy link
Collaborator

Our go tests are starting to take quite a long time now, we should look into performance improvements or gating tests behind flags so they are not always ran. (though caching should already be doing this)

Here are the tests that are currently taking more than 2 seconds, sorted in ascending order:

TestRun_MavenTransitive/scans_transitive_dependencies_by_specifying_pom.xml,2.04
TestRun_MavenTransitive/scans_transitive_dependencies_for_pom.xml_by_default,2.35
TestRun/config_file_can_be_broad,2.38
TestRun_Licenses/No_vulnerabilities_with_license_summary_in_markdown,2.42
TestRun_MavenTransitive/scans_dependencies_from_multiple_registries,2.52
TestRun/PURL_SBOM_case_sensitivity_(local),2.52
TestRun_MavenTransitive/resolve_transitive_dependencies_with_native_data_source,2.6
TestRun_Licenses/Some_packages_with_ignored_licenses,2.7
Test_runGovulncheck,2.9
TestRun/requirements.txt_can_have_all_kinds_of_names,2.97
TestRun_LocalDatabases/only_the_files_in_the_given_directories_are_checked_by_default_(no_recursion),3.39
TestRun_LocalDatabases/.gitignored_files,3.91
TestRun_LocalDatabases/all_supported_lockfiles_in_the_directory_should_be_checked#01,3.97
TestRun_LocalDatabases/nested_directories_are_checked_when_`--recursive`_is_passed,4.53
TestRun_Fix/fix_non-interactive_json_override_pom.xml,4.54
TestRun_Fix/fix_non-interactive_override_pom.xml,4.67
TestPrintCycloneDX14Results_WithVulnerabilities,4.87
TestPrintCycloneDX15Results_WithLicenseViolations,4.95
TestPrintOutputResults_WithVulnerabilities,4.96
TestFromCSVFile,5.05
TestRun_LocalDatabases/ignoring_.gitignore,5.08
TestRunCallAnalysis/Run_with_govulncheck,5.42
TestPrintTableResults_NoTerminalWidth_WithVulnerabilities,6.05
TestCVE202230633,6.12
TestRun/folder_of_supported_sbom_with_vulns,7.27
TestRun_Fix/fix_non-interactive_relax_package.json,8.87
TestRun_LocalDatabases/all_supported_lockfiles_in_the_directory_should_be_checked,9.46
TestRun_Fix/fix_non-interactive_json_relax_package.json,9.61
TestRun_Fix/fix_non-interactive_json_in-place_package-lock.json,10.18
TestRun_Fix/fix_non-interactive_in-place_package-lock.json,11.06
TestRun_LocalDatabases/one_specific_supported_sbom_with_vulns,22.01
@G-Rath
Copy link
Collaborator

G-Rath commented Feb 5, 2025

(though caching should already be doing this)

Sadly caching doesn't tend to help because our top-most packages by definition depend on everything meaning 1. they're generally the slowest of tests as they're the most e2e-y, and 2. pretty much all changes to the codebase invalid their test cache

@G-Rath
Copy link
Collaborator

G-Rath commented Feb 6, 2025

So I knew since we introduced them that the offline/local database based tests are a large source of time because they have to download the all.zip files multiple times and they can be quite large - I've had an idea or two on how we could possibly improve this which I'm exploring now and it looks like it's working (taking us down to 10 seconds!).

There's some trade-offs to discuss and I need to actually cleanup the code to confirm everything properly which is why I'm not sharing more details just yet, but mentioning it now in case anyone else looks at this to avoid possible double work

This was referenced Feb 9, 2025
Open
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
2 participants
@G-Rath @another-rex