Speed up CI to complete within 15min (down from 25min) by switching off coverage except for the matrix entry that's uploaded

[corneliusroemer]

Description of proposed changes

Speeds up CI to fully complete within 15min instead of 25, most tests complete within 11min inntead of 15min. How?

• Installing numpy=1 with mamba, to prevent downgrading with pip install which reduces performance
• Only doing coverage collection for a single matrix entry, the one we upload
• Setting COVERAGE_CORE=sysmon to speed up coverage calculation (available for Python 3.123.14+), see When we switch to Python 3.14, collect coverage with COVERAGE_CORE=sysmon #1862

This brings down CI time for Python 3.12 (now run without coverage collection) from 27min to 11min, and for Python <3.12 from 15 to 11min.

Collecting coverage costs ~4min: the matrix entry with coverage takes ~15min vs 11min without.

It also changes the default of ./run_tests.sh to not collect coverage as one rarely is interested in coverage locally and the cost is significant. One can simply add --cov arg to add coverage: ./run_tests.sh --cov

Energy savings

For fun I calculated saved energy costs (for Github) of this change:

• Monthly minutes roughly 24k -> 400h
• This PR reduces by ~25% (from 4h runner time to 3h)
• 2 vCPU runner ~2W
• Savings is 200 Wh/month
• So 2 kWh per year ~ 60ct

So not worth it for the savings, but for the developer experience.

Related issue(s)

Resolves #1681

Checklist

• Automated checks pass
• Check if you need to add a changelog message
• Check if you need to add tests
• Check if you need to update docs

[victorlin]

Nice find, thanks for looking into this!

[victorlin]

I assume you will squash the commits here as well?

[corneliusroemer]

I should rebase this and resolve merge conflicts

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.37%. Comparing base (9189a4a) to head (81ed5bc).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1861   +/-   ##
=======================================
  Coverage   73.36%   73.37%           
=======================================
  Files          80       80           
  Lines        8752     8754    +2     
  Branches     1784     1784           
=======================================
+ Hits         6421     6423    +2     
  Misses       2026     2026           
  Partials      305      305           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joverlee521]

🚀

[victorlin]

@corneliusroemer do you plan to revisit this any time soon? I don't mind picking things up to get it merged.