feat(cicd): codspeed benchmarks

[BobTheBuidler]

What do these changes do?

This PR adds codspeed benchmarks for various scenarios

Are there changes in behavior for the user?

N/A

Related issue number

N/A

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.22%. Comparing base (f60bf71) to head (20334fd).
⚠️ Report is 142 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #688      +/-   ##
==========================================
+ Coverage   96.88%   97.22%   +0.34%     
==========================================
  Files          12       12              
  Lines         770      793      +23     
  Branches       91       41      -50     
==========================================
+ Hits          746      771      +25     
+ Misses         22       20       -2     
  Partials        2        2              

Flag	Coverage Δ
unit	95.83% <ø> (-0.92%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[BobTheBuidler]

Update, I moved the benchmarks out of the test suite so they don't run with the regular tests. This should fix everything.

[BobTheBuidler]

Thanks for the review, I've committed your suggestion and replied to your other comment.

[BobTheBuidler]

Hmmm, what happened here?

[bdraco]

Looks like its getting closer

[BobTheBuidler]

woohoo, finally!

are you able to create a CODSPEED_TOKEN to add to the repo's secrets?

[bdraco]

I don’t have the necessary permissions to install it, so I’ll ask one of my colleagues to handle it.

[bdraco]

codspeed and token added. close/reopen to restart ci

[codspeed-hq]

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 39 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_cache_clear_benchmark[func-bounded] (21.3 µs)
• test_cache_clear_benchmark[func-unbounded] (21.4 µs)
• test_cache_clear_benchmark[meth-bounded] (22.3 µs)
• test_cache_clear_benchmark[meth-unbounded] (23.9 µs)
• test_cache_fill_eviction_benchmark (359.3 ms)
• test_cache_hit_benchmark[func-bounded] (4.9 ms)
• test_cache_hit_benchmark[func-unbounded] (4.9 ms)
• test_cache_hit_benchmark[meth-bounded] (10 ms)
• test_cache_hit_benchmark[meth-unbounded] (10 ms)
• test_cache_info_benchmark[func-bounded] (2 ms)
• test_cache_info_benchmark[func-unbounded] (2 ms)
• test_cache_info_benchmark[meth-bounded] (2.3 ms)
• test_cache_info_benchmark[meth-unbounded] (2.3 ms)
• test_cache_invalidate_benchmark[func-bounded] (375 µs)
• test_cache_invalidate_benchmark[func-unbounded] (373.9 µs)
• test_cache_invalidate_benchmark[meth-bounded] (1.1 ms)
• test_cache_invalidate_benchmark[meth-unbounded] (1.1 ms)
• test_cache_miss_benchmark[func-bounded] (1.1 ms)
• test_cache_miss_benchmark[func-unbounded] (1.1 ms)
• test_cache_miss_benchmark[meth-bounded] (1.4 ms)
• test_cache_miss_benchmark[meth-unbounded] (1.4 ms)
• test_cache_ttl_expiry_benchmark[func-bounded] (132.3 µs)
• test_cache_ttl_expiry_benchmark[func-unbounded] (132.3 µs)
• test_cache_ttl_expiry_benchmark[meth-bounded] (144.1 µs)
• test_cache_ttl_expiry_benchmark[meth-unbounded] (144.5 µs)
• test_concurrent_cache_hit_benchmark[func-bounded] (2.7 ms)
• test_concurrent_cache_hit_benchmark[func-unbounded] (2.7 ms)
• test_concurrent_cache_hit_benchmark[meth-bounded] (3.2 ms)
• test_concurrent_cache_hit_benchmark[meth-unbounded] (3.2 ms)
• test_internal_cache_hit_microbenchmark[func-bounded] (153.2 µs)
• test_internal_cache_hit_microbenchmark[func-unbounded] (153.2 µs)
• test_internal_cache_miss_microbenchmark[func-bounded] (109.7 µs)
• test_internal_cache_miss_microbenchmark[func-unbounded] (108.8 µs)
• test_internal_task_done_callback_microbenchmark[cancelled-func-bounded] (2.6 ms)
• test_internal_task_done_callback_microbenchmark[cancelled-func-unbounded] (2.6 ms)
• test_internal_task_done_callback_microbenchmark[exception-func-bounded] (964.3 ms)
• test_internal_task_done_callback_microbenchmark[exception-func-unbounded] (966.5 ms)
• test_internal_task_done_callback_microbenchmark[finished-func-bounded] (3 ms)
• test_internal_task_done_callback_microbenchmark[finished-func-unbounded] (3 ms)

[bdraco]

benchmarks are up. Probably need a few tweaks to make sure we are measuring more of the lib instead of asyncio

[BobTheBuidler]

I started to update the benchmarks by converting them into batched jobs, but after doing the first 3 I noticed there is a lot of stuff from what appears to be pytest-cov in the flamegraphs

I've just disabled pytest-cov for the benchmark runner, let's take a look at the outputs once the benchmarks run again and see if we still need to tweak them further

[BobTheBuidler]

Turns out they were from an unrelated coverage dep

I've now disabled both

[BobTheBuidler]

benchmarks are up. Probably need a few tweaks to make sure we are measuring more of the lib instead of asyncio

do you think the addition of batching and the internal sync microbenchmarks is suffcient to address your concern here? I can't think of any other strategies we might use to minimize the asyncio stuff in the flamegraphs because of how integrated everything is.

[bdraco]

I'm traveling until Wednesday night. I'll take a peek at the last turn when I get back home.