Skip to content

Fix/benchmark bugs#733

Merged
Dreamsorcerer merged 1 commit intoaio-libs:masterfrom
rodrigobnogueira:fix/benchmark-bugs
Jan 21, 2026
Merged

Fix/benchmark bugs#733
Dreamsorcerer merged 1 commit intoaio-libs:masterfrom
rodrigobnogueira:fix/benchmark-bugs

Conversation

@rodrigobnogueira
Copy link
Copy Markdown
Member

What do these changes do?

Fixed two logic bugs in benchmark.py that prevented accurate performance measurements:

  1. Cache Miss Benchmark: test_cache_miss_benchmark was previously improperly configured with only 128 distinct objects (matching the cache size), resulting in cache hits instead of misses. Increased the sample size to 2048 objects to ensure evictions and actual misses are measured.
  2. Concurrent Hit Benchmark: test_concurrent_cache_hit_benchmark contained a bug where the loop returned after the first iteration. Removed the premature return to ensure the benchmark runs for the intended 10 iterations.

Are there changes in behavior for the user?

No. These changes only affect the internal benchmark suite used for performance verification.

Related issue number

N/A

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes

@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.37%. Comparing base (4232c3f) to head (ba2c2a2).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #733   +/-   ##
=======================================
  Coverage   97.37%   97.37%           
=======================================
  Files          15       15           
  Lines         991      991           
  Branches       56       56           
=======================================
  Hits          965      965           
  Misses         23       23           
  Partials        3        3           
Flag Coverage Δ
unit 96.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Jan 21, 2026

Merging this PR will degrade performance by 99.22%

❌ 16 regressed benchmarks
✅ 47 untouched benchmarks
⏩ 4 skipped benchmarks1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
test_concurrent_cache_hit_benchmark[meth-bounded-ttl] 9.1 ms 85.9 ms -89.41%
test_cache_miss_benchmark[func-unbounded-ttl] 11.2 ms 173.4 ms -93.54%
test_cache_miss_benchmark[func-unbounded] 1.1 ms 144.2 ms -99.21%
test_cache_miss_benchmark[func-bounded-ttl] 11 ms 173.3 ms -93.65%
test_concurrent_cache_hit_benchmark[meth-unbounded-ttl] 10.9 ms 98.2 ms -88.87%
test_concurrent_cache_hit_benchmark[meth-bounded] 3.3 ms 30.7 ms -89.33%
test_cache_miss_benchmark[meth-bounded-ttl] 11.4 ms 179.1 ms -93.65%
test_concurrent_cache_hit_benchmark[meth-unbounded] 3.3 ms 30.7 ms -89.3%
test_cache_miss_benchmark[meth-bounded] 1.5 ms 150.1 ms -99.02%
test_cache_miss_benchmark[func-bounded] 1.1 ms 144.3 ms -99.22%
test_cache_miss_benchmark[meth-unbounded-ttl] 11.6 ms 179.2 ms -93.54%
test_cache_miss_benchmark[meth-unbounded] 1.5 ms 150.2 ms -99.02%
test_concurrent_cache_hit_benchmark[func-bounded-ttl] 8.5 ms 79.8 ms -89.41%
test_concurrent_cache_hit_benchmark[func-bounded] 2.8 ms 25.5 ms -89.2%
test_concurrent_cache_hit_benchmark[func-unbounded-ttl] 10.3 ms 91.9 ms -88.75%
test_concurrent_cache_hit_benchmark[func-unbounded] 2.7 ms 25.5 ms -89.25%

Comparing rodrigobnogueira:fix/benchmark-bugs (ba2c2a2) with master (4232c3f)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@rodrigobnogueira
Copy link
Copy Markdown
Member Author

there are some opportunities of refactoring in the benchmark code. But I've kept the changes to the minimum necessary to fix the issues.

@Dreamsorcerer Dreamsorcerer merged commit c0f36c4 into aio-libs:master Jan 21, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants