Add CodSpeed continuous performance benchmarks#54
Conversation
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7b6399cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut frame = PyFrame::new_with_context(code, execution_context) | ||
| .expect("benchmark frame creation must succeed"); | ||
|
|
||
| let _ = eval_with_jit(&mut frame); |
There was a problem hiding this comment.
Propagate benchmark execution failures
Handle the eval_with_jit result instead of discarding it: if a benchmarked script starts raising (for example after a regression in bytecode execution or JIT tracing), this line treats the run as successful and CodSpeed will time a fast failing path, producing misleading performance data rather than surfacing a broken workload. Benchmark runs should fail immediately on Err (e.g., with expect) so regressions do not silently corrupt measurements.
Useful? React with 👍 / 👎.
Summary
This PR integrates CodSpeed for continuous performance monitoring of the pyre interpreter and JIT pipeline. Every push to
mainand every pull request now runs reproducible benchmarks and reports performance changes automatically.Changes
New benchmark crate:
pyre-benchA new
pyre/pyre-benchcrate usescodspeed-criterion-compatto benchmark 6 Python workloads through the full interpreter + JIT pipeline:Each benchmark compiles a Python script, creates an interpreter frame, and evaluates it through
eval_with_jit, mirroring the real execution path. Iteration counts are scaled down from the existingpyre/bench/scripts to keep CI times reasonable while still exercising the same code paths.CI workflow:
.github/workflows/codspeed.ymlmain, pull requests, andworkflow_dispatch(for CodSpeed backtesting)majit/andpyre/path changes to avoid unnecessary runsREADME badge
Added a CodSpeed badge to the project README alongside the existing Discord badge.
Note on runners
This repository belongs to a personal GitHub account, so
codspeed-macrorunners are not available. The workflow usesubuntu-latestrunners instead. This means performance variance will be higher than with dedicated bare-metal runners. If this repository is moved to a GitHub organization in the future, switching to macro runners is recommended for more stable measurements. See CodSpeed walltime docs for details.Next steps
main