Skip to content

Add CodSpeed continuous performance benchmarks#54

Open
codspeed-hq[bot] wants to merge 1 commit into
mainfrom
codspeed-wizard-1778829502038
Open

Add CodSpeed continuous performance benchmarks#54
codspeed-hq[bot] wants to merge 1 commit into
mainfrom
codspeed-wizard-1778829502038

Conversation

@codspeed-hq

@codspeed-hq codspeed-hq Bot commented May 15, 2026

Copy link
Copy Markdown

Summary

This PR integrates CodSpeed for continuous performance monitoring of the pyre interpreter and JIT pipeline. Every push to main and every pull request now runs reproducible benchmarks and reports performance changes automatically.

Changes

New benchmark crate: pyre-bench

A new pyre/pyre-bench crate uses codspeed-criterion-compat to benchmark 6 Python workloads through the full interpreter + JIT pipeline:

  • int_loop -- integer accumulation in a tight loop
  • fib_loop -- iterative Fibonacci with bigint arithmetic
  • fib_recursive -- recursive Fibonacci (function call overhead)
  • inline_helper -- nested helper functions exercising JIT inlining
  • nested_loop -- double-nested loop with multiplication
  • float_loop -- floating-point accumulation loop

Each benchmark compiles a Python script, creates an interpreter frame, and evaluates it through eval_with_jit, mirroring the real execution path. Iteration counts are scaled down from the existing pyre/bench/ scripts to keep CI times reasonable while still exercising the same code paths.

CI workflow: .github/workflows/codspeed.yml

  • Triggers on pushes to main, pull requests, and workflow_dispatch (for CodSpeed backtesting)
  • Uses the same action pinning style (commit hashes) as the existing CI workflows
  • Uses simulation mode for accurate, hardware-independent measurements
  • Scoped to majit/ and pyre/ path changes to avoid unnecessary runs

README badge

Added a CodSpeed badge to the project README alongside the existing Discord badge.

Note on runners

This repository belongs to a personal GitHub account, so codspeed-macro runners are not available. The workflow uses ubuntu-latest runners instead. This means performance variance will be higher than with dedicated bare-metal runners. If this repository is moved to a GitHub organization in the future, switching to macro runners is recommended for more stable measurements. See CodSpeed walltime docs for details.

Next steps

  1. Merge this PR and let CodSpeed generate baseline data on main
  2. Subsequent PRs will show performance diffs automatically
  3. Consider adding more benchmarks as new Python features are JIT-compiled (e.g., float-heavy paths, exception handling)

@codspeed-hq

codspeed-hq Bot commented May 15, 2026

Copy link
Copy Markdown
Author

Congrats! CodSpeed is installed 🎉

🆕 6 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@codspeed-hq codspeed-hq Bot marked this pull request as ready for review May 15, 2026 07:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af7b6399cc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

let mut frame = PyFrame::new_with_context(code, execution_context)
.expect("benchmark frame creation must succeed");

let _ = eval_with_jit(&mut frame);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate benchmark execution failures

Handle the eval_with_jit result instead of discarding it: if a benchmarked script starts raising (for example after a regression in bytecode execution or JIT tracing), this line treats the run as successful and CodSpeed will time a fast failing path, producing misleading performance data rather than surfacing a broken workload. Benchmark runs should fail immediately on Err (e.g., with expect) so regressions do not silently corrupt measurements.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants