Skip to content

docs(benchmarks): add v0 starting-point design spec#115

Open
lsetiawan wants to merge 2 commits into
uw-ssec:mainfrom
lsetiawan:docs/benchmarking-v0-spec
Open

docs(benchmarks): add v0 starting-point design spec#115
lsetiawan wants to merge 2 commits into
uw-ssec:mainfrom
lsetiawan:docs/benchmarking-v0-spec

Conversation

@lsetiawan

@lsetiawan lsetiawan commented Apr 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds a new design document scoping a minimal, trustworthy v0 for the HPyX benchmarking and profiling subsystem, before the Python API surface expands further.
  • Complements (does not replace) the existing benchmarking_profiling_sdd.md, which remains the long-term roadmap. This spec explicitly defers pyperf, asv, CI jobs, macro/memory suites, and numeric targets to that companion doc.

Changes

  • New: docs/design-docs/2026-04-24-benchmarking-v0-design.md
    • Section 5: 7-rule benchmark authoring contract (no timed setup, 3-order-of-magnitude size sweep, required NumPy + pure-Python + concurrent.futures baselines in the same group, explicit group names, minimize incidental Python overhead, thread-scaling parametrization, free-threading gating).
    • Section 6: shared benchmarks/conftest.py fixtures (pin_cpu, seed_rng, no_gc, hpx_runtime, hpx_threads, requires_free_threading, env_sanity_check) with an explicit resolution for the session-runtime vs. thread-count conflict.
    • Section 7: profile CMake preset (RelWithDebInfo + -fno-omit-frame-pointer, LTO off, default visibility) so py-spy --native, perf, and memray --native can resolve C++ frames.
    • Section 8: scripts/run_bench_local.sh subcommands and a one-page contributor benchmarks/README.md.
    • Section 9: planned updates to the companion SDD (directory rename bench/benchmarks/, elevate Python 3.13 free-threading to a first-class axis, prepend "Phase 0" to the rollout plan).
    • Section 10: acceptance criteria for v0.

Test plan

  • Reviewer confirms v0 scope is small enough to land in a single implementation plan.
  • Reviewer agrees with the session-scoped hpx_runtime fixture + dedicated file for thread-count-varying benchmarks (Section 6.1).
  • Reviewer agrees that concurrent.futures.ThreadPoolExecutor is the right third baseline alongside NumPy + pure Python.
  • Reviewer agrees with deferring pyperf/asv/CI/macro/memory/TARGETS.md to the existing SDD (Section 12).
  • No follow-up edits block moving to an implementation plan via the writing-plans workflow.

Scopes a minimal, trustworthy starting point for the benchmarking and
profiling subsystem: rewrites existing benchmarks against an authoring
contract, adds shared fixtures, a profile CMake preset, NumPy/pure-Python/
concurrent.futures baselines, and a free-threading smoke benchmark. Defers
pyperf, asv, CI integration, and macro/memory suites to the companion SDD
as the forward roadmap.
Adds the long-form design document referenced by the v0 starting-point
spec as its forward roadmap. Covers the full benchmarking subsystem
vision — pyperf, asv, CI integration, macro/memory suites, dedicated
runner, and numeric targets — scoped as Phase 1+ work to follow once
v0 lands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant