Skip to content

feat: pluggable execution-plan scheduling and profile-guided parallel-rt placement - #1

Open
Scofield626 wants to merge 7 commits into
masterfrom
pluggable-plan-policy
Open

feat: pluggable execution-plan scheduling and profile-guided parallel-rt placement#1
Scofield626 wants to merge 7 commits into
masterfrom
pluggable-plan-policy

Conversation

@Scofield626

@Scofield626 Scofield626 commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Resolves the TODO(gbin): Make that heuristic pluggable in compute_runtime_plan(), then uses the same measured profile to place parallel-rt stage workers.

What

  • Split the planner into two phases: a policy picks the step order (Vec<NodeId>), a shared build phase assigns copperlist slots, wires inputs, and validates. Any topological order is a correct plan, so a policy can only pick a better or worse order — it cannot corrupt wiring. The build phase rejects non-topological orders from a buggy policy.
  • Algorithm and measurement are separate config fields. A variant names an algorithm and nothing else; the numbers it reads live next to it:
    runtime: (
        plan_policy: CriticalPathFirst,                       // ordering algorithm
        plan_profile: (task_duration_ns: {"cam": 1200, ...}), // measurement
        core_placement: LongestFirst,                         // placement algorithm
    )
    Folding the profile into CriticalPathFirst(task_duration_ns: ...) would name the input where a variant should name the algorithm, and would force every future profile-guided algorithm to repeat the field. Kept apart, adding LongestTaskFirst or MinSlack costs one unit variant, switching algorithm needs no re-measurement, and placement reads the same profile without going through a policy at all.
  • Ordering policies. TopoBfs (default) — the historical source-BFS order, byte-identical output, pinned by a golden test. CriticalPathFirst — critical-path-first list scheduling over the profile. Deterministic: ties break on node id; unmeasured tasks weigh zero.
  • schedule-profile exporter: cu29_export <log> schedule-profile [--stat mean|p99|max] reads the process_time window each message already records and writes the pasteable plan_profile: RON value. No new instrumentation. It writes a profile, never a policy — picking the algorithm stays the user's decision.
  • Bottleneck report. cu29_export <log> log-stats gains a pipeline section: per-plan-step duration stats, serial_cycle_ns (the serial engine's cycle), bottleneck (the slowest step, i.e. the parallel-rt cycle), and max_pipeline_speedup = serial / bottleneck. The CLI prints one line. Schema version 1 → 2.
  • Profile-driven core placement. runtime.core_placement selects how parallel-rt stage workers map onto the rt pool's affinity list: Spread (default, stage % cores, unchanged) or LongestFirst, an LPT bin-pack over the profile. Resolved at compile time — both inputs are in the config — so the generated worker passes an already-chosen affinity slot and the hot path is untouched.

A profile-guided policy or placement with an empty plan_profile fails the build with a message pointing at schedule-profile, rather than silently falling back to another order.

The profile lives inline in the config on purpose: the unified log embeds the config, so offline tools (logstats) recompute the exact same plan. Design notes and the PGO workflow are in sched-v0.md.

Why placement, and not just another policy

parallel-rt is a stage-affine pipeline: one worker per plan step, each pinned to cores[stage_index % cores.len()].

  • Pipeline throughput is the duration of the slowest single step, not the sum. Reordering cannot change a maximum, so CriticalPathFirst buys parallel-rt essentially nothing. Ordering is a latency tool; parallel-rt needs a balance tool.
  • Worker count equals step count, independent of core count. A 20-step graph on 4 cores spawns 20 threads.

Hence the profile feeds three separate consumers, cheapest first: report the ceiling (done), pack cores by load (done), fuse cheap adjacent stages (still open — it breaks the stage-index = plan-index identity, so it is a larger change).

Workflow

build (TopoBfs) → run robot or resim → cu29_export mylog.copper log-stats   # is it worth it?
      → cu29_export mylog.copper schedule-profile
      → paste as runtime.plan_profile, pick plan_policy / core_placement
      → rebuild → compare logstats

Verification

  • Golden test pins the default plan order and slot assignment; derive snapshot/compile-pass tests unchanged.
  • Ordering: critical-path priority vs default order, deterministic tie-break, partial-profile handling, empty-profile rejection.
  • Placement: Spread reproduces the historical round-robin; LongestFirst balances load; all-equal weights degenerate back to Spread; unmeasured steps weigh zero; empty profile and zero slots are rejected. Two derive-level tests cover reading the rt pool and skipping when there is no affinity.
  • Pipeline stats: bottleneck selection, an unsampled step leaving the cycle unknown, and the no-sample case.
  • Verified end-to-end on real recordings: cu_caterpillar for the bottleneck report (212k samples/step, correctly fingers the 20 µs GPIO write), and cu-runtime-matrix under parallel-rt with a pinned rt pool for the placement.
  • cu_caterpillar determinism regression passes; just clippy-nostd passes; fmt/clippy clean.
  • Note: the pipeline section reports only exact stats (min/max/mean/stddev). CuDurationStatistics is 1024 linear buckets over its configured max, so its percentile() is meaningless at microsecond scale — an exact percentile comes from schedule-profile --stat p99, which keeps the raw samples.
  • Side fix: boxed RenderInput::Single (clippy large_enum_variant after CuConfig grew).

@github-actions

Copy link
Copy Markdown

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, please read our contributor guide:
https://github.com/copper-project/copper-rs/blob/master/CONTRIBUTING.md

PlanPolicy variants now name only the ordering algorithm; measured task
durations move to a separate runtime.plan_profile field. A profile-guided
policy with an empty profile fails the build instead of silently reordering.
logstats gains a pipeline section: per-plan-step duration stats, the serial
cycle, the slowest step, and the speedup a pipelining engine could reach.
The CLI prints the bottleneck line. Schema version goes to 2.
runtime.core_placement picks how stage workers map onto the rt pool's
affinity list: Spread (default, unchanged) or LongestFirst, an LPT bin-pack
over plan_profile. Resolved at compile time; an all-zero profile degenerates
back to Spread.
@Scofield626 Scofield626 changed the title feat: pluggable execution-plan scheduling with profile-guided ordering feat: pluggable execution-plan scheduling and profile-guided parallel-rt placement Jul 29, 2026
- share LOGSTATS_SCHEMA_VERSION between the exporter and rendercfg so the
  bump to 2 stops warning on every render
- fail the build when a non-default core_placement has no rt affinity list
  instead of silently keeping the spread
- restore the qualified cu29::config::RT_POOL path in generated code
- keep one placement slot per plan step whatever the placement
- warn when the --config file no longer plans the way the log was recorded
- warn on an empty exported profile; use nearest-rank for --stat p99
- correct the StageStats::index and bottleneck docs; move sched-v0.md to doc/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant