bench_kit

A small, dev-only benchmarking & regression toolkit for Rust crates. It has zero runtime dependencies and is meant for [dev-dependencies] only, so it never enters your crate's real dependency graph.

What it gives you:

1. a scaling gate that fails cargo bench when an operation's complexity class regresses (e.g. an accidental O(n²)), measured in a machine-independent way (a log-log slope, immune to CPU speed / CI noise / debug-vs-release);
2. deterministic metrics (Counter + Metrics) you snapshot to catch changes in how much work a computation does (iteration/eval/step counts);
3. an allocation counter (CountingAllocator) — a deterministic perf signal (an accidental clone/box shows up as an allocation-count regression);
4. supporting helpers — a deterministic Rng (with shuffle/permutation), amortizing timers, and an A/B comparison reporter (report_comparison).

Capability index (jump to the section you need):

You want to…	Use	Section
Fail the build if an op's complexity class regresses	assert_scaling	Pillar 1
Time a fast (µs) op stably	time_amortized_ms	Pillar 1 → timing
Compare two implementations (naive vs optimized)	report_comparison	Pillar 1 → A/B
Snapshot deterministic work counts (iterations, evals)	Counter + Metrics	Pillar 2
Catch an accidental allocation regression	CountingAllocator	Pillar 2 → allocations
Generate reproducible random inputs / orders	Rng (shuffle, permutation)	API reference

Install

[dev-dependencies]
bench_kit = { git = "https://github.com/codepaws/crate_bench_kit.git" }

Pin a specific commit or tag for reproducibility:

bench_kit = { git = "https://github.com/codepaws/crate_bench_kit.git", tag = "v0.1.0" }
# or: rev = "<commit-sha>"

Step 1 — decide which pillar your crate needs

Pick by what actually regresses in your crate:

Your crate is…	Examples	The risk	Use
Time / scaling-based — cost grows with input size; you make complexity claims	graph algorithms, geometry, tessellation, gfx	an op silently slips to a worse complexity class	Pillar 1: scaling gate
Operation / count-based — deterministic numeric work; cost is "how many iterations/evals"	solvers, integrators, linear algebra	iteration/eval count drifts on a refactor	Pillar 2: metrics snapshot

Many crates want both (gate the wall-clock-shaped ops, snapshot the deterministic counts). A purely deterministic-count crate needs no timing at all. If unsure: if your story is "this is O(n log n)," use Pillar 1; if it's "this converges in N iterations," use Pillar 2.

---

Pillar 1 — the scaling gate (time-based crates)

1a. Wire up the bench target

Add to your Cargo.toml:

[[bench]]
name = "perf_gate"     # -> benches/perf_gate.rs
harness = false        # REQUIRED: it's a plain `fn main()`, not libtest/criterion

harness = false means the bench is an ordinary fn main() — no criterion, no nightly. cargo bench compiles it in release and runs it; a panic fails the run.

1b. Write benches/perf_gate.rs

use bench_kit::{assert_scaling, time_amortized_ms, black_box, Rng};
use my_crate::build_index;

fn main() {
    // assert_scaling(label, sizes, max_exponent, measure):
    //   measure(size) -> milliseconds for that size.
    assert_scaling("build_index", &[2_000, 4_000, 8_000, 16_000], 1.4, |n| {
        // Build the input OUTSIDE the timed closure (setup must not be measured):
        let input = make_input(&mut Rng::new(0xABCD ^ n as u64), n);
        // Time ONLY the hot op. black_box stops the optimizer eliding it:
        time_amortized_ms(|| { black_box(build_index(&input)); })
    });

    println!("perf_gate: OK");
}

Run it: cargo bench (or cargo bench --bench perf_gate). On success it prints a size/time/slope table per op and exits 0; on a regression it panics with the measured points and exponent, failing the build.

1c. How the gate works (so you can reason about failures)

It measures the op at each size, fits the empirical exponent b from a least-squares line through (ln size, ln time), and asserts b ≤ max_exponent. Because a log-log slope ignores the constant factor, the result is the same on a fast laptop and a slow CI box, in debug or release — it measures complexity class, not raw speed. (That's why an absolute-time threshold would be useless here and we don't use one.)

1d. Choosing max_exponent (the ceiling)

Set it to your expected complexity plus headroom for measurement noise:

Expected complexity	Suggested ceiling
O(n) linear	1.3
O(n log n)	1.4
O(n^1.5)	1.7
O(n²) (and you want to catch worse, e.g. cubic)	2.3

You are policing the class, not constants — pick the smallest ceiling your op should never exceed.

1e. Choosing input sizes

• Use ≥4 sizes spanning a ≥8× range (e.g. 2k, 4k, 8k, 16k). More points / wider range = more stable slope.
• Each size should produce a measurable op. With time_amortized_ms (below) even microsecond ops are fine, so you can keep sizes small and the bench fast.
• Vary inputs deterministically with Rng::new(seed) so a failure means a code change, not input luck.

1f. Timing helpers — pick the right one

Helper	Use when
time_amortized_ms(|| op)	Default. Fast ops (microseconds) and slow ones. Auto-calibrates a batch size so each timed interval clears the ~20 ns Instant::now() overhead + jitter, then takes the min per-op time. For ops already ≥1 ms it settles on a batch of 1 (≡ min-of-N), so it's always safe.
time_min_ms(runs, || op)	Op already takes ≥~1 ms and you want an explicit run count. Returns the min of runs runs.
time_amortized_ms_cfg(op, min_batch, samples)	You need to tune the calibration target or sample count.

This is not a timer-precision issue: Instant already uses QueryPerformanceCounter (Windows) / clock_gettime(CLOCK_MONOTONIC) (Linux). The floor on timing a microsecond op is call overhead + jitter, which amortization removes. Run cargo run --release --example timer_probe to see your machine's floor.

1g. Comparing two implementations (report_comparison)

When you keep a "simple-correct" path alongside an optimized one (or are weighing a rewrite), measure each and print a ratio table:

use bench_kit::{report_comparison, time_amortized_ms, black_box};

let naive  = time_amortized_ms(|| { black_box(my_crate::build_naive(&input)); });
let tuned  = time_amortized_ms(|| { black_box(my_crate::build(&input)); });
report_comparison("build: naive vs tuned", &[("naive", naive), ("tuned", tuned)]);
// prints each variant's time and its ×-vs-fastest.

This is reporting only (it never fails the build) — use it to inform decisions, and the scaling gate to guard them.

1h. Common pitfalls

• Build inputs outside the timed closure. Timing setup pollutes the slope.
• Always black_box the result (and sometimes the input) so the optimizer can't delete the work and report ~0 ms.
• A gate failure ≠ "the gate is wrong." First assume a real regression. Only if the algorithm is genuinely unchanged should you suspect a too-tight ceiling, too-few/too-close sizes, or an input that doesn't actually scale.
• Make the input actually exercise the worst case. A slope near 1.0 on an op you expect to be superlinear usually means your input is too easy (it never triggers the expensive path), not that the op is fast.

---

Pillar 2 — deterministic metrics snapshot (count-based crates)

Instrument the hot path with counters, collect them into a stable text snapshot, and pin it. The counts are deterministic for deterministic inputs, so an exact-match snapshot is the right tool: a margin/threshold would silently hide small drift, whereas a snapshot forces a conscious review of every change.

2a. Instrument and snapshot

use bench_kit::{Counter, Metrics};

// Declare counters as statics (cheap atomic; usable in const context).
static RESIDUAL_EVALS: Counter = Counter::new();
static ITERATIONS: Counter = Counter::new();

// ... in the hot path of your library code's solver, call:
//        RESIDUAL_EVALS.bump();      // +1
//        ITERATIONS.add(k);          // +k
// (Gate this behind a cfg/feature if you don't want counters in release.)

#[test]
fn solver_work_is_stable() {
    RESIDUAL_EVALS.reset();           // reset before each measured run
    ITERATIONS.reset();

    let _ = my_crate::solve(&fixture());

    let snap = Metrics::new()
        .count("residual_evals", RESIDUAL_EVALS.get())
        .count("iterations", ITERATIONS.get())
        .snapshot();                  // stable, name-sorted, one "name = value" per line

    // Option A — review-diff workflow (recommended): add `insta` to your own
    // [dev-dependencies], then:
    //     insta::assert_snapshot!(snap);
    //     // first run writes the baseline; `cargo insta accept` updates it on
    //     // intended changes; any drift shows a reviewable diff.
    //
    // Option B — no extra deps: pin the expected value directly.
    assert_eq!(snap, "iterations = 7\nresidual_evals = 42");
}

Metrics::snapshot sorts entries by name, so insertion order and HashMap nondeterminism never cause spurious diffs. bench_kit itself takes no insta dependency — choose insta (nicer review workflow) or a pinned assert_eq!.

2b. Allocation counting (CountingAllocator)

Allocation count is a deterministic perf signal — an accidental clone, boxing, or a Vec that should have been reused shows up as a count regression with no wall-clock noise. Opt your bench/test binary into the counting allocator, then snapshot the counts like any other metric.

use bench_kit::{CountingAllocator, Metrics};

#[global_allocator]                       // one per binary
static ALLOC: CountingAllocator = CountingAllocator::new();

#[test]
fn allocations_are_stable() {
    ALLOC.reset();                        // counts the WHOLE process, so reset
    let _out = my_crate::build(&fixture()); //   immediately before the op
    let snap = Metrics::new()
        .count("allocations", ALLOC.allocations())
        .count("bytes", ALLOC.bytes())
        .snapshot();
    insta::assert_snapshot!(snap);        // or assert_eq! against a pinned value
}

Because it counts process-wide, keep the measured work between reset() and the .allocations()/.bytes() reads, and avoid asserting exact counts across parallel tests sharing the binary (give it its own test file — see tests/alloc.rs).

2c. Pitfalls

• reset() before every measured run — counters/allocator are process-global.
• Inputs must be deterministic (fixed fixtures / seeded Rng), or the counts won't be reproducible and the snapshot will be useless.
• Keep counters cheap (Counter is a relaxed atomic add); don't instrument so finely that you change behaviour.

---

API reference

Everything public, at a glance:

Deterministic RNG

• Rng::new(seed: u64) -> Rng — seedable SplitMix64; same seed ⇒ same stream everywhere.
• .next_u64() -> u64, .next_f64() -> f64 ([0,1)), .range_f64(lo, hi) -> f64, .range_usize(lo, hi) -> usize, .take_u64(n) -> Vec<u64>.
• .shuffle(&mut [T]) — in-place Fisher-Yates; .permutation(n) -> Vec<usize> — a shuffled 0..n. Both deterministic per seed (randomized insertion order, etc.).

Timing

• time_amortized_ms(op) -> f64 — auto-calibrating per-op ms (use this by default).
• time_amortized_ms_cfg(op, min_batch: Duration, samples: usize) -> f64.
• time_min_ms(runs, f) -> f64, time_min(runs, f) -> Duration.
• black_box(x) -> x — re-export of std::hint::black_box.

Scaling gate & comparison

• assert_scaling(label, sizes: &[usize], max_exponent: f64, measure: FnMut(usize)->f64) — measure + print + assert in one call (the typical bench body).
• measure_scaling(label, sizes, measure) -> ScalingReport — if you want the report.
• ScalingReport { label, points: Vec<(usize, f64)> } with .slope() -> f64, .print(), .assert_max_exponent(max: f64).
• report_comparison(label, named_ms: &[(&str, f64)]) — A/B(/C…) ratio table (report only).

Deterministic metrics

• Counter::new(), .bump(), .add(n: u64), .get() -> u64, .reset().
• Metrics::new(), .count(name, value) -> Metrics (chainable), .snapshot() -> String.

Allocation counting

• CountingAllocator::new() — wraps the system allocator; use as #[global_allocator]. CountingAllocator::with_inner(a) to wrap a custom one.
• .reset(), .allocations() -> u64, .bytes() -> u64 (process-wide; reset before the op).

A runnable demo of both pillars — also a copy-paste template:

cargo run --release --example demo

---

Running benchmarks

cargo bench                       # runs all [[bench]] targets (gates included)
cargo bench --bench perf_gate     # just the gate
cargo test                        # runs Pillar-2 snapshot/allocation tests (+ insta)

To sweep many crates in a workspace (no built-in runner), script it — e.g. for d in crates/*; do (cd "$d" && cargo bench) || exit 1; done.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.