perf(qwen35): add opt-in F32 rollback checkpoints for single-GPU inference

[Amidwestnoob]

Summary

Adds an experimental, default-off F32 checkpoint mode for single-chain speculative rollback on memory-constrained single-GPU Qwen deployments.

The existing behavior remains unchanged unless explicitly enabled:

DFLASH_SINGLE_CHAIN_CHECKPOINT_F32=1
DFLASH_FAST_ROLLBACK_THRESHOLD=1   # valid range: 1-5

Optional diagnostics:

DFLASH_SINGLE_CHAIN_ROLLBACK_DIAG=1

Without the F32 opt-in, checkpoints remain F16 and the rollback threshold remains 5.

Motivation

On a single RTX 3090 running Qwen3.6-27B Q4, replaying rejected speculative tokens can cost more than restoring a saved recurrent-state checkpoint. F32 checkpoints permit direct restoration and avoid the conversion/replay path for sufficiently deep rollbacks.

This is intentionally separate from the split/layer-sharded work in #506. It targets a different deployment: one 24 GiB GPU, with no layer split.

Implementation

• Centralizes checkpoint dtype, threshold, and diagnostic policy resolution.
• Centralizes rollback counters and their output format across both speculative-decode loops.
• Allocates F32 single-chain rollback checkpoints only when opted in.
• Uses direct F32 checkpoint copy/restore when available.
• Preserves the existing F16 conversion path by default.
• Adds unit coverage for defaults, opt-in behavior, threshold validation, diagnostics, and the exact diagnostic output contract.
• Leaves split/multi-GPU checkpoint allocation unchanged.

Validation

Final standalone branch, based on current main:

• CUDA release build: passed
• check: 15/15 tests passed
• Single RTX 3090 context stress: 8K through 65K active tokens
• OOMs: 0
• Segfaults: 0
• Second GPU usage: 15 MiB (idle)

Fresh A/B/A validation on the standalone build, six prompt families and three repetitions per arm:

• F16 threshold 5, first baseline: 53.299 tok/s
• F32 threshold 1: 56.764 tok/s
• F16 threshold 5, repeated baseline: 53.338 tok/s
• F32 threshold 1 versus bracketed baseline: +6.46% weighted decode throughput

Independent final verification rebuilt and tested the exact diagnostics-centralized candidate, then repeated the A/B/A measurement:

• F16 threshold 5, first baseline: 53.348 tok/s
• F32 threshold 1: 56.057 tok/s
• F16 threshold 5, repeated baseline: 51.083 tok/s
• Observed bracket: +5.08% to +9.74% (interpolated midpoint: +7.36%)
• Every F32 repetition outperformed every F16 repetition; the repeated F16 arm showed 4.25% thermal/clock drift while retaining byte-identical outputs and rollback counters.
• Fresh exact-tree verification: CUDA Release build passed, 15/15 tests passed, focused policy test passed, and the single-GPU smoke test kept GPU 1 at its 15 MiB idle baseline.

The broader development campaign previously found threshold 2 to be the best pooled policy, but this PR deliberately uses the 6.46% threshold-1 result from the drift-free standalone candidate as its directly reproducible point estimate. The independent rerun supports that figure while showing the honest run-to-run range above.

Limitations

• Experimental and default-off.
• F32 checkpoints require approximately 1.15 GiB more VRAM than F16 checkpoints in the tested configuration.
• Generated text can differ from the legacy replay trajectory; this is not a claim of bit-identical output.
• Performance evidence is specific to Qwen3.6-27B Q4 on an RTX 3090 and should not be generalized to other models or GPUs without measurement.
• Some workloads may see smaller gains or no gain.
• Split/multi-GPU behavior is out of scope and remains unchanged.

Suggested test configuration

cmake -S server -B server/build -DDFLASH27B_TESTS=ON
cmake --build server/build --target check

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[davide221]

@dusterbloom can you review this

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/common/chain_rollback_policy.h">

<violation number="1" location="server/src/common/chain_rollback_policy.h:61">
P3: `RollbackDiag::print` passes `out` directly to `std::fprintf` after only checking `policy.diagnostics`. If someone ever calls it with a null FILE* while diagnostics is enabled — say, after a failed `tmpfile()` or `fopen` — the result is UB. Since `diagnostics` already has a guard, adding `if (!out) return;` would be consistent and prevent that crash class.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>