Add Auto Review proof metrics and dogfood diagnostics

[shiny-code-bot]

Summary

Add proof instrumentation so the durable Auto Review concept can be evaluated with data, not vibes.

Scope

• Emit structured counters/events for run lifecycle, duplicate reuse/skips, supersede/cancel reasons, findings surfaced/inspected/applied/dismissed, ledger tokens, detail tokens, and token-spend estimates.
• Add a diagnostic surface such as /review-stats if useful.
• Build deterministic scanners or fixtures for stale/superseded/duplicate/fix-train signatures from logs/rollouts where appropriate.

Acceptance Criteria

• Metrics include duplicate review rate, skipped/adopted/superseded/cancelled counts, unsurfaced terminal findings, ledger overhead, avoided token estimate, time to surface findings, and finding usefulness/disposition.
• Each Auto Review run records enough proof data to explain latency: model, reasoning effort, resolve model/effort, phase timing, follow-up count, token count when available, prompt token estimate, and terminal reason.
• Restart recovery and duplicate avoidance are testable without a live TUI where possible.
• Dogfood diagnostics can compare before/after behavior across real sessions and identify whether slowness came from first review pass, follow-up loops, worktree/lock contention, retries, or prompt bloat.
• Metrics do not inject bulky telemetry into normal assistant context; ordinary turns receive only bounded actionable review state.

Relationships

Parent: #324
Depends on: #325, #327, #329
Related: #43, #50

Finish Line

Every Code emits enough Auto Review metrics and diagnostics to prove duplicate review reduction, avoided token spend, surfaced findings, ledger overhead, restart recovery, and finding usefulness during dogfooding.

Current Status

State: Scoped proof-metrics implementation merged.
Merged PR: #381 feat(auto-review): add proof metrics to compact ledger
Merge commit: 5bb9fbf704aa2968bac1e44f328e8f4b3d0c458c
Branch: fix/auto-review-proof-metrics (remote branch deleted after merge).
Next action: #331 Auto Review lifecycle docs can proceed against the merged durable Auto Review behavior. Broader prompt/context/token-budget/request-shape accounting remains gated by #92 and should not be started while the token-count refactor is active.
Blocked by: None for #331 docs. Broader prompt/context/token metrics remain blocked by #92.
Last verified: 2026-06-05 after focused tests, required build, PR CI, Claude review, and merge.

Completed in #381:

• Compact Auto Review diagnostics now count duplicate-skipped runs, skipped runs without saved tokens, superseded clean duplicates, failed/cancelled/lost terminal proof outcomes, saved token estimates, and existing prompt/token/timing signals.
• Duplicate/superseded/cancelled/lost proof stays in compact diagnostics without surfacing bulky run details.
• Focused review-store tests cover duplicate proof, superseded proof, terminal outcome counts, old proof-run omission, and active-run plus dedupe proof combinations.

Review and validation:

• Claude Sonnet final review found no correctness bugs and marked the PR green for merge.
• cargo test -p code-core compact_ledger --lib
• ./build-fast.sh
• PR feat(auto-review): add proof metrics to compact ledger #381 GitHub check passed before merge.

Residual scope:

• This PR does not implement broader prompt/context budget enforcement or request-shape accounting; those remain behind Add context source ledger and prompt observability #92.
• Future polish noted by review: add symmetric tests for clean Failed/Lost omission and Cancelled/Lost with error detail if those paths are touched again.

[cbusillo]

Dogfood benchmark evidence from 2026-06-02 while waiting on Auto Review snapshot 9cc60a611f899c031ab1502153ca2184c54f55c5 / worktree auto-review-9cc60a61:\n\n- Review launched under the pre-restart harness but using PATH code exec at code 0.6.113.\n- Command line confirmed review model gpt-5.4-mini with model_reasoning_effort=medium and auto_drive.model_reasoning_effort=medium.\n- Scope was limited to 4 files: code-rs/cli/src/llm.rs, code-rs/core/src/agent_tool.rs, code-rs/core/src/review_store.rs, and code-rs/tui/src/chatwidget.rs.\n- Rollout token events reached about 609k total tokens by 2026-06-02T16:47:48Z, with no review-output JSON visible yet at that checkpoint.\n- The process remained alive after about 7 minutes but had no newer token/progress events in the session tail.\n\nImplication for this plan: the first review pass is still the likely slow/burny phase even with mini+medium and reduced follow-ups. Diagnostics should explicitly record prompt byte/token estimate, phase timing, last token/progress timestamp, actual token usage when stream events expose it, and whether the supervising harness was old vs restarted/new behavior. This should become a before/after benchmark for the love gate.

[cbusillo]

Suggestion from the v0.6.115 dogfood/release closeout, not a firm design decision:

The worktree decision gate and hardening make Auto Review safer to leave on, but the next proof layer should make token burn and review usefulness visible enough that we can answer "is it fixed?" without reading logs by hand.

Specific diagnostics that would have helped during this release train:

• A compact per-run cost line: prompt estimate, provider token count if available, wall time, model, reasoning effort, resolve model/effort, follow-up count, and terminal reason.
• A phase breakdown so we can tell initial review cost from resolve/follow-up loops and from process/retry/worktree contention.
• A session-level summary of recent Auto Review/Auto Resolve spend, including skipped/adopted duplicate runs and avoided-token estimates.
• A way to flag suspicious accounting, e.g. tiny prompt/short wall-clock but unexpectedly high provider token count, without treating it as a hard failure.
• Dogfood comparison output before/after releases so we can verify that enabling Auto Review does not reintroduce high-burn behavior after harness restarts.