perf: zero-validation light witness decode; switch debug-trace-server onto it#154
Merged
Conversation
Port the R2 witness primitives (keys / sigv4 / endpoint / client) from mega-reth c290c3e into a dedicated leaf crate, replacing the git-rev dependency; mega-reth will delete its copy and consume this one via git tag, making the cross-repo dependency one-way. - add keys::block_object_key as the single primary-key template (object_keys delegates, the validator's R2 source calls it directly) - add golden wire-vector tests: a real migrated mainnet object key and a full SigV4 header set computed with an independent implementation - unify the workspace on reqwest 0.12 (0.13 was only used by debug-trace-server dev-deps); stateless-r2 keeps an explicit 0.12 pin since &reqwest::Client is part of its public API - r2_witness: carry Bytes end-to-end (no multi-MB copy per block), dedicated DecodePanicked variant, module-level retry consts, drop the unused deadline parameter and the per-fetch debug header log - docs: neutral witness-source wording; R2 is a first-class source Co-Authored-By: Claude Fable 5 <[email protected]>
- Record R2 witness fetch metrics: fetch+decode duration histogram, GET retry counter, surfaced-error counter labelled by kind, and the same witness-size breakdown the RPC path reports (pre-registered at startup like the RPC method counters). - Add an end-to-end happy-path test: a fixture witness encoded with encode_witness_payload served over the mock R2 server must decode back to the original tuple in exactly one GET (mock now serves byte bodies). - Warn at startup when --witness-endpoint is set but ignored because --witness-source r2 is active. - Fix rustdoc intra-doc links introduced by this branch. Co-Authored-By: Claude Fable 5 <[email protected]>
…fore exit Addresses the two open Codex review findings on this PR: - R2WitnessClient now enforces --witness-max-concurrent-requests with its own semaphore (permit scoped per GET attempt, like the RPC path's), so the documented knob for bounding witness I/O works in R2 mode instead of being silently ignored. Queue wait on the cap is excluded from the fetch-duration histogram so it cannot masquerade as R2 latency. Concurrency-probe test asserts the in-flight high-water mark. - run_with_signals sends one final validation report after the pipeline (and any signal drain) has fully stopped. The periodic reporter is cancelled the instant the pipeline completes, so an --end-block slice run could previously finish without ever reporting its tail upstream (no later restart re-reports it). The reporting round is extracted into report_range_once, shared by the tick loop and the final flush. New end-to-end regression test drives run_with_signals to a sync target and asserts the mock upstream saw the final tip; both new tests were mutation-tested against the reverted fixes. Co-Authored-By: Claude Fable 5 <[email protected]>
The final shutdown report inherited the periodic reporter's best-effort semantics (log + skip), but it has no next tick behind it: a transient endpoint blip at exactly shutdown would permanently lose an --end-block slice run's tail report while the process exits 0. report_range_once now returns whether the round settled, and the final flush retries up to 3 attempts (1s apart) before giving up with an ERROR log; a detected validation gap stays non-retried. Regression test fails the first report call via the mock and asserts the retry lands the tail (mutation-tested against FINAL_REPORT_ATTEMPTS = 1). Also fixes both pre-existing rustdoc warnings: re-export ValidationTask (it is ValidatorFetcher's public Output type) and backtick the <metrics-port> placeholder that rustdoc read as an unclosed HTML tag. Co-Authored-By: Claude Fable 5 <[email protected]>
Comment-only pass over PR #152: compress multi-sentence narratives to the minimal statement of the constraint, keep every load-bearing "why" (throttle rationale, redirect policy, wire-format contracts, golden-vector docs), and give each rationale a single home instead of restating it at every call site. No code changes. Co-Authored-By: Claude Fable 5 <[email protected]>
Address review feedback: R2WitnessClient::new now takes the same BackoffPolicy that app.rs builds from --rpc-initial-backoff-ms / --rpc-max-backoff-ms instead of hard-coded INITIAL_BACKOFF/MAX_BACKOFF constants, so R2 retry pacing is field-tunable. The retry loop gains up to 50% jitter (mirroring round_robin_with_backoff) so parallel validators slicing a range don't retry in lockstep through a shared R2 brownout, and the cfg!(test) constant shrinking is replaced by tests passing a millisecond-scale policy directly. Co-Authored-By: Claude Fable 5 <[email protected]>
…erver onto it Extracted from the coverage-replayer branch (PR #153) so it can merge independently, on top of PR #152. - stateless-core: LightWitnessFromSalt decodes kvs+levels directly from full SaltWitness bytes via a layout-mirroring deserializer — proof material is structurally consumed as raw bytes, no curve point is ever constructed (~1.4ms vs ~110ms wall on a 6.3MiB mainnet witness). collect_code_hashes helper; metadata-to-None corruption is now an error instead of a panic. - stateless-common: get_witness_light / get_witness_light_with_deadline on RpcClient, decode_witness_payload_light / decode_witness_response_light, WitnessSizeBreakdown::new_light (documented lower bound). - debug-trace-server: both RPC witness paths (data_provider fetch, chain-sync prefetch) use the light decode; the server never verifies proofs, so the full decode's EC work bought nothing. salt dep dropped. Co-Authored-By: Claude Fable 5 <[email protected]>
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…erf docs Review follow-ups on this PR: - estimate_witness_size removed: its only caller was the data-provider path this PR migrates to WitnessSizeBreakdown::new_light; no other in-workspace or mega-reth usage exists. - Doc comments no longer pre-reference the coverage-replayer (it lands with PR #153); consumers are named generically. - Perf numbers in docs recalibrated to a reproducible measurement on the committed 6.3MiB mainnet payload (14-core M4 Pro, median of 20): full decode ~112ms wall / ~1.4 core.s CPU, light ~3.7ms single-threaded (bincode stage); payload-level incl. zstd: 124.8ms vs 6.0ms. Co-Authored-By: Claude Fable 5 <[email protected]>
- RpcClient: the primary-failover round_robin_with_backoff invocation (6 constant args) lives once in a private witness_round_robin<T>; the fetch_witness_raw / fetch_witness_light thin wrappers are gone (callers pass the decoder + trace message directly). - witness_encoding: response framing (version prefix + base64) and payload decode (zstd + bincode-legacy) are each fixed in one place (decode_response_with / decode_payload_as); the four public decoders are now one-line delegations. - WitnessSizeBreakdown: new / new_light share from_counts; the light path passes 0 commitments, so the size constants and the MPT byte-sum live once. - fetch_witness_with: the trace message is emitted as the log message again instead of a field named trace_msg. No behavior change; 4 review agents (reuse / simplification / efficiency / altitude) found nothing else actionable. Co-Authored-By: Claude Fable 5 <[email protected]>
Machine-specific measurements go stale silently in code; the PR description is their canonical home. Comments keep the qualitative reasoning (per-commitment EC work dominates; light decode skips it) and light_witness.rs points at PR #154 for the numbers. Co-Authored-By: Claude Fable 5 <[email protected]>
Provenance belongs to git blame / GitHub, not comments; the qualitative constraint stands on its own. Co-Authored-By: Claude Fable 5 <[email protected]>
flyq
added a commit
that referenced
this pull request
Jul 17, 2026
…tion over the light decode Adds a validate_block_updates variant that consumes a zero-validation LightWitness (PR #154's light decode), so proof-skipping embedders never construct a curve point anywhere on the decode + replay path: the env oracle comes from from_light_witness, replay runs over WitnessDatabase<LightWitnessExecutor>, and derive_state_updates is now generic over the StateReader store. No IPA verification or pre-state-root anchor is possible in this form (the light decode discards all proof material); the caller's binding is the returned updates against a header-committed changeset. Locked by a fixture test asserting the light path derives byte-identical StateUpdates to the full-witness path and still reproduces the header state root. Co-Authored-By: Claude Fable 5 <[email protected]>
Troublor
approved these changes
Jul 20, 2026
vincent-k2026
approved these changes
Jul 20, 2026
flyq
added a commit
that referenced
this pull request
Jul 20, 2026
…erify_witness knob The full-node light validation mode is being removed on security-assumption grounds: without IPA verification and without a pre-state anchor, the witness is bound to the chain only through replay-changeset equality against the header-committed SaltDeltas, and that trust model is not worth carrying as a production switch. validate_block_updates now always verifies the witness IPA proof; ValidationOptions keeps only the expected_pre_state_root anchor. The light witness decode itself (#154) stays — it still serves debug-trace-server. Co-Authored-By: Claude Fable 5 <[email protected]>
vincent-k2026
added a commit
that referenced
this pull request
Jul 22, 2026
## Summary
Witness fetches hit the primary (witness-generator) and fall back to Cloudflare/R2 under one shared 3s deadline, but the metrics collapsed every provider — and every failure reason — into a single `method`-keyed series, and the retry loop's failure logs named neither the endpoint nor the block. There was no way to answer "did this 3s go to the witness-generator or the failover?", "how many were timeouts vs errors?", or "which block stalled on which endpoint?".
This PR attributes upstream RPC **latency, failure reason, timeout counts, and per-attempt logs per endpoint**, for any number of fallback endpoints. The mechanism lives in the shared `round_robin_with_backoff` retry loop, so both binaries benefit; the debug-trace-server (the witness path the on-call was investigating) gets the full per-endpoint Prometheus treatment.
## What changed
**`stateless-common` (shared retry loop + trait)**
- `RpcMetrics::on_rpc_complete(method, success, duration)` → `on_rpc_attempt(method, provider, outcome, duration)`, where `outcome` is a new `RpcAttemptOutcome { Success, Error, Timeout }` — splits a provider that *returned an error* from one that *stalled and had to be timed out*.
- New `on_rpc_deadline_exceeded(method, elapsed)` (default no-op) — the operator-facing "the whole logical call blew its deadline" signal, recorded exactly once per call.
- `round_robin_with_backoff` classifies each attempt, tags it with a bounded, credential-stripped endpoint label (`endpoint_label`, format `{idx}:{host}`, strips `user:pass@`/path/query), and threads that label into the per-attempt WARN/DEBUG logs.
- **`timeout` counts genuine provider stalls only** — a timeout that fired merely because the caller's deadline clamped the attempt window (deadline pressure, not an upstream stall) is attributed to `deadline_exceeded`, not blamed on the provider.
- Successful witness fetches log the serving endpoint and split `request_ms` (RPC round trip) from `decode_ms` (CPU decode).
- A `warn`-level span carries `block_number`/`block_hash` around the witness retry, so the generic loop's per-attempt failure logs inherit the block identifier (they have no block context of their own).
- `set_validated_blocks` (single attempt, outside the loop) classifies its own outcome the same way.
**`debug-trace-server`**
- Replaced the method-only `UpstreamMetrics` (a `success` bool) with `(method, provider, outcome)`-keyed attempt series plus a per-method deadline-exceeded counter.
**`stateless-validator`**
- Adapted `ValidatorMetrics` to the new trait; added `stateless_validator_rpc_deadline_exceeded_total{method}`.
## Metrics
debug-trace-server:
- `debug_trace_upstream_requests_total{method, provider, outcome}` — one per provider attempt that ran its course (`outcome` ∈ `success|error|timeout`).
- `debug_trace_upstream_duration_seconds{method, provider, outcome}` — per-attempt latency, per endpoint.
- `debug_trace_upstream_deadline_exceeded_total{method}` — **new**: logical calls that exhausted their deadline budget (also absorbs deadline-pressure timeouts).
stateless-validator:
- `stateless_validator_rpc_deadline_exceeded_total{method}` — **new**.
`provider` is `{idx}:{host}` — the host is human-readable, and the leading index keeps two endpoints that share an authority distinct while encoding primary (0) vs failover (1+). So `sum by (provider)` gives the per-endpoint split for any number of fallbacks.
## Logs
Per-attempt failure logs (stall / error) carry `method`, `provider=<idx:host>`, `provider_idx`, and — via the span — `block_number`. Successful witness fetches trace `provider`, `request_ms`, `decode_ms`. So a stall reads directly: which block, which endpoint, which method, how long. (A stall now logs once with its reason instead of also emitting a redundant "trying next".)
## Operational note (dashboards)
Same class of change as #154 — dashboards/alerts need a small rebase when this deploys:
- `debug_trace_upstream_errors_total` is **removed**; error count is now `debug_trace_upstream_requests_total{outcome=~"error|timeout"}`, timeouts separable as `{outcome="timeout"}` (genuine stalls; deadline pressure lives in `debug_trace_upstream_deadline_exceeded_total`).
- `debug_trace_upstream_requests_total` / `debug_trace_upstream_duration_seconds` gained `provider` + `outcome` labels — aggregating queries keep working; add `by (...)` where needed.
- Per-attempt upstream series carry a dynamic `provider` label, so they appear on first attempt; the deadline-exceeded counter is pre-registered per method.
## Testing
- `cargo test --workspace`: passes (network integration tests ignored as usual); `fmt` / `clippy --workspace --all-targets --all-features` clean.
- New tests: `endpoint_label` host extraction + credential/userinfo stripping + same-host `{idx}` disambiguation; `RpcAttemptOutcome`; a capturing `RpcMetrics` mock asserting per-endpoint attempt outcomes on a primary-fails/backup-succeeds failover and exactly one `on_rpc_deadline_exceeded` per deadline-exhausted call; and a captured-log test asserting the stall WARN carries `block_number` at the default `warn` filter (deadline-independent so it can't flake on CI timing).
## Notes
- `mega-reth` consumes `stateless-common` but does not implement `RpcMetrics`, so the trait change is contained to this repo; it needs no change when it next bumps the `stateless-common` tag.
- `provider_labels` are parallel-by-index to `providers`; a `debug_assert_eq!` at the retry-loop entry makes that invariant explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the zero-validation "light" witness decode to stateless-core/common and switches debug-trace-server onto it. Extracted from PR #153 so it can merge independently.
Light witness decode
LightWitnessFromSalt(stateless-core) decodes kvs+levels directly from fullSaltWitnessbytes via a layout-mirroring deserializer — proof material is structurally consumed as raw bytes, no curve point constructed. Locked by byte-exact tests against every mainnet fixture and the committedtest_data/mainnet/bench/6906405.zst. New API:get_witness_light/get_witness_light_with_deadline(RpcClient),decode_witness_payload_light/decode_witness_response_light,WitnessSizeBreakdown::new_light,collect_code_hashes.LightWitness::metadataon a corrupt witness now errors instead of panicking.Measured performance
On the committed real-mainnet payload
6906405.zst(3.72 MB compressed / 6.30 MiB uncompressed, 36,970 kvs, 65,358 parent commitments), Apple M4 Pro (14 cores), median of 20 runs after warmup:CPU measured via
/usr/bin/timeover 10-iteration runs (full-vs-light user-time delta ≈ 1.36 core·s per full decode). This table is the canonical home for the numbers — inline doc comments keep only the qualitative reasoning, since machine-specific measurements go stale silently in code.debug-trace-server
Both RPC witness paths (data_provider fetch, chain-sync prefetch) switch to the light decode — the server never verifies proofs, so the full decode's per-point EC work bought nothing. Unused
saltdep dropped.Operational note: the
witness_generatorsize metric now records a documented lower bound (excludes parent commitments), shifting the histogram down ~40-60% — dashboards/alerts keyed to it need rebasing when this deploys.Cleanup
estimate_witness_sizeis removed: its only caller was the data-provider path this PR migrates toWitnessSizeBreakdown::new_light(..).total(), and it has no other consumer in this workspace or in mega-reth. Doc comments no longer pre-reference the coverage-replayer (its consumer lands with #153).Testing
Full workspace suite green, incl. 8 new light-decode tests (layout-congruence locks on all mainnet fixtures + the committed 6.3MiB payload with full-stream-consumption asserts, bincode-config-agnosticism, EC-validation-skip proof, corrupt-metadata error path, size lower bound).
fmt/clippy --all-targets --all-features/cargo sortclean.Notes
#153 rebases over this branch; the doc rewording and
estimate_witness_sizeremoval will surface there as trivial conflicts inlight_witness.rs/data_types.rs/witness_size.rs.🤖 Generated with Claude Code