feat(debug-trace-server): age-based witness routing, configurable old-block budget, tip buffer#158
feat(debug-trace-server): age-based witness routing, configurable old-block budget, tip buffer#158flyq wants to merge 3 commits into
Conversation
…-block budget, tip buffer Witness fetches route by block age: blocks at least --witness-local-window (default 4096, matching the witness generator's BACKUP retention) below the local DB tip skip the first witness endpoint — the internal generator, a guaranteed miss for anything it has pruned — via a skip parameter on the shared RpcClient witness path: one client, one witness semaphore, and retry-loop provider indices stay aligned with the configured endpoint list. Routing requires two or more witness endpoints and --data-dir (the local tip anchors block age); otherwise all fetches use the full chain and startup logs say so. The old-block witness budget (previously a hardcoded 3s fail-fast that was the dominant source of customer-visible -32001 timeouts) becomes --witness-old-block-timeout, defaulting to the full witness budget (8s). New --tip-buffer (default 2) keeps the chain-sync pipeline from racing the generator at the head, validated at startup to stay below --blocks-to-keep. Witness-fetch failure logs carry source/old_block/budget_ms/elapsed_ms for incident attribution, and the historical route gets its own witness_historical metrics label. The routing dispatch is tested end-to-end against two counting mock endpoints, plus route/label selection, env parsing of the new knobs, and a skip-all panic guard. README/AGENTS document the positional first-endpoint contract and the new knobs. Co-Authored-By: Claude Fable 5 <[email protected]>
…iered-fetch # Conflicts: # crates/stateless-common/src/rpc_client.rs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 569c41fcc8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // tip skip the first witness endpoint (the internal generator, which only retains about | ||
| // that window and is a guaranteed miss for anything older). Needs a fallback endpoint to | ||
| // route to, and a local DB whose tip anchors block age. | ||
| let route_historical = witness_apis.len() >= 2 && args.data_dir.is_some(); |
There was a problem hiding this comment.
In local-cache deployments that keep more history than --witness-local-window (for example --blocks-to-keep 10000) and restart while several thousand blocks behind, this route flag only feeds the request-time DataProvider; the background TraceFetcher constructed below still calls get_witness_light over the full endpoint chain. During catch-up, blocks older than the generator retention still pay one guaranteed generator miss or per-attempt stall before fallback on every block, so the chain-sync path can remain dominated by the pruned endpoint despite the local-cache routing contract.
AGENTS.md reference: AGENTS.md:L109-L110
Useful? React with 👍 / 👎.
…ite pressure The stall-WARN capture test raced two ways: a fixed 1s wall-clock window around the whole fetch could expire before a loaded runner scheduled the first 50ms attempt timeout, and concurrent tests driving the same tracing callsites with no subscriber installed could race set_default's interest-cache rebuild, re-caching a stale `never` for the instrument span so every WARN lost its block_number field. The fetch now runs as a background task while the test polls the captured logs (finishing on the first span-carrying WARN, 30s ceiling), periodically rebuilding the interest cache and respawning the fetch so a clobbered cache self-heals. Reproduced 6/15 before, 0/30 after under --test-threads=2. Co-Authored-By: Claude Fable 5 <[email protected]>
1 participant
Summary
Witness fetches in the debug-trace-server now route by block age: blocks at least
--witness-local-window(default 4096, matching the witness generator'sBACKUPretention) below the local DB tip skip the first witness endpoint — the internal generator, a guaranteed miss for anything it has pruned — and fetch straight from the fallback endpoints. Previously every historical request burned a doomed generator probe plus a failover round trip before reaching an endpoint that could actually serve it, and the hardcoded 3s old-block fail-fast was the dominant source of customer-visible-32001timeouts.Design
The historical route is a skip parameter on the shared witness path, not a second client:
RpcClient::get_witness_light_with_deadline_from(skip, ..)(crates/stateless-common/src/rpc_client.rs:643) sliceswitness_providers[skip..]and its parallel endpoint labels into the existing retry loop, whose newprovider_idx_base(rpc_client.rs:932) keeps logged provider indices aligned with the operator's configured endpoint order. One client means one witness concurrency semaphore (the--witness-max-concurrent-requestscap stays global) and no duplicate connection pools. Route selection is the purewitness_route(bin/debug-trace-server/src/data_provider.rs:789), and thewitness_historicalmetrics label splits the new route fromwitness_generatorin bothDataSourceMetricsandWitnessSourceMetrics.Routing activates only with ≥2 witness endpoints and
--data-dir(main.rs:370) — the local tip anchors block age — and startup logs state which mode is in effect, including a warning when multiple endpoints are configured without a DB. The first--witness-endpointis positionally the internal generator; README and AGENTS.md document the contract.New knobs:
--witness-old-block-timeout(main.rs:232) replaces the hardcoded 3s old-block fail-fast, defaulting to the full 8s witness budget and clamped to--witness-timeout;--witness-local-window(main.rs:222) sets the routing threshold;--tip-buffer(main.rs:239, default 2) keeps the chain-sync fetcher from racing the generator at the head, withvalidate_args(main.rs:295) rejectingtip_buffer >= blocks_to_keep(the built-in lag would otherwise trip the stale-reset threshold on every transient restart). Witness-fetch failure logs carrysource/old_block/budget_ms/elapsed_msfor incident attribution.Testing
The routing dispatch is pinned end-to-end: two counting mock endpoints assert a historical block never touches the generator and a recent block probes it first (
data_provider.rs), plus client-level skip isolation and a skip-all panic guard inrpc_client.rs. Unit tests cover theis_historicalboundary (including overflow and zero-window), the old-block budget clamp,witness_routelabel selection, CLI+env parsing of all three knobs, and thetip_buffer/blocks_to_keepvalidation boundary. Full workspacecargo test,cargo fmt --check, andcargo clippyare clean.Notes
--data-dir, routing is inert (unchanged fetch order), but the old-block budget default moves 3s → 8s everywhere — deployments that relied on the fail-fast should set--witness-old-block-timeout 3.origin/main(feat(metrics): per-endpoint RPC attempt metrics + timeout reasons #156, per-endpoint RPC attempt metrics) rather than rebasing; the retry-loop conflict was resolved by layering the skip/provider_idx_baseslicing onto feat(metrics): per-endpoint RPC attempt metrics + timeout reasons #156'sRpcAttemptOutcomeclassification, so sliced routes keep globally-correct endpoint labels in metrics and logs.