Split out of PR #3256 review (found by Dawn).
crates/buzz-push-gateway/src/main.rs:22 still initializes logging with a plain blocking writer:
tracing_subscriber::fmt().json().with_env_filter(...).init();
It is deployed with a liveness probe (deploy/charts/buzz-push-gateway/templates/deployment.yaml:45 — timeoutSeconds: 3, failureThreshold: 3). Same node, same containerd log copier, same failure mode as the relay: a stalled copier fills the ~64KB stdout pipe, the next write(2) blocks holding Rust's global stdout lock, the Tokio runtime parks, the probe times out, kubelet kills the pod. If the root cause is node-level disk-IO pressure, it hits both workloads simultaneously.
Fix: reuse the relay's approach from #3256 — tracing_appender::non_blocking lossy mode + BoundedWorkerGuard + drop-counter metric (crates/buzz-relay/src/logging.rs). Likely worth hoisting that module somewhere shared rather than copy-pasting.
Refs: PR #3256, RESEARCH/RELAY_STALL_LOCAL_REPRO_2026_07_27.md (Eva's nest).
Split out of PR #3256 review (found by Dawn).
crates/buzz-push-gateway/src/main.rs:22still initializes logging with a plain blocking writer:It is deployed with a liveness probe (
deploy/charts/buzz-push-gateway/templates/deployment.yaml:45—timeoutSeconds: 3, failureThreshold: 3). Same node, same containerd log copier, same failure mode as the relay: a stalled copier fills the ~64KB stdout pipe, the nextwrite(2)blocks holding Rust's global stdout lock, the Tokio runtime parks, the probe times out, kubelet kills the pod. If the root cause is node-level disk-IO pressure, it hits both workloads simultaneously.Fix: reuse the relay's approach from #3256 —
tracing_appender::non_blockinglossy mode +BoundedWorkerGuard+ drop-counter metric (crates/buzz-relay/src/logging.rs). Likely worth hoisting that module somewhere shared rather than copy-pasting.Refs: PR #3256, RESEARCH/RELAY_STALL_LOCAL_REPRO_2026_07_27.md (Eva's nest).