Skip to content

feat(write-stream): WALM-116 relayer write-stream redesign#283

Open
DalenMax wants to merge 29 commits into
devfrom
maxmai/walm-116-relayer-redesign-the-write-stream
Open

feat(write-stream): WALM-116 relayer write-stream redesign#283
DalenMax wants to merge 29 commits into
devfrom
maxmai/walm-116-relayer-redesign-the-write-stream

Conversation

@DalenMax

@DalenMax DalenMax commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Moves the effective Walrus upload concurrency budget from the TypeScript sidecar into the Rust relayer via a new WriteStreamLimiter.
  • Gates every write path: /api/remember, /api/remember/bulk, /api/remember/manual, /api/analyze, the wallet-worker upload path, and the legacy execute_remember upload path.
  • Adds cancellation-safe waiter tracking, per-item split_one batch permits, and WouldExceedCapacity fast-fail mapped to HTTP 429.
  • Wires Prometheus gauges/counters for permit state, acquisition outcomes, and per-route 429s.
  • Raises the sidecar WALRUS_UPLOAD_MAX_CONCURRENCY safety-net floor to 12.
  • Documents new env vars (WRITE_STREAM_MAX_CONCURRENCY, WRITE_STREAM_ACQUIRE_TIMEOUT_MS) and metrics in docs/relayer/.

Test Plan

  • cargo check passes
  • cargo fmt --check passes
  • cargo clippy --all-targets is clean for changed files (pre-existing unrelated lints allowed)
  • cargo test --bin memwal-server313 passed (with local Postgres + Redis via docker-compose.yml)
  • npm test in services/server/scripts → 78 passed
  • Full signed happy-path e2e against live Walrus/Sui not run — requires TEST_DELEGATE_KEY, on-chain account, and funded server wallets.

DalenMax added 28 commits June 15, 2026 20:39
@jessiemongeon1

jessiemongeon1 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Style Guide Audit

Audited 4 file(s) against the Sui Documentation Style Guide.

1 violation(s) found. All must be fixed before merge.

docs/superpowers/plans/2026-06-15-relayer-write-stream-redesign.md (1 violation(s))

1 violation(s) (1 regex, 0 claude)

  • Line 384 — Headings use sentence case (acronyms stay capitalized)
    • Current: Task 3: Add Prometheus metrics for the limiter
    • Fix: Capitalize only first word, proper nouns, and acronyms

Automated audit using the Sui Documentation Style Guide.

@DalenMax

Copy link
Copy Markdown
Collaborator Author

End-to-end verification update

I re-ran the live Walrus/Sui end-to-end tests that were deferred in the PR description. Both blockers from the earlier session are now resolved.

What changed to enable testing

  • Switched services/server/.env to testnet so local dev matches the Railway dev deployment.
  • Created a fresh testnet MemWalAccount (0xa8438a84a032d47603fe924f2b115f4800443e50039cd00c64650781b348a714) and registered a delegate key because the existing dev server wallet account had hit the 20-delegate-key cap.
  • Added the dev server wallet (0x3103b5ddad293bb00cf9b54061684293a829f2a65a7c560925e954f6e14a781f) as a delegate on that account so the sidecar can SEAL-decrypt during recall.

Railway dev results (https://relayer.dev.memwal.ai)

All auth/contract checks pass, and the full remember/recall happy path now works:

Check Result
GET /health
GET /version
Unsigned / wrong-key / expired auth → 401
POST /api/remember → poll → done blob_id=CEdARlLJqVrqxUxA58IaY5om0fF60i5J_CH1fWv17DA
POST /api/recall ✅ 1 hit returned
64 KiB remember ✅ server completed with blob_id; test script expects wrong field name (id vs job_id)
512 KiB remember ✅ server completed with blob_id; same test-script field issue
1 MiB + 1 remember ✅ 400 handler-level reject

Write-stream limiter verification

  • Rate limiter returns HTTP 429 after the delegate-key budget is exhausted.
  • Prometheus metrics are emitted: memwal_write_stream_permits_available, memwal_write_stream_waiters_total, memwal_write_stream_acquired_total, memwal_rate_limit_denials_total.

Minor follow-ups (not WALM-116 blockers)

  1. services/server/tests/e2e_test.py checks for "id" in the remember response, but the API returns "job_id". The size tests fail in the script even though the server jobs succeed.
  2. Henry's Linear comment about combining /api/analyze facts into "1 quilt" is a separate storage-layout feature request and should not block this PR.

I believe WALM-116 is ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants