[RFC]: Classic TE RDMA Two-Sided Transport (`rdma_twosided`)

[zhtshr]

Changes proposed

Summary

Add an opt-in classic Transfer Engine transport rdma_twosided for
PD-oriented workloads that prefer simpler buffer lifetime and TE-managed
bounce memory over peak one-sided bandwidth.

This is not a replacement for classic rdma (ibverbs WRITE/READ).
A process installs either rdma or rdma_twosided, not both.

Work is split into four reviewable PRs (large functional modules). Each PR
that introduces a runnable network path ships correctness tests plus a minimal
benchmark/smoke metric. End-to-end effectiveness vs one-sided is validated in
the final PR.

Related: #3324 (PR1, merged), #3440 (PR2, control plane).

Motivation

Classic one-sided RDMA requires applications to keep buffers registered for
the lifetime of inflight work and exposes remote rkeys to peers. For some PD
paths this is awkward:

1. Buffer lifetime — avoid repeated register/unregister around transfers.
2. TE-owned networking buffers — capacity can expand/shrink under credit.
3. Decouple remote memory ownership — SEND/RECV + local placement; no peer
   app rkey for the default path.
4. Acceptable latency — bounce/copy cost is OK for the target PD scenarios;
   peak GB/s remains on classic rdma.

Non-goals

• Replacing classic rdma as the default high-bandwidth path.
• Implementing this behind libfabric for the first version (EFA/CXI already use
  OFI for their fabrics; this path targets verbs/RoCE/eRDMA and stays on
  ibverbs to align with classic rdma).
• Changing the existing TE handshake/discovery stack (only extend capability
  fields as needed).

Architecture

TransferEngine / MultiTransport
  ├─ install("rdma")           → classic one-sided (unchanged)
  └─ install("rdma_twosided")  → new transport (mutually exclusive)

rdma_twosided
  ├─ CtrlChannel (per-peer notify QP)  — typed CtrlFrames, credit, notify
  ├─ MsgChannel  (per-peer/rail msg QP) — SEND/RECV + bounce pool
  └─ TE managed buffers + credit admission (WAITING when short)

Roadmap (4 PRs)

PR1 — Protocol primitives (#3324, merged)

Deliverable: CtrlFrame codec + SenderCreditLedger under
transport/rdma_twosided/, unit tests only.

Validation: unit tests (round-trip, epoch fencing, strict decode).
No network benchmark (no transport data path yet).

PR2 — Control plane (#3440)

Deliverable: RdmaTwoSidedTransport + CtrlChannel + handshake fields +
RDMA sendNotify (OOB fallback configurable).

Validation:

• Correctness: dual-engine notify tests (single / bidi / burst).
• Smoke metric: notify ops/s and latency on a real RDMA device.

PR3 — Data-plane channel

Deliverable: MsgChannel + bounce expand/shrink, with credit used for
RQ/bounce admission.

Validation:

• Correctness: SEND/RECV loopback / dual-engine data messages.
• Smoke: bounce expand under pressure.

PR4 — TE integration

Deliverable: managed-buffer APIs + submitTransfer two-sided path + docs;
wire PD install selection.

Validation:

• Correctness: managed WRITE/READ e2e, credit backpressure.
• Effectiveness: latency/bandwidth smoke vs classic one-sided; extend
  tebench/validators where practical.

Alternatives considered

Option	Why not (for v1)
Fold into classic rdma	Too invasive; hard to reason about and test.
libfabric / reuse EFA stack	Different fabric family; does not remove TE protocol work; weaker alignment with existing verbs rdma.
Single monolithic PR	Previously too large for review; split for reviewability and staged validation.

Open questions

• Final transport name (rdma_twosided vs something shorter).
• How aggressively tebench should grow in PR3 vs a dedicated follow-up.
• Whether credit-on-wire belongs entirely in PR2 or partially in PR3.

Ask

Please review the motivation, the separate-transport boundary, and the 4-PR
validation plan. PR1 is merged; PR2 is #3440.

[github-actions]

Thanks for opening this issue, @zhtshr!

Field	Value
Issue	#3377
GitHub user ID	44193225
Reporter	@zhtshr

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.