[TENT] Integrate receiver-credit pull control with adaptive dispatch protection#2925
[TENT] Integrate receiver-credit pull control with adaptive dispatch protection#2925catyans wants to merge 51 commits into
Conversation
Snapshot receiver credit identity and charge at queue admission, reserve atomically before dispatch, and roll back exact reservations when transport submission fails. Add a default-off configuration and ownership-safe installation API, plus queue, restart, exhaustion, rollback, and multi-owner coverage.
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
# Conflicts: # mooncake-transfer-engine/tent/src/runtime/transfer_engine_impl.cpp
# Conflicts: # mooncake-transfer-engine/tent/tests/CMakeLists.txt
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a receiver-credit flow control mechanism for the Mooncake Transfer Engine (TENT) to manage outstanding bytes and request slots between senders and receivers. It adds key components including SenderCreditLedger, ReceiverCreditAllocator, ReceiverCreditPullController, and an adaptive dispatch limiter, integrating them into the transfer engine and admission queue. The review feedback highlights two critical issues: a protocol loop and credit leak when a sender restarts while the receiver remains active, and a potential null pointer dereference of the remote segment descriptor that could lead to a segmentation fault.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
# Conflicts: # mooncake-transfer-engine/benchmark/tent_backend.cpp # mooncake-transfer-engine/benchmark/utils.cpp # mooncake-transfer-engine/benchmark/utils.h # mooncake-transfer-engine/tent/include/tent/runtime/receiver_credit.h # mooncake-transfer-engine/tent/src/runtime/receiver_credit.cpp # mooncake-transfer-engine/tent/tests/CMakeLists.txt # mooncake-transfer-engine/tent/tests/receiver_credit_test.cpp
2561043 to
24a6510
Compare
|
Updated in 24a6510:
Cluster validation on lingjun-100 / codex_mooncake_pr_eval: Executed tests:
|
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
2 participants
Summary
Connect the receiver-credit model to a production-shaped sender-pull control path and protect queued RDMA dispatch with an adaptive owner-window limiter.
This is a stacked follow-up to #2860, #2861, and #2862. Until those prerequisites merge, GitHub will temporarily show their commits in this PR. The branch will be rebased and the visible diff reduced after the stack lands.
Related RFC: #2849
Benchmark harness: #2859
What this adds
ControlClient -> ControlService -> ReceiverCreditAllocatorpull RPCruntime_queue/max_dispatch_ownersWhy
Independent sender-local limits cannot bound aggregate receiver occupancy under many-to-one load. Receiver-owned credits provide the aggregate capacity contract, but the first TCP control rollout must also preserve control progress while TCP RPC and RDMA payload share an mlx5 path. The adaptive limiter prevents a persistently unsafe data window from repeatedly triggering TCP RTO-scale stalls.
Real-cluster results
Two-node H20/RoCE cluster, one NUMA node, one reachable
mlx5_0rail, four submit threads, 4 MiB WRITE, batch size 1, 20-second measurement interval:The default adaptive case was +0.19% versus the matched no-credit/static-2 baseline. Starting at the known-unsafe level improved throughput by 106.82% versus static-3 behavior and reduced RTO-scale pulls from 12 to 2.
Control RPC measurements
Dual-endpoint packet capture showed that the roughly 200 ms outliers were TCP retransmission events: the request was visible at the sender but first appeared at the receiver 201--207 ms later; once received, the receiver replied in roughly 10--50 us. DSCP 46 alone did not remove the unsafe-window loss on this VPC.
Validation
Remote-only validation on the H20 cluster:
receiver_credit_testreceiver_credit_control_testreceiver_credit_protocol_testreceiver_credit_allocator_testreceiver_credit_config_testreceiver_credit_control_service_testreceiver_credit_controller_testreceiver_credit_dispatch_testtent_runtime_queue_dispatch_testResult: 9/9 suites passed.
Compatibility
Receiver credits remain disabled by default. When enabled, runtime queue support and installed control state are required; incomplete setup fails closed rather than treating missing receiver capacity as unlimited.
Claims boundary
The primary demonstrated benefit is receiver-capacity protection and bounded recovery from an unsafe dispatch window without a measured healthy-path throughput regression in this setup. This PR does not yet claim SGLang/vLLM TTFT improvement.
The adaptive limiter bounds repeated damage but cannot guarantee zero first-loss if an operator raises the ceiling above a validated level. A typed independent control QP with reserved WR/CQ capacity remains the durable follow-up for eliminating shared-path TCP progress dependence.
Stack note
This draft intentionally opens before prerequisite merge, per the current development workflow. Merge conflicts and duplicate prerequisite commits will be cleaned after #2860--#2862 land.