Changes proposed
Background
Mooncake Transfer Engine supports selecting local and remote RNICs in multi-RDMA-NIC environments through topology and NIC priority matrices. In production, however, a single RNIC may experience link down events, port errors, GID changes, QP errors, or other transient failures.
The current implementation already includes partial recovery mechanisms, such as endpoint reconstruction, GID reprobe, ready ACK, and metadata refresh. However, real RNIC down/up testing still exposed several gaps:
- When a preferred local RNIC becomes unavailable, queued or newly submitted slices may not smoothly fail over to another local RNIC.
- Peer RNIC failure and local RNIC failure are not clearly separated, so a local hardware issue may be misclassified as a remote rail issue.
- After an RNIC transitions from
PORT_ERR to PORT_ACTIVE, immediately publishing it as available may be too early. In real RoCE/mlx5 environments, the data path may still reject connections shortly after link-up.
- Passive handshake failures do not always return structured error information to the active side, making diagnosis difficult.
- Reusing old QPs after a pre-connected endpoint handshake failure may conflict with QP information already cached by the peer.
This RFC proposes improving RDMA RNIC failover recovery so that transfers can continue through fallback RNICs when a single RNIC fails, and recovered RNICs are safely republished.
Goals
- Support handing off unfinished slices to another active local RNIC after a local RNIC failure.
- Distinguish local RNIC failures from peer rail failures.
- Update and publish local topology when a context becomes inactive or recovers.
- Delay reactivating a recovered RNIC after
IBV_EVENT_PORT_ACTIVE.
- Improve handshake failure diagnostics.
- Preserve compatibility with existing RDMA ready ACK, GID reprobe, and endpoint lifecycle mechanisms.
- Provide a real link down/up validation script.
Non-Goals
- Do not introduce a new transport protocol.
- Do not change the upper-level Transfer Engine API.
- Do not guarantee lossless recovery from every hardware fatal error.
- Do not refactor Store or TENT failover mechanisms in this RFC.
- Do not change the consistency model of metadata backends.
Design Overview
1. Local RNIC Handoff
When a local RDMA context is marked inactive, or when the worker pool detects a local completion failure, the current worker no longer fails the slice immediately. Instead, it tries to hand the slice off to another active local context.
During handoff, the following remote-side information is preserved:
target_idpeer_nic_pathdest_addrdest_rkey
Only local send-side information is updated:
source_lkeyrdma.endpoint = nullptr
The slice is then re-enqueued into the fallback context's worker pool.
This avoids treating a local RNIC failure as a remote rail failure and reduces the need to re-resolve remote metadata.
2. Peer Rail Failure
For non-local completion failures or handshake failures, if an alternative peer rail is available, the current peer rail is paused and the slice is redispatched to another peer RNIC.
Peer rail pause state is managed with rail state tracking:
- Track error count.
- Set
pause_until_ns once the threshold is reached.
- Automatically make the rail available again after the pause expires.
For explicit handshake or ready ACK timeouts, the peer rail can be paused immediately.
3. Context Inactive State and Topology Publishing
When any of the following async events are received, the RDMA context is marked inactive:
IBV_EVENT_DEVICE_FATALIBV_EVENT_CQ_ERRIBV_EVENT_WQ_FATALIBV_EVENT_PORT_ERRIBV_EVENT_LID_CHANGE
After the context becomes inactive, Transfer Engine refreshes the local topology, disables that RNIC in the topology, and republishes the local segment descriptor.
Once peers refresh metadata, they can avoid selecting the failed RNIC.
4. Delayed Recovery
Receiving IBV_EVENT_PORT_ACTIVE does not necessarily mean the data path is immediately usable. In real RoCE/mlx5 environments, RTR/RTS transitions or path resolution may still fail for a short period after link-up.
The recovery flow therefore does not immediately republish the RNIC as active. Instead:
- Receive
PORT_ACTIVE.
- Set
recovery_activate_after_ns.
- The monitor worker waits until the recovery deadline.
- Refresh the current GID.
- If GID refresh succeeds, mark the context active again.
- Republish local topology.
The default recovery delay is 30 seconds.
5. Endpoint and QP Lifecycle Improvements
For endpoints that have not yet completed connection setup, handshake failure no longer causes the old QPs to be reset and reused.
The reason is that when the active side sees a failure, the passive side may already have received and cached the old QP numbers. Reusing those QPs could make the next handshake ambiguous with the previous partially processed handshake.
The new behavior is:
- For pre-connected endpoint disconnects, rebuild fresh QPs.
- For connected endpoint destruction, transition QPs to
ERR so outstanding WRs can be flushed.
6. Passive Handshake Error Replies
When the passive handshake callback fails, the server still returns a structured handshake response and fills reply_msg.
This lets the active side observe a concrete rejection reason instead of seeing an empty response or decode failure.
Example errors include:
- Invalid
peer_nic_path.
- Local RDMA context not found.
- Endpoint unavailable.
- Fresh endpoint unavailable after retiring a stale endpoint.
7. Submit-Path Fallback for Inactive Devices
The submit path may preselect a local device at request granularity. If that selected device later becomes inactive, the request-level selection must not be reused.
The new behavior is:
- Reuse the preselected context only if it is still active.
- If the preselected context is inactive, fall back to slice-level retry selection.
- Slice-level selection skips inactive contexts and chooses a fallback RNIC.
Validation
Unit and Integration Tests
The following scenarios were validated with mlx5_1 through mlx5_4:
- Sender primary RNIC inactive, transfer succeeds through fallback local RNIC.
- Sender RNIC receives an injected
PORT_ERR, while transfers continue for 20 seconds.
- Receiver RNIC receives an injected
PORT_ERR, while transfers continue for 20 seconds.
- Topology selection does not choose empty/disabled devices.
- RDMA context GID reprobe behavior remains correct.
Real Link Test
A real RDMA link failover validation script is added:
mooncake-transfer-engine/scripts/real_rdma_link_failover.sh
Default topology:
- Sender primary:
mlx5_2
- Sender fallback:
mlx5_1
- Receiver primary:
mlx5_3
- Receiver fallback:
mlx5_4
The script starts a target and an initiator, then runs:
ip link set <fault_netdev> down
sleep <down_for_sec>
ip link set <fault_netdev> up
Pass criteria:
- The initiator completes successfully.
- The primary RNIC is observed as inactive.
- The fallback RNIC is not marked inactive.
- No data integrity error is detected.
Compatibility
- No user-facing API changes.
- No required metadata format changes.
- Existing
ready_ack, GID reprobe, and metadata refresh behavior remains backward compatible.
- The new failover behavior only applies to RDMA multi-RNIC/fallback scenarios.
- In single-RNIC scenarios, if no handoff target exists, transfers still fail according to the existing retry/failure semantics.
Risks
1. Local Failure Misclassification
Some WC statuses may be caused by either local or remote issues. The implementation only classifies clearly local WC statuses as local failures to avoid disabling a context too aggressively.
2. Slow Recovery
The 30-second recovery delay may cause a recovered RNIC to rejoin later than strictly necessary. This is intentionally conservative and prioritizes stability.
3. Metadata Refresh Delay
Whether peers quickly avoid an inactive RNIC depends on metadata refresh or remote segment descriptor reload timing.
4. Multi-Worker Concurrency
Handoff enqueues slices across worker pools, so slice lifecycle and completion counters must remain correct under concurrency.
Future Work
- Make the recovery delay configurable.
- Integrate the real link failover test into a hardware CI environment.
- Add more WC status classification and vendor-specific handling.
- Expose RNIC inactive/recovered metrics.
- Extend show-link diagnostics to display inactive topology state.
Before submitting a new issue...
Changes proposed
Background
Mooncake Transfer Engine supports selecting local and remote RNICs in multi-RDMA-NIC environments through topology and NIC priority matrices. In production, however, a single RNIC may experience link down events, port errors, GID changes, QP errors, or other transient failures.
The current implementation already includes partial recovery mechanisms, such as endpoint reconstruction, GID reprobe, ready ACK, and metadata refresh. However, real RNIC down/up testing still exposed several gaps:
PORT_ERRtoPORT_ACTIVE, immediately publishing it as available may be too early. In real RoCE/mlx5 environments, the data path may still reject connections shortly after link-up.This RFC proposes improving RDMA RNIC failover recovery so that transfers can continue through fallback RNICs when a single RNIC fails, and recovered RNICs are safely republished.
Goals
IBV_EVENT_PORT_ACTIVE.Non-Goals
Design Overview
1. Local RNIC Handoff
When a local RDMA context is marked inactive, or when the worker pool detects a local completion failure, the current worker no longer fails the slice immediately. Instead, it tries to hand the slice off to another active local context.
During handoff, the following remote-side information is preserved:
target_idpeer_nic_pathdest_addrdest_rkeyOnly local send-side information is updated:
source_lkeyrdma.endpoint = nullptrThe slice is then re-enqueued into the fallback context's worker pool.
This avoids treating a local RNIC failure as a remote rail failure and reduces the need to re-resolve remote metadata.
2. Peer Rail Failure
For non-local completion failures or handshake failures, if an alternative peer rail is available, the current peer rail is paused and the slice is redispatched to another peer RNIC.
Peer rail pause state is managed with rail state tracking:
pause_until_nsonce the threshold is reached.For explicit handshake or ready ACK timeouts, the peer rail can be paused immediately.
3. Context Inactive State and Topology Publishing
When any of the following async events are received, the RDMA context is marked inactive:
IBV_EVENT_DEVICE_FATALIBV_EVENT_CQ_ERRIBV_EVENT_WQ_FATALIBV_EVENT_PORT_ERRIBV_EVENT_LID_CHANGEAfter the context becomes inactive, Transfer Engine refreshes the local topology, disables that RNIC in the topology, and republishes the local segment descriptor.
Once peers refresh metadata, they can avoid selecting the failed RNIC.
4. Delayed Recovery
Receiving
IBV_EVENT_PORT_ACTIVEdoes not necessarily mean the data path is immediately usable. In real RoCE/mlx5 environments, RTR/RTS transitions or path resolution may still fail for a short period after link-up.The recovery flow therefore does not immediately republish the RNIC as active. Instead:
PORT_ACTIVE.recovery_activate_after_ns.The default recovery delay is 30 seconds.
5. Endpoint and QP Lifecycle Improvements
For endpoints that have not yet completed connection setup, handshake failure no longer causes the old QPs to be reset and reused.
The reason is that when the active side sees a failure, the passive side may already have received and cached the old QP numbers. Reusing those QPs could make the next handshake ambiguous with the previous partially processed handshake.
The new behavior is:
ERRso outstanding WRs can be flushed.6. Passive Handshake Error Replies
When the passive handshake callback fails, the server still returns a structured handshake response and fills
reply_msg.This lets the active side observe a concrete rejection reason instead of seeing an empty response or decode failure.
Example errors include:
peer_nic_path.7. Submit-Path Fallback for Inactive Devices
The submit path may preselect a local device at request granularity. If that selected device later becomes inactive, the request-level selection must not be reused.
The new behavior is:
Validation
Unit and Integration Tests
The following scenarios were validated with
mlx5_1throughmlx5_4:PORT_ERR, while transfers continue for 20 seconds.PORT_ERR, while transfers continue for 20 seconds.Real Link Test
A real RDMA link failover validation script is added:
Default topology:
mlx5_2mlx5_1mlx5_3mlx5_4The script starts a target and an initiator, then runs:
Pass criteria:
Compatibility
ready_ack, GID reprobe, and metadata refresh behavior remains backward compatible.Risks
1. Local Failure Misclassification
Some WC statuses may be caused by either local or remote issues. The implementation only classifies clearly local WC statuses as local failures to avoid disabling a context too aggressively.
2. Slow Recovery
The 30-second recovery delay may cause a recovered RNIC to rejoin later than strictly necessary. This is intentionally conservative and prioritizes stability.
3. Metadata Refresh Delay
Whether peers quickly avoid an inactive RNIC depends on metadata refresh or remote segment descriptor reload timing.
4. Multi-Worker Concurrency
Handoff enqueues slices across worker pools, so slice lifecycle and completion counters must remain correct under concurrency.
Future Work
Before submitting a new issue...