[RFC]: Mooncake EGM Store pool over Multi-Node NVLink for HBM–DRAM memory semantics

[neverhook]

Changes proposed

Summary

This RFC proposes adding a Mooncake Extended GPU Memory (EGM) Store pool backed by CPU DRAM and exposed through Multi-Node NVLink.

The goal is to support direct Mooncake Store transfers between GPU HBM and CPU DRAM inside one NVLink scale-up domain:

Put: GPU HBM -> NVLink Fabric -> remote EGM DRAM
Get: remote EGM DRAM -> NVLink Fabric -> GPU HBM

This provides the Mooncake-side foundation for the open Memory Semantics item in the vLLM Mooncake Store Connector roadmap: vllm-project/vllm#45036.

Here, memory semantics means that HBM buffers and Store-managed DRAM can participate directly in Mooncake Put/Get operations without an application-visible CPU staging copy. It does not mean cache-coherent CPU/GPU load/store, identical virtual addresses across nodes, or arbitrary remote pointer dereference.

Motivation

In disaggregated inference, active KV cache is typically stored in GPU HBM, while a larger external cache tier is stored in CPU DRAM. A conventional implementation may stage data through local host buffers before transferring it to the remote Store pool:

GPU HBM -> local CPU staging -> transport -> remote CPU DRAM

On GB200/NVL72-class systems, CPU DRAM can be allocated through CUDA VMM with __CUDA_EGM_LOCATION__, granted GPU access, exported as a Fabric handle, and mapped by GPUs in the same scale-up domain.

Mooncake already provides the required ownership boundaries:

• Mooncake Store owns capacity, object placement, allocation, and eviction.
• Transfer Engine owns memory registration, capability publication, remote mapping, and copy execution.
• Master tracks allocatable Store segments without needing CUDA-specific allocation metadata.

The proposed design changes the backing memory of Store segments while preserving the existing Store and NVLink transport model.

Potential use case: sparse-attention KV-pool offload

Sparse-attention models such as DeepSeek Sparse Attention (DSA) do not need the complete KV working set in GPU HBM for every decode step. Systems such as SGLang HiSparse keep the full KV cache in a host-memory pool, retain only a bounded hot KV buffer in HBM, and swap selected top-k KV entries into the device buffer on demand.

NVLink Fabric memory semantics could provide a scale-up-native backing tier for a similar design:

full decode KV pool in EGM DRAM
  -> sparse selector identifies required KV blocks
  -> selected blocks move over NVLink Fabric
  -> bounded hot KV buffer in GPU HBM

This could enable KV offload at the KV-pool level for future sparse-attention models while avoiding a PCIe-only host staging path. The EGM pool would hold the complete KV state, while the inference runtime controls sparse selection, prefetch, replacement, and the size of the HBM working set.

This is a potential downstream use case rather than a V1 requirement. Framework integration would still need KV block metadata, sparse-selection and prefetch scheduling, consistency with cache eviction, and interference evaluation against TP/EP traffic.

Goals

1. Allocate EGM Store capacity from CPU DRAM using CUDA VMM HOST_NUMA allocations.
2. Export each allocation with CU_MEM_HANDLE_TYPE_FABRIC.
3. Support HBM-to-EGM Put and EGM-to-HBM Get within one scale-up domain.
4. Preserve NUMA locality by using independent allocations for selected CPU NUMA nodes.
5. Reuse the existing nvlink protocol and Fabric-handle metadata format.
6. Keep the feature disabled by default.
7. Make setup, rollback, teardown, and partial CUDA failures explicit and testable.

Non-goals

The first version does not include:

• cache-coherent CPU/GPU load/store semantics;
• a new egm_store_pool protocol;
• cross-scale-up-domain access;
• RDMA fallback or dual-protocol selection;
• TENT MnnvlTransport integration;
• NUMA-aware object placement in Master;
• EGM-to-EGM performance guarantees;
• online pool expansion;
• new Store or Transfer Engine wire fields for memory kind or NUMA identity.

Proposed architecture

EGM is the user-facing Store capability. CUDA VMM HOST_NUMA is the V1 allocation mechanism and NUMA placement control, not a new transport type.

flowchart LR
    HBM[Consumer GPU HBM]
    Consumer[Mooncake Store Client]
    CNV[Consumer NvlinkTransport]
    PNV[Provider NvlinkTransport]
    VMM[CUDA VMM HOST_NUMA allocations]
    Pool[Mooncake EGM Store pool]
    Master[Mooncake Master]

    HBM --> Consumer --> CNV
    CNV <-->|Fabric import and CUDA copy| PNV
    PNV --> VMM --> Pool
    Pool <--> Master

Loading

The Store segment continues to use protocol="nvlink". The Consumer imports the existing serialized Fabric handle and does not need to know whether the remote allocation is backed by HBM or EGM DRAM.

Provider allocation and NUMA layout

The Provider discovers or accepts an explicit set of CPU NUMA nodes. In automatic mode it:

1. enumerates visible CUDA devices;
2. obtains their PCI BDFs;
3. resolves the corresponding CPU NUMA nodes;
4. validates and deduplicates the node set.

Store capacity is divided across these NUMA nodes. Each allocation is aligned to both CUDA VMM granularity and Mooncake Store allocator requirements, and is split into chunks no larger than max_mr_size.

Each chunk corresponds to:

• one CUDA VMM HOST_NUMA allocation;
• one Fabric-exportable handle;
• one Transfer Engine registered range;
• one Mooncake Store segment.

No allocation or Store segment crosses NUMA-node boundaries.

Setup and ownership

RealClient owns the EGM allocations and their mounted Store segments.

Setup proceeds as follows:

1. validate CUDA, MNNVL, Fabric, protocol, and NUMA configuration;
2. calculate the NUMA and chunk plan;
3. allocate all required VMM ranges;
4. register local-only workspace memory if configured;
5. register and mount global Store chunks;
6. expose readiness only after all chunks are published.

If setup fails, the Provider reverses completed operations in order:

1. unmount Store segments;
2. unregister Transfer Engine ranges;
3. release local allocator views;
4. unmap and release CUDA VMM allocations.

Cleanup failures retain ownership for retry instead of releasing an address that may still be referenced by Master or Transfer Engine metadata.

Transfer Engine registration

NvlinkTransport::registerLocalMemory() should distinguish:

• local-only memory: available as a local transfer source or destination but not remotely published;
• remote-accessible Store memory: exported through a Fabric handle and published in the existing BufferDesc format.

For a remote EGM allocation, registration retains the allocation handle, validates the complete mapped range, exports a CUmemFabricHandle, and publishes the range through existing metadata.

Consumer data path

The Consumer reuses the current lazy Fabric mapping flow:

1. locate the remote BufferDesc covering the object;
2. import its Fabric handle;
3. reserve and map a local VA;
4. grant access to visible GPUs;
5. cache the mapping;
6. execute the HBM-to-DRAM or DRAM-to-HBM CUDA copy.

No new Store API or replica type is required for the basic Put/Get path.

Configuration

The first version can use the existing ConfigDict setup interface:

enable_egm_store_pool = false | true
egm_numa_nodes  = auto | <comma-separated NUMA node IDs>

Existing fields retain their current meaning:

• global_segment_size: total Store capacity across all selected NUMA nodes;
• local_buffer_size: optional local transfer workspace;
• protocol: must be nvlink when EGM is enabled.

A Consumer does not need to enable EGM allocation in order to access a remote EGM-backed Provider segment.

End-to-end target scenario

The RFC is considered functionally complete when the following vLLM-oriented scenario works end to end:

1. A standalone Mooncake Store Provider on an NVL72-class system creates a DRAM pool distributed across its CPU NUMA nodes.
2. The pool is registered and mounted as ordinary Mooncake Store capacity using protocol="nvlink".
3. A vLLM worker registers an HBM KV-cache buffer with Mooncake.
4. The worker stores KV blocks directly from HBM into the remote EGM-backed Store pool.
5. Another vLLM worker in the same scale-up domain retrieves those blocks directly from Store DRAM into its HBM buffer.
6. Retrieved KV data is byte-correct and can be consumed by the model execution path without an application-visible CPU staging buffer.
7. Repeated transfers reuse imported mappings, while Provider setup failure and normal shutdown leave no published Store capacity or CUDA allocation ownership behind.
8. Disabling the feature preserves the existing Mooncake NVLink/HBM behavior.

Performance should be reported for this scenario, but this RFC does not define a fixed bandwidth or latency threshold before the hardware and workload configuration are agreed upon.

Known issues and follow-up work

Cross-domain RDMA fallback and dual protocol

Multi-Node NVLink only applies within one scale-up Fabric domain. A complete cluster-wide solution may need the same EGM allocation to be accessible through both NVLink and RDMA, with the Consumer selecting a transport based on topology and capability.

This requires a separate dual-protocol design covering:

• registration of one allocation with multiple transports;
• publication of multiple transport capabilities;
• topology-aware selection;
• health state and fallback policy;
• failure and consistency behavior during transport switching.

The initial RFC intentionally limits the contract to one scale-up domain and does not silently fall back to RDMA.

NVLink mapping ABA after Provider restart

The current NVLink mapping cache can reuse a mapping based on endpoint and remote address even after a Provider restarts and publishes a different allocation at the same address. This restart and allocation-identity problem is tracked separately in #2832.

EGM support should remain compatible with the capability-identity and mapping-retirement model selected there. This RFC does not introduce a second lifecycle mechanism.

NVLink Fabric contention with TP and EP traffic

KV-cache Put/Get traffic shares the NVLink/NVSwitch fabric with tensor-parallel, expert-parallel, and other scale-up communication. Large or concurrent KV transfers may therefore compete with latency-sensitive collective traffic and introduce transfer-latency or model-execution jitter.

NVLink currently does not expose a documented, generally applicable public interface for assigning per-transfer traffic priority or bandwidth isolation comparable to network SL/TC or DSCP. CUDA stream priority controls GPU work scheduling, but it should not be treated as a documented NVLink Fabric QoS guarantee.

V1 therefore cannot guarantee traffic isolation. The end-to-end evaluation should include concurrent TP/EP and KV-cache traffic and report interference and tail-latency variation. Admission control, transfer chunking, concurrency limits, or topology-aware scheduling may be required as follow-up work.

Open questions

1. Should V1 remain limited to legacy NvlinkTransport, or should the first implementation also target TENT?
2. Is keeping the existing Store and TE metadata unchanged acceptable for V1?
3. Should EGM Store pool configuration remain in ConfigDict, or should it become a first-class public setup API?
4. Should dual-protocol NVLink/RDMA support be designed before merge, or handled as a follow-up RFC?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation.

[github-actions]

Thanks for opening this issue, @neverhook!

Field	Value
Issue	#2914
GitHub user ID	41352674
Reporter	@neverhook

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[LCAIZJ]

I find this RFC quite intriguing. Leveraging memory semantics to enable direct Device-to-Remote-Host (D2RH) transfers over the NVLink Fabric on hardware platforms like NVL72 and GB200 represents a novel approach. I'd be interested to hear your thoughts on this. @stmatengss @ykwd

[neverhook]

Implementation PRs:

• [TransferEngine] Add NVLink VMM Fabric memory lifecycle #2934 — Transfer Engine NVLink VMM/Fabric lifecycle
• [Store] Add EGM-backed Store pool over NVLink #2935 — EGM-backed Store pool (stacked on [TransferEngine] Add NVLink VMM Fabric memory lifecycle #2934)

GB200 dual-node data-plane acceptance uses development-branch run gb200-0714-2. Capacity, correctness, concurrency, payload-size, and throughput results are embedded directly in both PR descriptions, including the comparison against the expected 50 GiB/s single-GPU RDMA ceiling.

[ykwd]

Thanks for sharing this RFC! Really promising direction. Looking forward to seeing it land.