From 729848e0d6ae4b9b564e34f5891a39f61aa4957c Mon Sep 17 00:00:00 2001 From: laminar Date: Fri, 17 Jul 2026 20:05:32 +0800 Subject: [PATCH 1/2] docs(llm-d): add the llm-d Integration page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document Mooncake's two integration points with llm-d as a link-first reference across three layers (vLLM connector, Mooncake storage, llm-d packaging): - vLLM connector contract — the single-node MooncakeStoreConnector command, the mooncake_config.json mode topology (embedded/standalone-store), and the PYTHONHASHSEED requirement. - KV cache offloading on K8s — deferred to llm-d's Tiered Prefix Cache guide and overlays, with a note on how to verify the Mooncake offload path specifically. - P/D KV transfer — vLLM's runnable MooncakeConnector example, the sidecar flags, and the capability-vs-packaged-path distinction. Pinned to llm-d v0.8.0; field references defer to the upstream vLLM docs, llm-d guide, and Mooncake store guide. --- .../llm-d-integration.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/source/deployment/kubernetes-deployment-guide/llm-d-integration.md diff --git a/docs/source/deployment/kubernetes-deployment-guide/llm-d-integration.md b/docs/source/deployment/kubernetes-deployment-guide/llm-d-integration.md new file mode 100644 index 0000000000..e9f7a0ca46 --- /dev/null +++ b/docs/source/deployment/kubernetes-deployment-guide/llm-d-integration.md @@ -0,0 +1,76 @@ +# llm-d Integration + +[llm-d](https://github.com/llm-d/llm-d) packages vLLM inference on Kubernetes; Mooncake is the KV layer beneath it. The integration is a **three-layer stack**: + +- **Engine — vLLM.** The Mooncake connectors live here: `MooncakeStoreConnector` (offload) and `MooncakeConnector` (P/D transfer), driven by `--kv-transfer-config`, `mooncake_config.json`, and `PYTHONHASHSEED`. +- **Storage — Mooncake.** The Master/Client binaries, the Transfer Engine (RDMA), and the SSD tier. +- **Packaging — llm-d.** Kustomize overlays that wire the engine and storage into a Deployment + DaemonSet, plus a routing sidecar. + +Mooncake plugs in at **two independent points**, both configured on the vLLM side: + +- **KV cache offloading (storage)** — `MooncakeStoreConnector` uses a Mooncake Store pool as a shared offload tier. This is llm-d's primary Mooncake path. +- **P/D KV transfer** — the routing sidecar's `--kv-connector=mooncake` drives vLLM's `MooncakeConnector` (see [below](#pd-transfer)). + +The two share the Transfer Engine but serve different purposes; vLLM's `MultiConnector` can compose them when a deployment needs both. For each layer, this page points at the authoritative source and keeps only what that source leaves for the integrator to reconcile. + +## vLLM connector contract + +The vLLM side of the integration is a single command. With the Mooncake Master already running and a `mooncake_config.json` on disk, `MooncakeStoreConnector` is wired on a single node with that config file and a connector spec: + +```bash +MOONCAKE_CONFIG_PATH=mooncake_config.json \ +vllm serve meta-llama/Llama-3.1-8B-Instruct \ + --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}' +``` + +The `mooncake_config.json` this command names has a `mode` field that selects the storage topology: + +| Mode | Who owns the DRAM pool | `global_segment_size` | Components | +|---|---|---|---| +| `embedded` | each vLLM rank contributes DRAM in-process | **> 0** (e.g. `"80GB"`) | Master + vLLM | +| `standalone-store` | an external `mooncake_client` owns the CPU DRAM + SSD pool; vLLM ranks are pure requesters | **`0`** | Master + Client + vLLM | + +The connector enforces the pairing at startup: `embedded` requires `global_segment_size > 0`, `standalone-store` requires `global_segment_size == 0`, and `local_buffer_size` must be `> 0` in both. `standalone-store` decouples the pool from vLLM — it survives vLLM restarts and can live on GPU-less nodes with large DRAM and NVMe. The SSD tier is a separate switch, not implied by the mode: `enable_offload` must be set together on the vLLM config, the Master, and the Client (vLLM even accepts `embedded` with `enable_offload`). + +llm-d instantiates this same contract at scale: the `--kv-transfer-config`, the `MOONCAKE_CONFIG_PATH` JSON, and `PYTHONHASHSEED` set here are what the guide's `patch-vllm.yaml` sets on each model-server Pod. vLLM's docs are the authoritative field reference: + +- [MooncakeStore connector usage](https://github.com/vllm-project/vllm/blob/main/docs/features/mooncake_store_connector_usage.md) — the `mooncake_config.json` schema (including the `mode` field), the single-node form above, and the `MultiConnector[MooncakeConnector + MooncakeStoreConnector]` composition for XpYd. +- [Mooncake connector usage](https://github.com/vllm-project/vllm/blob/main/docs/features/mooncake_connector_usage.md) — the P/D transfer connector and its bootstrap port (`8998`). + +```{important} +Set `PYTHONHASHSEED` to the **same fixed value on every instance sharing a Store**. Mooncake keys blocks by a content hash derived from vLLM's block hashes, and Python randomises its hash seed per process by default — leave it unset and two instances compute different keys for identical tokens, so the pool stays healthy while the cross-instance hit rate sits at zero. +``` + +## KV cache offloading on K8s (llm-d) + +llm-d's [Tiered Prefix Cache guide](https://github.com/llm-d/llm-d/tree/v0.8.0/guides/tiered-prefix-cache) packages MooncakeStore as an offload backend: evicted KV blocks move out of GPU HBM into a Mooncake Store pool that serves them back on a later hit, saving the recompute. Because the pool is shared, multiple vLLM instances reuse each other's cached prefixes. The guide carries the deployment sequence, node sizing, and manifests; deploy it with `helm install` for the router chart and `kubectl apply -k` over the Mooncake overlays. + +```{note} +**Upstream sources** (llm-d `v0.8.0`): +- Guide: [Tiered Prefix Cache](https://github.com/llm-d/llm-d/tree/v0.8.0/guides/tiered-prefix-cache) — the deployment walkthrough; the MooncakeStore path has `cpu` and `fs` variants. +- Architecture: [KV Offloader](https://github.com/llm-d/llm-d/blob/v0.8.0/docs/architecture/advanced/kv-management/kv-offloader.md) — Master/Client roles and a `mooncake_config.json` field table. +- Overlays: [`helpers/mooncake-master-store/`](https://github.com/llm-d/llm-d/tree/v0.8.0/helpers/mooncake-master-store) (Master; both variants), [`modelserver/gpu/vllm/mooncake-store/`](https://github.com/llm-d/llm-d/tree/v0.8.0/guides/tiered-prefix-cache/modelserver/gpu/vllm/mooncake-store) (`cpu`/`fs` vLLM overlays), [`helpers/mooncake-client/`](https://github.com/llm-d/llm-d/tree/v0.8.0/helpers/mooncake-client) (Client DaemonSet base; `fs` patches it up). +``` + +```{note} +**Confirming Mooncake specifically.** The guide's offload verification inspects the decode Pod's `/mnt/files-storage/kv-cache`, which is the vLLM-native/LMCache filesystem tier — a Mooncake `fs` deployment writes nowhere near it, so an all-`Ready` cluster does not prove the Store is working. Confirm Mooncake on the `mooncake_client` Pod instead: its `/data/mooncake-offload` should grow after a long request, and the Master should show the pool registered (metrics on `:9003`). +``` + +(pd-transfer)= +## P/D KV transfer (MooncakeConnector) + +The P/D transfer path is independent of the Store. vLLM ships a runnable single-node example for it — [`examples/disaggregated/mooncake_connector/`](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/mooncake_connector) (proxy + launch scripts, configurable model and bootstrap port). In a prefill/decode disaggregated deployment, llm-d's routing sidecar drives the same connector: + +```bash +# on the decode pod's routing sidecar +--kv-connector=mooncake +``` + +The sidecar queries a bootstrap endpoint on the prefill pods to resolve the target engine, then dispatches prefill and decode so the decoder pulls KV directly from the prefiller. + +| Setting | Default | Notes | +|---|---|---| +| `--kv-connector=mooncake` | — | Selects vLLM's `MooncakeConnector` on the sidecar. The router accepts only fixed connector names (`mooncake`, not `MultiConnector`), so the serving pods must run `MooncakeConnector` to match it — either as the top-level `kv_connector`, or nested inside a `MultiConnector`. | +| `--mooncake-bootstrap-port` / `MOONCAKE_BOOTSTRAP_PORT` | `8998` | Port of the Mooncake bootstrap endpoint on prefill pods. Corresponds to vLLM's `VLLM_MOONCAKE_BOOTSTRAP_PORT`. | + +**Capability vs. packaged path.** The `mooncake` connector reached a released sidecar in **llm-d-router v0.9.0** ([`supportedKVConnectors` at that tag](https://github.com/llm-d/llm-d-router/blob/v0.9.0/pkg/sidecar/proxy/options.go)), and the umbrella [llm-d v0.8.0](https://github.com/llm-d/llm-d/tree/v0.8.0) release pins that sidecar — so the sidecar **capability** is present. The **deployment path** is reference-only: neither v0.8.0 nor `main` ships a Mooncake P/D overlay, and llm-d's packaged [P/D disaggregation guide](https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation) still wires `--kv-connector=nixlv2`, so standing up Mooncake P/D means adapting the serving-pod and sidecar manifests by hand. For the current connector table and flags, see the sidecar's [disaggregation reference](https://github.com/llm-d/llm-d-router/blob/main/docs/disaggregation.md). From 3bdea5372858a66f137b24520657a64a7263cfeb Mon Sep 17 00:00:00 2001 From: laminar Date: Fri, 17 Jul 2026 20:05:32 +0800 Subject: [PATCH 2/2] docs: link the llm-d Integration page into the Kubernetes deployment guide Add the page to the guide's toctree and page list, and broaden the intro so the Store is framed as backing multiple orchestrators (SGLang/RBG and llm-d), not SGLang alone. --- docs/source/deployment/kubernetes-deployment-guide/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/source/deployment/kubernetes-deployment-guide/index.md b/docs/source/deployment/kubernetes-deployment-guide/index.md index c87126f436..315458d7ea 100644 --- a/docs/source/deployment/kubernetes-deployment-guide/index.md +++ b/docs/source/deployment/kubernetes-deployment-guide/index.md @@ -1,6 +1,6 @@ # Kubernetes Deployment Guide -Run Mooncake on Kubernetes as a shared **Store** cluster paired with SGLang **prefill/decode** inference — the Store serves as a HiCache L3 backend, while Mooncake's **Transfer Engine** moves KV cache directly between prefill and decode. +Run Mooncake on Kubernetes as a shared **Store** cluster. The primary scenario below pairs it with SGLang **prefill/decode** inference — the Store serves as a HiCache L3 backend, while Mooncake's **Transfer Engine** moves KV cache directly between prefill and decode. The same Store also backs other orchestrators and engine stacks: see [RBG Integration](rbg-integration) for the RBG operator, and [llm-d Integration](llm-d-integration) for vLLM-based KV offloading and P/D transfer under llm-d. --- @@ -33,6 +33,7 @@ A long-lived `mooncake-master` plus a replicated set of `mooncake-store` nodes f - [Mooncake on Kubernetes](mooncake-on-kubernetes) — stand up the Mooncake Store cluster with plain `Deployment` / `Service` objects. - [RBG Integration](rbg-integration) — the full Store + P/D scenario with the [sgl-project/rbg](https://github.com/sgl-project/rbg) operator, including a production Mooncake cluster case. +- [llm-d Integration](llm-d-integration) — Mooncake's two integration points in [llm-d](https://github.com/llm-d/llm-d): the Store as a vLLM KV offload tier, and `MooncakeConnector` for P/D transfer. Links to the upstream llm-d examples. See also the [Mooncake Store Deployment & Tuning Guide](../mooncake-store-deployment-guide.md) for the component overview, client configuration, and tuning knobs. @@ -42,4 +43,5 @@ See also the [Mooncake Store Deployment & Tuning Guide](../mooncake-store-deploy mooncake-on-kubernetes rbg-integration +llm-d-integration :::