From c4ac988f08e8008fc0049e490cc38c212824f23a Mon Sep 17 00:00:00 2001 From: rch Date: Fri, 17 Jul 2026 10:27:49 +0800 Subject: [PATCH] [docs] conductor docs --- .../conductor-architecture-design.md | 343 ++++---- docs/source/design/conductor/index.md | 41 + .../design/conductor/indexer-api-design.md | 741 ++++++++---------- docs/source/design/conductor/usage.md | 399 ++++++++++ docs/source/design/kv-event/index.md | 42 + .../design/kv-event/publisher-design.md | 378 ++++++--- .../design/kv-event/subscriber-guide.md | 350 +++++++-- .../conductor-architecture-design.md | 140 ++++ docs/source/zh/design/conductor/index.md | 38 + .../zh/design/conductor/indexer-api-design.md | 373 +++++++++ docs/source/zh/design/conductor/usage.md | 315 ++++++++ docs/source/zh/design/kv-event/index.md | 40 + .../zh/design/kv-event/publisher-design.md | 271 +++++++ .../zh/design/kv-event/subscriber-guide.md | 252 ++++++ 14 files changed, 2949 insertions(+), 774 deletions(-) create mode 100644 docs/source/design/conductor/index.md create mode 100644 docs/source/design/conductor/usage.md create mode 100644 docs/source/design/kv-event/index.md create mode 100644 docs/source/zh/design/conductor/conductor-architecture-design.md create mode 100644 docs/source/zh/design/conductor/index.md create mode 100644 docs/source/zh/design/conductor/indexer-api-design.md create mode 100644 docs/source/zh/design/conductor/usage.md create mode 100644 docs/source/zh/design/kv-event/index.md create mode 100644 docs/source/zh/design/kv-event/publisher-design.md create mode 100644 docs/source/zh/design/kv-event/subscriber-guide.md diff --git a/docs/source/design/conductor/conductor-architecture-design.md b/docs/source/design/conductor/conductor-architecture-design.md index e21cb74e6d..5345494558 100644 --- a/docs/source/design/conductor/conductor-architecture-design.md +++ b/docs/source/design/conductor/conductor-architecture-design.md @@ -1,182 +1,221 @@ # Mooncake Conductor Architecture -## Overview +[中文](../../zh/design/conductor/conductor-architecture-design.md) -Mooncake Conductor is the kv-cache indexer used by cache-aware routers. It -subscribes to KV cache events from inference engines or storage backends, -normalizes those events, maintains a global prefix cache table, and exposes -HTTP APIs for dynamic service registration and cache-hit queries. +Mooncake Conductor reads live key-value (KV) cache events and keeps an +in-memory cache index for routing decisions. This page explains how vLLM GPU +information and Mooncake CPU or Disk information enter that index, how query +tokens become lookup values, and how Low-Rank Adaptation (LoRA) adapters keep +cache entries separate. It describes the current C++ service, including the +limits that affect what a cache hit means. -The design goal is to let routers answer a simple scheduling question: -for this request prefix, which registered instance has the best reusable KV -cache locality, and on which cache tiers is that prefix available? - -## Architecture +## One event-to-result flow ```mermaid -flowchart TD - subgraph Router["Gateway or router"] - HTTPClient["HTTP client"] - end - - subgraph Conductor["Mooncake Conductor"] - EventManager["EventManager"] - ZMQClient["ZMQClient"] - KVEventHandler["KVEventHandler"] - PrefixCacheTable["PrefixCacheTable"] - end - - subgraph Publishers["vLLM, SGLang, Mooncake, or cache daemons"] - InferenceService["Inference or cache service"] - KVEvent["KV event stream"] - end - - EventManager -->|creates and manages| ZMQClient - ZMQClient -->|registers handler| KVEventHandler - InferenceService -->|publishes events| KVEvent - KVEvent -->|ZMQ SUB or DEALER| ZMQClient - ZMQClient -->|decoded batches| KVEventHandler - KVEventHandler -->|normalizes events| PrefixCacheTable - HTTPClient -->|/query, /register, /unregister| EventManager - EventManager -->|cache-hit computation| PrefixCacheTable - PrefixCacheTable -->|hit result by instance| EventManager - EventManager -->|HTTP response| HTTPClient +flowchart LR + V["vLLM KV events"] --> VR["Registered as type vLLM"] + M["Mooncake Master KV events"] --> MR["Registered as type Mooncake"] + VR --> VD["Read vLLM event maps"] + MR --> MD["Read Mooncake event maps"] + VD --> G["GPU blocks for one engine and data-parallel rank"] + MD --> S["Shared CPU or Disk objects"] + G --> I["In-memory cache index
tenant + model + LoRA + block size"] + S --> I + Q["POST /query
token IDs"] --> H["Hash each complete token block
use the final eight digest bytes"] + H --> I + I --> R["Result for each registered vLLM engine
and each data-parallel rank"] ``` -## Components +The registered `type` decides how Conductor reads a message. Topic text and +payload shape do not change that choice. Within one source, valid events are +applied in their received order. + +A vLLM event updates GPU information for the registered engine and +data-parallel (DP) rank. A Mooncake event updates shared CPU or Disk +information. A query hashes its complete token blocks, looks up each block in +order, and stops counting a particular result at that result's first missing +block. + +## What can share cache -| Component | Responsibility | +Four fields decide whether registrations, events, and queries use the same +cache information: + +| Field | Why it must match | |---|---| -| `EventManager` | Owns the Conductor lifecycle, HTTP server, dynamic registration, active service map, and tenant-to-instance map. | -| `ZMQClient` | Connects to publisher endpoints, consumes event frames, decodes event batches, tracks sequence numbers, and requests replay after reconnects. | -| `KVEventHandler` | Adapts decoded engine events into Conductor store/remove events enriched with registration metadata. | -| `PrefixCacheTable` | Maintains model-context-specific prefix maps, engine-hash to conductor-hash mappings, medium metadata, DP-rank metadata, and query-time hit computation. | +| `tenant_id` | Keeps one tenant's cache information separate from another tenant. | +| Model name | Prevents blocks from different models from being mixed. Registrations use `modelname`; queries use `model`; views use `model_name`. | +| `lora_name` | Separates the base model from each LoRA adapter. An empty string means the base model. | +| `block_size` | Sets both the number of tokens in one block and the step used when hashing a query. It must be positive. | +An engine's `instance_id` and DP rank do not create a separate sharing group. +They identify which GPU records and result rows belong to that engine. Request +`cache_salt` also does not change the four fields; it changes the block hash +chain for that query instead. -## Data model +Each four-field group has one registered `hash_profile`. A later registration +for the same fields must use the identical strategy, algorithm, exact +`python_hash_seed` text, derived root digest, and lookup rule. Only cache group +`0`, or no cache group, is currently supported. -The prefix index is scoped by `ModelContext`: +## vLLM GPU and Mooncake CPU/Disk information -```text -(tenant_id, model_name, lora_name, block_size, additional_salt, instance_id) -``` +| Registered source | Accepted cache location | What one stored block means | Where it appears in `/query` | +|---|---|---|---| +| `vLLM` | GPU, case-insensitive | This exact registered endpoint, engine, and DP rank reported the block. | Under that engine's `instances` row and DP rank. | +| `Mooncake` | CPU or Disk, case-insensitive | A Mooncake object provides the block to every registered engine with the same four cache-sharing fields. | As shared `cpu` or `disk` counts under each compatible vLLM engine. | -Within each context, Conductor stores: - -- a mapping from engine-provided block hash to Conductor prefix hash; -- a prefix hash map that records replica count, medium set, DP-rank set, and - per-instance access metadata; -- a DP-rank set used to report rank-level hit information. - -The current implementation computes complete-block prefix hashes from token IDs -and ignores trailing partial blocks during `/query`. - -## Event flow - -1. A service is registered statically from `conductor_config.json` or - dynamically through `POST /register`. -2. `EventManager` creates one `ZMQClient` per `(instance_id, tenant_id, - dp_rank)` service key. -3. `ZMQClient` subscribes to the publisher endpoint and consumes frames in the - form `[topic, sequence, payload]`. -4. The payload is decoded into a batch of engine events. Today, the implemented - parser supports vLLM `BlockStored` and `BlockRemoved` msgpack events. -5. `KVEventHandler` enriches events with registration metadata such as model, - LoRA, tenant, instance, block size, and additional salt. -6. `PrefixCacheTable` updates the prefix map for stored or removed blocks. -7. If a reconnect detects missed sequence numbers, the `replay endpoint` can be - used to request missed events. - -## Query flow - -1. A router obtains prompt token IDs, usually from an engine tokenizer endpoint. -2. The router calls `POST /query` with `model`, `token_ids`, `block_size`, and - optional `tenant_id`, `instance_id`, `lora_name`, and `cache_salt`. -3. Conductor computes complete-block prefix hashes for the request. -4. The prefix table is scanned in order. The first miss terminates the scan so - prefix continuity is preserved. -5. Conductor returns per-instance `longest_matched`, medium hit counts, and - DP-rank hit counts. -6. The router selects the best target instance and forwards the request. - -## Dynamic registration - -Conductor supports runtime registration so routers or control planes can add -and remove KV event publishers without restarting the process. +A Mooncake registration is a subscription name, not an inference engine. It +does not create another row in `/query`. Conductor can accept the subscription +before a compatible vLLM engine is registered, but Mooncake events cannot add +shared cache information until that four-field group exists. This is why the +[usage guide](./usage.md) registers engines before the shared pool. -```json -{ - "endpoint": "tcp://127.0.0.1:5557", - "replay_endpoint": "tcp://127.0.0.1:5558", - "type": "vLLM", - "modelname": "qwen2.5", - "lora_name": "", - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 128, - "dp_rank": 0, - "additionalsalt": "" -} +## How token blocks become lookup values + +Conductor hashes one complete token block at a time. The full 32-byte digest +from one block becomes the parent input for the next block; Conductor does not +chain from the shorter 64-bit lookup value. + +Before hashing blocks, Conductor resolves the supported profile as follows: + +```text +seed_text = exact `random` or ASCII decimal text in 0..4294967295 +seed_cbor = canonical-CBOR text(seed_text) +root_digest = lowercase_hex(SHA256(seed_cbor)) ``` -Static configuration uses the same fields under `kvevent_instance`: +The source profile supplies `python_hash_seed`, not `root_digest`. Its exact +text must equal the `PYTHONHASHSEED` environment value on every compatible +vLLM process. Conductor does not trim or normalize it, read its own process +environment, or infer it from KV events. Thus `"0"` and `"00"` produce +different roots, while numeric JSON `0` is invalid. The explicit text +`"random"` is supported; an unset `PYTHONHASHSEED` is not, because vLLM then +uses random root bytes that registration cannot reproduce. + +vLLM `--seed` controls model and sampling random-number generators and is +unrelated to prefix-cache hash identity. LoRA affects every block, a non-empty +request `cache_salt` affects the first block and all its descendants, and +Mooncake `additional_salt` remains diagnostic. `PYTHONHASHSEED` is compatibility +metadata, not a tenant-isolation or security key. + +For the currently supported resolved profile, each block is processed as +follows: + +1. The first parent is the root derived from `python_hash_seed`. Every later + parent is the complete SHA-256 digest of the previous block. +2. Conductor encodes a three-item canonical Concise Binary Object + Representation (CBOR) array: the parent digest as bytes, the block's signed + token integer array, and either `null` or an ordered array of extra strings. +3. A non-empty `lora_name` is the first extra string on every block. A + non-empty query `cache_salt` follows it on the first block only. Because the + first digest changes, the salt also changes every later digest in the chain. +4. Conductor calculates SHA-256 over those canonical CBOR bytes. +5. It reads the final eight digest bytes as one unsigned, big-endian 64-bit + integer and uses that integer to look up the block. + +A trailing group with fewer tokens than `block_size` is not hashed and does +not count as a cache hit. Mooncake event `additional_salt` is decoded for +diagnostics but does not currently enter the four cache-sharing fields or this +lookup calculation. + +### Labelled golden vector + +The repository fixture records **vLLM 0.22.0 `hash_block_tokens` semantics +with cbor2 6.1.1 and `canonical=True`**. Its example resolved profile records +`python_hash_seed` `"0"` and uses: ```json { - "http_server_port": 13333, - "kvevent_instance": { - "vllm-prefill-node1": { - "endpoint": "tcp://127.0.0.1:5557", - "replay_endpoint": "tcp://127.0.0.1:5558", - "type": "vLLM", - "modelname": "qwen2.5", - "lora_name": "", - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 128, - "dp_rank": 0, - "additionalsalt": "" - } - } + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" } ``` -## Environment variables +Canonical CBOR encodes the text string `"0"` as hexadecimal `6130`; this is +not the encoding of integer zero. SHA-256 of those two bytes produces the +shown root digest, which becomes the first block's parent. -| Variable | Default | Description | -|---|---|---| -| `CONDUCTOR_LOG_LEVEL` | `INFO` | Log level: `DEBUG`, `INFO`, `WARN`, or `ERROR`. | -| `CONDUCTOR_CONFIG_PATH` | `~/.mooncake/conductor_config.json` | Path to the static configuration file. | -| `CONDUCTOR_SEED` | random | Legacy seed option for hash computation experiments. | +For `block_size` `4`, empty `lora_name`, no `cache_salt`, and token IDs +`[1, 2, 3, 4, 5, 6, 7, 8]`, the fixture gives: -## Build and run +| Block | Full SHA-256 digest | Final eight bytes | Unsigned decimal lookup value | +|---|---|---|---| +| 1 | `c9d58ba695280d69b243e1e0df813136ca9196b286fb1a021e0b2e028ef071cb` | `1e0b2e028ef071cb` | `2164874634404590027` | +| 2 | `24125b23e68883b5c2141db2959d48433fe6bde2f26bd914efad121d154ab2d6` | `efad121d154ab2d6` | `17270480062156288726` | -```bash -cd mooncake-conductor/conductor-ctrl -go mod tidy -go build -o mooncake_conductor . -``` +This seed/root pair is a test vector for the producer and library versions +named above, not a default for every deployment. Registrations provide the +exact seed assertion; Conductor derives and reports the root as a diagnostic. -```bash -export CONDUCTOR_CONFIG_PATH=../example/conductor_config.json -export CONDUCTOR_LOG_LEVEL=INFO -./mooncake_conductor -``` +## What query fields mean -## Project structure +All counts are token counts, not block counts. Conductor returns one result for +each selected registered vLLM engine. -```text -mooncake-conductor/ -+-- conductor-ctrl/ -| +-- common/ # shared types, helpers, and sync map -| +-- kvevent/ # EventManager and KVEventHandler -| +-- prefixindex/ # prefix cache table and hit computation -| +-- zmq/ # ZMQ client, event decoding, event types -| +-- main.go # process entry point -+-- example/ # demo config and cache-aware proxy -+-- build.sh -+-- CMakeLists.txt -``` - -See [Indexer API](./indexer-api-design.md) for the HTTP API and KV Events wire -format. +| Result field | Meaning | +|---|---| +| `dp` | For each registered DP rank, the consecutive GPU prefix held by that exact engine and rank. Rank keys are decimal JSON strings. | +| `gpu` | The largest `dp` value for the engine. Blocks from different ranks are never joined to make a longer GPU prefix. | +| `cpu` | The consecutive prefix, starting at the first query block, that has at least one compatible shared CPU object. | +| `disk` | The consecutive prefix, starting at the first query block, that has at least one compatible shared Disk object. | +| `longest_matched` | The largest consecutive prefix one DP rank can serve using that rank's GPU blocks plus compatible shared CPU or Disk blocks. A block present in several locations is counted once. | + +For example, suppose one rank has the first two blocks on GPU, only the third +block is present in shared CPU cache, and `block_size` is `16`. That rank can +report `longest_matched: 48` and `gpu: 32`, while the independent `cpu` prefix +is still `0` because the first block is not in CPU cache. If the first and +second GPU blocks belong to different DP ranks, Conductor does not combine +those ranks into a two-block hit. + +## Cleanup + +Cleanup removes only information contributed by the affected source: + +- A vLLM remove or clear affects GPU records for the reporting endpoint, + engine, and DP rank. It preserves other ranks, other engines, and shared + cache information. +- Mooncake records keep the reporting endpoint, backend, tenant, object key, + full connector hash, and CPU or Disk location. Removing one object therefore + preserves another object even when their final eight hash bytes match. +- A Mooncake clear affects shared objects from the reporting endpoint, backend, + and tenant. It preserves vLLM GPU records and other Mooncake sources. +- Unregistering stops the selected `(instance_id, tenant_id, dp_rank)` + subscription before removing that endpoint's contributions. A vLLM + unregister also removes that rank from query results; a Mooncake unregister + removes all saved object bindings from that endpoint. + +The [Conductor subscriber guide](../kv-event/subscriber-guide.md) owns the +detailed event validation and cleanup rules. + +## Current limits + +- Conductor sees live events after it connects. The current Mooncake publisher + does not send a startup list of objects that were cached earlier. +- A jump in transport sequence numbers produces a warning and leaves existing + cache records in place. After reconnect, Conductor asks for missed events + only when `replay_endpoint` is configured and a previous sequence is known; + that request does not guarantee recovery. The current Mooncake publisher has + no replay service. +- Conductor processes Mooncake events in received batch order. It does not + automatically ignore an event merely because its `event_id` repeats. +- vLLM contributes only GPU information. Mooncake contributes only CPU or Disk + information. Other cache locations are ignored with a warning. +- Conductor reads layer and parallel-rank metadata from connector keys, but it + does not check whether all layers or all tensor-parallel (TP), prefill + context-parallel (PCP), decode context-parallel (DCP), or pipeline-parallel + (PP) parts are present before reporting shared availability. + +## Maintainer source note + +The main implementation paths for this page are +`mooncake-conductor/src/prefixindex/hash_strategy.cpp`, +`mooncake-conductor/src/prefixindex/prefix_indexer.cpp`, +`mooncake-conductor/src/kvevent/event_handler.cpp`, and +`mooncake-conductor/src/kvevent/event_manager.cpp`. The exact hash example is +from `mooncake-conductor/tests/fixtures/hash_golden_vectors.json`; query result +behavior is covered by `mooncake-conductor/tests/prefix_indexer_test.cpp` and +`mooncake-conductor/tests/event_manager_test.cpp`. diff --git a/docs/source/design/conductor/index.md b/docs/source/design/conductor/index.md new file mode 100644 index 0000000000..0df3df8d0b --- /dev/null +++ b/docs/source/design/conductor/index.md @@ -0,0 +1,41 @@ +# Mooncake Conductor + +[中文](../../zh/design/conductor/index.md) + +Mooncake Conductor keeps an in-memory view of reusable key-value (KV) cache +blocks reported by inference engines and Mooncake Store. A router can query +that view before choosing an inference engine. Use this page to find the guide +for running Conductor, understanding its results, or integrating a client. + +## Choose a task + +### Run Conductor + +Follow the [usage guide](./usage.md) to build the C++ service, configure event +sources, check registrations, query cache availability, and unregister a +source. + +### Understand the result + +Read the [architecture guide](./conductor-architecture-design.md) to see how +vLLM GPU events and shared Mooncake CPU or Disk events become per-engine query +results. + +### Call the HTTP API + +Use the [HTTP API reference](./indexer-api-design.md) for the five implemented +endpoints, their accepted fields, response bodies, and error formats. + +### Connect event sources + +Start with [KV Events](../kv-event/index.md) to compare the vLLM and Mooncake +message paths. That section also explains what Mooncake publishes and what +Conductor accepts. + +```{toctree} +:maxdepth: 1 +:hidden: + +conductor-architecture-design +usage +``` diff --git a/docs/source/design/conductor/indexer-api-design.md b/docs/source/design/conductor/indexer-api-design.md index 71ffa76bd5..b56fdde021 100644 --- a/docs/source/design/conductor/indexer-api-design.md +++ b/docs/source/design/conductor/indexer-api-design.md @@ -1,521 +1,392 @@ -# Mooncake Conductor Indexer API +# Mooncake Conductor HTTP API -## Overview +[中文](../../zh/design/conductor/indexer-api-design.md) -Mooncake Conductor is a KV cache indexer used by routers and gateways to make -cache-aware scheduling decisions. It consumes KV cache events from inference -engines or storage backends, maintains prefix-hit metadata across cache tiers, -and exposes HTTP APIs for service registration and cache-hit queries. +This reference describes the five HTTP endpoints implemented by the current +C++ Conductor service. Use it to register live event sources, remove them, +inspect Conductor's in-memory state, and query reusable cache prefixes. Field +names, allowed values, response casing, and error formats below follow the +current parser and serializer. -This document incorporates the latest API direction from: +## Choose an endpoint -- [RFC #1403: Mooncake KV-Store Indexer API Standardization](https://github.com/kvcache-ai/Mooncake/issues/1403) -- [RFC #1527: KV Events API Standardization](https://github.com/kvcache-ai/Mooncake/issues/1527) +| Method | Path | Purpose | +|---|---|---| +| `POST` | `/register` | Start one vLLM or Mooncake event subscription. | +| `POST` | `/unregister` | Stop one subscription and clean up that endpoint's cache information. | +| `POST` | `/query` | Query cache availability from prompt token IDs. | +| `GET` | `/global_view` | Inspect cache-sharing groups and registered vLLM ranks. | +| `GET` | `/services` | List active event subscriptions and their exact configuration. | -The Conductor can serve multiple model groups in one process. Each query is -scoped by model identity, block size, LoRA identity, tenant isolation, and the -registered instance that can receive traffic. +## Common request and response rules -## Concepts +All `POST` bodies are JSON objects. Unknown fields are rejected. Successful +responses use `application/json`; JSON object key order is not part of the +contract. -### Storage tiers +Most validation failures return status `400` with an +`application/json` object: -The indexer tracks KV cache availability across three logical tiers: +```json +{ + "error": "unsupported request field: root_digest", + "reason": "unknown_field", + "field": "root_digest" +} +``` -- **G1, Device Pool**: Device-resident KV blocks, such as GPU, NPU, HBM, or - other accelerator memory owned by inference engines. -- **G2, Host Pool**: CPU or host DRAM KV blocks, including Mooncake registered - memory pools. -- **G3, Disk Pool**: SSD, 3FS, DFS, NFS, or other disk-backed KV storage. +`field` is present when one field caused the error. `index` is also present +when one array element caused it. The [error formats](#understand-errors) +section lists the cases that return plain text instead. -The `medium` field identifies the concrete tier or device type. Common values -are `gpu`, `cpu`, and `disk`. Engines may add other values as new media are -supported. +## `POST /register` -### Identity dimensions +This endpoint starts one event subscription. Register each vLLM data-parallel +(DP) rank with a separate endpoint. A Mooncake subscription supplies shared CPU or Disk +information but does not create an inference-instance row in `/query`. -KV cache hits are interpreted under the following dimensions: +### Request fields -| Dimension | Description | +| Field | Required | Accepted value and when it matters | +|---|---|---| +| `endpoint` | Yes | Non-empty ZeroMQ live-publisher endpoint. It must not already belong to another active registration. | +| `type` | Yes | Exactly `vLLM` or `Mooncake`. It decides which event message format Conductor reads. | +| `modelname` | Yes | Non-empty registered model name. It defines vLLM context; Mooncake events carry the model that selects their actual shared context. | +| `instance_id` | Yes | Non-empty inference engine name for vLLM, or subscription name for Mooncake. It is part of the service key; a Mooncake value does not become a query instance. | +| `block_size` | Yes | Positive registered token count per block. It defines vLLM context; Mooncake events carry their actual block size. | +| `dp_rank` | Yes | Integer from `0` through the platform's maximum `int`. It selects the vLLM DP rank and is part of every service key; for Mooncake it is subscription identity only. | +| `hash_profile` | Yes | Object containing all four supported hash fields described below. | +| `replay_endpoint` | No | String endpoint used after a reconnect to ask for missed events. Defaults to `""`, which disables the replay socket. | +| `lora_name` | No | Registered Low-Rank Adaptation (LoRA) adapter name, default `""`. It defines vLLM context; Mooncake events carry their actual LoRA name. | +| `tenant_id` | No | Registered tenant name. Omitted or `""` becomes `"default"`. It defines vLLM context and the service key; Mooncake events carry their actual tenant. | +| `cache_group` | No | Integer `0` or `null`. Omission also means no explicit group. Other values and arrays are rejected. | + +The only supported `hash_profile` shape is: + +| Hash field | Supported value | |---|---| -| `model_name` or `model` | Model identifier. KV blocks from different models are incompatible. | -| `block_size` | Number of tokens per KV block. Different block sizes produce different token-to-block mappings. | -| `additional_salt` | Opaque salt used to separate hash namespaces for quantization, model revision, tenant isolation, or other deployment-specific dimensions. | -|`cache_salt`| Ensure cached data blocks are kept separate for different customers| -| `lora_name` | LoRA adapter name. Empty or `null` means the base model. | -| `tenant_id` | Upstream tenant or customer identity. Used for isolation and to keep query output bounded. | -| `instance_id` | Routable API server or engine instance returned by the Indexer API. Routers use this value as the scheduling target. | -| `backend_id` | KV Events identity for the entity that owns the KV blocks. It may be an inference worker, a Mooncake storage daemon, or another cache backend. | -| `medium` | Cache medium where the blocks are present. | -| `dp_rank` | Data-parallel rank that owns or can serve the blocks. | - -`instance_id` and `backend_id` intentionally have different meanings. -`instance_id` is the router-facing target in the Indexer API. `backend_id` is -the event-facing cache owner in the KV Events API. In deployments where cache -storage is decoupled from inference workers, `backend_id` can identify a cache -daemon while `instance_id` still identifies the engine endpoint that receives -requests. - -## Hashing standard - -The standardized event contract recommends **XXH3-64 with seed `S`**. - -- **Local block hash**: - `XXH3(token_bytes_le, S)`, where tokens are little-endian `u32` values - concatenated for one block. -- **Rolling sequence hash**: - The first block uses `seq_hash[0] = local_block_hash[0]`. Each subsequent - block uses: - - ```text - seq_hash[i] = XXH3(seq_hash[i-1]_le || local_block_hash[i]_le, S) - ``` - - Here `||` means byte concatenation, not a logical OR. - -All hashes used by the standardized KV Events API are rolling sequence hashes. -A `seq_hash` identifies the whole prefix up to that block depth, so equal -prefixes produce equal hashes until the first differing block. - -If an engine does not follow the standardized hashing scheme, it must provide -`token_ids` in `stored` events so the consumer can recompute the indexer's hash -representation. - -## HTTP APIs - -### `POST /register` - -Registers a KV event publisher and starts consuming events from it. - -```json -{ - "endpoint": "tcp://1.1.1.1:5557", - "replay_endpoint": "tcp://1.1.1.1:5558", - "type": "vLLM", - "modelname": "deepseek", - "lora_name": "sql-adapter", - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 128, - "dp_rank": 0, - "additionalsalt": "w8a8" -} +| `strategy` | `vllm_v1` | +| `algorithm` | `sha256_cbor` | +| `python_hash_seed` | String containing exactly `random` or ASCII decimal text whose numeric value is in `0..4294967295`. The original accepted text, including leading zeroes, is preserved. | +| `index_projection` | `low64_be` | + +The input object must contain exactly those four fields. Empty, signed, +whitespace-padded, non-string, invalid UTF-8, and out-of-range seeds are +rejected. `"0"` and `"00"` are distinct; numeric JSON `0` is invalid. +Registration-time `root_digest` is a legacy unknown field and is rejected. + +Each four-field cache-sharing group must use one identical resolved hash +profile. For vLLM those fields come from registration; for Mooncake they come +from each event, while the profile comes from the Mooncake registration. +Conductor canonical-CBOR encodes the exact seed text and calculates SHA-256 to +derive the root. The exact text must match `PYTHONHASHSEED` on every compatible +vLLM process. The explicit string `random` is supported, but leaving the +environment variable unset makes vLLM choose random root bytes that Conductor +cannot reproduce and is unsupported. vLLM `--seed` controls model and sampling +randomness, not prefix-cache hash identity. + +See [how token blocks become lookup values](./conductor-architecture-design.md#how-token-blocks-become-lookup-values) +for the canonical Concise Binary Object Representation (CBOR) input order, +LoRA-before-`cache_salt` ordering, complete parent-digest chaining, +final-eight-byte big-endian lookup rule, and labelled golden vector. +`cache_salt` is a query field, not a registration field. +Mooncake event `additional_salt` is diagnostic and is not accepted by this +HTTP endpoint. + +### Minimal request + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5557", + "type": "vLLM", + "modelname": "test-model", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' ``` -| Field | Required | Description | -|---|---|---| -| `endpoint` | Yes | ZMQ KV event publisher endpoint. | -| `replay_endpoint` | No | ZMQ replay endpoint used to recover missed events. | -| `type` | Yes | Publisher type, such as `vLLM`, `SGLang`, or `Mooncake`. | -| `modelname` | Yes | Model name for this publisher. This is the HTTP API wire name for `model_name`. | -| `lora_name` | No | LoRA adapter name. Empty or omitted means base model. | -| `tenant_id` | No | Tenant identity. Defaults to `default`. | -| `instance_id` | Yes | Router-facing engine or API server instance identity. | -| `block_size` | Yes | KV block size in tokens. | -| `dp_rank` | Yes | Data-parallel rank for this publisher. | -| `additionalsalt` | No | HTTP API wire name for `additional_salt`. Defaults to an empty string. | - -Successful response: +Status `200` returns: ```json { "status": "registered successfully", - "instance_id": "vllm-prefill-node1" + "instance_id": "engine-a" } ``` -### `POST /unregister` +Submitting the exact same active registration again returns the same success +without starting another subscriber. A conflicting service key, endpoint, or +hash profile returns a JSON `400` with `reason` `invalid_registration`. +Failure to start the local subscription client returns plain-text `500`. -Stops consuming events for a registered publisher. +## `POST /unregister` -```json -{ - "type": "vLLM", - "modelname": "deepseek", - "lora_name": "sql-adapter", - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 128, - "dp_rank": 0 -} -``` +This endpoint stops one exact subscription. It waits for that subscriber to +stop before removing cache information contributed by its endpoint. + +### Request fields + +| Field | Required | Accepted value and when it matters | +|---|---|---| +| `instance_id` | Yes | Non-empty string used when the source was registered. | +| `dp_rank` | Yes | Non-negative integer rank used when the source was registered. | +| `tenant_id` | No | String used when the source was registered. Omitted or `""` becomes `"default"`. | + +No other fields are accepted. + +### Minimal request -`tenant_id` defaults to `default`. The current implementation removes the -subscription identified by `(instance_id, tenant_id, dp_rank)`. +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","dp_rank":0}' +``` -Successful response: +Status `200` returns the exact service key that was removed: ```json { "status": "unregistered successfully", - "removed_instances": ["vllm-prefill-node1|default|0"] + "removed_instances": [ + "engine-a|default|0" + ] } ``` -### `POST /query` +An unknown service key returns plain-text status `404`. A cleanup failure after +the subscriber stops returns plain-text status `500`. Conductor keeps that +service key and endpoint reserved, so neither can be registered again. Retry +the same `/unregister` request until cleanup succeeds. -Query cache hits by token IDs. +## `POST /query` -```json -{ - "model": "deepseek", - "lora_name": "sql-adapter", - "token_ids": [101, 15, 100, 55, 89], - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 64, - "cache_salt": "w8a8" -} -``` +This endpoint hashes complete token blocks with the profile already registered +for the requested tenant, model, LoRA name, and block size. The request cannot +override the strategy, algorithm, `python_hash_seed`, derived root digest, or +final-eight-byte lookup rule. + +### Request fields -| Field | Required | Description | +| Field | Required | Accepted value and when it matters | |---|---|---| -| `model` | Yes | Model name. | -| `lora_name` | No | LoRA adapter name. Empty or omitted means base model. | -| `lora_id` | No | Deprecated compatibility field. Do not use together with `lora_name`. | -| `token_ids` | Yes | Prompt token IDs. Only complete blocks are considered. | -| `tenant_id` | No | Tenant identity. Defaults to `default`. | -| `instance_id` | No | If set, query one instance. If omitted, query all instances registered under the tenant. | -| `block_size` | Yes | KV block size in tokens. | -| `cache_salt` | No | Query-side hash namespace salt. Corresponds to the event `additional_salt` concept. | +| `model` | Yes | Non-empty model name. It must match registration `modelname`. | +| `block_size` | Yes | Positive integer. Only complete groups of this many tokens are hashed. | +| `token_ids` | Yes | JSON array of signed 32-bit integers. An empty array is valid and reports zero hits for compatible registered ranks. | +| `tenant_id` | No | String. Omitted or `""` becomes `"default"`. | +| `lora_name` | No | String. Defaults to `""` for the base model. | +| `cache_salt` | No | String, `null`, or omitted. `null`, `""`, and omission all select the no-salt hash path; a non-empty value must match the producer. | +| `instance_id` | No | String filter. A matching registered instance is returned alone; an unknown value returns an empty `instances` object. | + +No hash-profile override fields are accepted. A missing cache-sharing group +also returns status `200` with an empty `instances` object and does not create +state. + +### Minimal request + +```bash +curl -sS -X POST http://127.0.0.1:13333/query \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "test-model", + "block_size": 16, + "token_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] + }' +``` -Response: +If `engine-a` rank `0` is registered and no matching cache event has arrived, +the response is: ```json { - "default": { - "vllm-prefill-node1": { - "longest_matched": 256, - "GPU": 128, - "DP": { - "0": 128, - "1": 256 + "instances": { + "engine-a": { + "longest_matched": 0, + "dp": { + "0": 0 }, - "CPU": 256, - "DISK": 0 + "gpu": 0, + "cpu": 0, + "disk": 0 } } } ``` -| Field | Description | -|---|---| -| `longest_matched` | Longest continuous prefix hit in tokens across all tracked media and DP ranks for this instance. | -| `GPU`, `CPU`, `DISK` | Matched prefix tokens available on each medium. Names are examples; future media can be added. | -| `DP` | Matched prefix tokens grouped by data-parallel rank. | +### Result fields -### `POST /query_by_hash` +| Field | Meaning | +|---|---| +| `instances` | Map keyed by selected registered vLLM `instance_id`. Mooncake subscriptions are not result rows. | +| `longest_matched` | Largest consecutive token count one rank can serve using its GPU plus compatible shared CPU or Disk blocks. | +| `dp` | Per-rank consecutive GPU token count. Rank keys are decimal JSON strings. | +| `gpu` | Maximum consecutive GPU token count among this engine's ranks. | +| `cpu` | Independent consecutive shared CPU token count starting at the first block. | +| `disk` | Independent consecutive shared Disk token count starting at the first block. | -Queries cache hits by precomputed rolling sequence hashes. This API avoids -sending long token lists over the network. +For the current HTTP test state, instance `1` rank `0` has a 32-token GPU +prefix, instance `2` rank `1` has none, and both see 48-token shared CPU and +Disk prefixes. The exact response is: ```json { - "model": "deepseek", - "lora_name": "sql-adapter", - "seq_hashes": [1234567890, 9876543210], - "tenant_id": "default", - "instance_id": "vllm-prefill-node1", - "block_size": 64, - "cache_salt": "w8a8" + "instances": { + "1": { + "longest_matched": 48, + "gpu": 32, + "dp": { + "0": 32 + }, + "cpu": 48, + "disk": 48 + }, + "2": { + "longest_matched": 48, + "gpu": 0, + "dp": { + "1": 0 + }, + "cpu": 48, + "disk": 48 + } + } } ``` -For compatibility with earlier drafts, clients may call the hash list -`block_hash`, but new clients should use `seq_hashes` to make it explicit that -the values are rolling sequence hashes rather than local block hashes. - -The response shape is the same as `/query`. For this API, -`longest_matched = block_size * matched_hash_count`. +See [what query fields mean](./conductor-architecture-design.md#what-query-fields-mean) +for the continuity and per-rank rules. +## `GET /global_view` -## KV Events API +This endpoint shows the cache-sharing groups known to this Conductor process. +It takes no request body. -The KV Events API is the wire contract between cache owners and indexers. -Conductor normalizes engine-specific events into this model. +### Minimal request -### Event envelope +```bash +curl -sS http://127.0.0.1:13333/global_view +``` -Every standardized event carries the same envelope: +After registering `engine-a` ranks `0` and `1`, before cache events arrive, a +one-context response has this shape: ```json { - "event_id": 42, - "timestamp": 1739145600000, - "event_type": "stored", - "model_name": "llama-3.1-8b", - "block_size": 64, - "additional_salt": null, - "lora_name": null, - "tenant_id": "default", - "backend_id": "worker-0", - "medium": "gpu", - "dp_rank": 0 + "context_count": 1, + "contexts": [ + { + "model_name": "test-model", + "lora_name": "", + "block_size": 16, + "tenant_id": "default", + "prefix_count": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" + }, + "instances": { + "engine-a": [ + 0, + 1 + ] + } + } + ] } ``` -| Field | Type | Description | -|---|---|---| -| `event_id` | `u64` | Monotonically increasing sequence number scoped by the event stream dimensions. Authoritative for ordering. | -| `timestamp` | `u64 or null` | Unix epoch milliseconds. Informational only, not used for ordering. | -| `event_type` | `string` | One of `stored`, `removed`, or `cleared`. | -| `model_name` | `string or null` | Model identifier. | -| `block_size` | `u32 or null` | Tokens per block. | -| `additional_salt` | `string or null` | Opaque deployment salt or namespace. | -| `lora_name` | `string or null` | LoRA adapter name, or `null` for the base model. | -| `tenant_id` | `string` | Tenant or customer identity. | -| `backend_id` | `string` | Entity that owns the KV blocks. This can be an engine worker or a decoupled cache daemon. | -| `medium` | `string or null` | Cache medium such as `gpu`, `cpu`, or `disk`. | -| `dp_rank` | `u32 or null` | Data-parallel rank. | - -Events must be processed in consecutive `event_id` order within each stream -identified by `(model_name, block_size, additional_salt, lora_name, tenant_id, -backend_id, medium, dp_rank)`. - -### `stored` - -Published when one or more consecutive blocks are committed to a KV cache. +`context_count` is the number of entries in `contexts`. `prefix_count` counts +distinct final-eight-byte lookup values that still have at least one GPU, CPU, +or Disk record. `instances` maps registered vLLM engines to numeric rank +arrays; Mooncake subscriptions are not added to this map. Context array order +is not guaranteed. The resolved `hash_profile` reports the exact configured +seed and its derived lowercase root together. -```json -{ - "event_id": 42, - "timestamp": 1739145600000, - "event_type": "stored", - "model_name": "llama-3.1-8b", - "block_size": 64, - "additional_salt": null, - "lora_name": null, - "tenant_id": "default", - "backend_id": "worker-0", - "medium": "gpu", - "dp_rank": 0, - "seq_hashes": [1234567890, 9876543210, 1122334455], - "base_block_idx": 5, - "parent_hash": 9999999999, - "token_ids": null -} -``` +## `GET /services` -| Field | Type | Description | -|---|---|---| -| `seq_hashes` | `u64[]` | Rolling sequence hashes of consecutive stored blocks. | -| `base_block_idx` | `u32 or null` | Zero-based depth of the first block in this event. | -| `parent_hash` | `u64 or null` | Rolling sequence hash at depth `base_block_idx - 1`; `null` at the root. | -| `token_ids` | `u32[] or null` | Tokens across all blocks in this event. Required when the publisher does not use the standardized hash. | +This endpoint lists active subscription configurations. It takes no request +body. Field names intentionally use the casing shown below, which differs from +the register request. -At least one of `base_block_idx` or `parent_hash` must be present so the -consumer can locate the blocks in the sequence. +### Minimal request -### `removed` +```bash +curl -sS http://127.0.0.1:13333/services +``` -Published when one or more blocks are evicted. +After the minimal register example, status `200` returns: ```json { - "event_id": 43, - "timestamp": 1739145601000, - "event_type": "removed", - "model_name": "llama-3.1-8b", - "block_size": 64, - "additional_salt": null, - "lora_name": null, - "tenant_id": "default", - "backend_id": "worker-0", - "medium": "gpu", - "dp_rank": 0, - "seq_hashes": [1122334455], - "base_block_idx": 7 + "count": 1, + "services": [ + { + "Endpoint": "tcp://127.0.0.1:5557", + "ReplayEndpoint": "", + "Type": "vLLM", + "ModelName": "test-model", + "LoraName": "", + "TenantID": "default", + "InstanceID": "engine-a", + "BlockSize": 16, + "DPRank": 0, + "CacheGroup": null, + "HashProfile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" + } + } + ] } ``` -`seq_hashes` is required. `base_block_idx` is optional but recommended for -collision detection and observability. +`count` is the number of active service keys. `CacheGroup` is `null` when the +register request omitted it. Service array order is not guaranteed. +Registration appearing here confirms local subscription setup, not remote +event delivery or recovery of earlier events. Compare both +`python_hash_seed` and derived `root_digest` with `/global_view` before sending +cache-producing traffic. + +## Understand errors -### `cleared` +Conductor deliberately uses both JSON validation errors and plain-text +operational errors: + +| Situation | Status | Content type and body | +|---|---|---| +| Field validation for any `POST` endpoint | `400` | `application/json` with `error`, `reason`, and applicable `field` or `index`. | +| Malformed `/query` JSON or a non-object body | `400` | `application/json` with `{"error":"Invalid JSON object","reason":"invalid_json"}`. | +| Malformed `/register` or `/unregister` JSON, or a non-object body | `400` | `text/plain; charset=utf-8` with `Invalid JSON\n`. | +| `/unregister` service key not found | `404` | `text/plain; charset=utf-8`, for example `service not found: engine-a\|default\|0\n`. | +| `GET` on `/register`, `/unregister`, or `/query`; `POST` on `/global_view` or `/services` | `405` | `text/plain; charset=utf-8` with `Method not allowed\n`. | +| `/register` cannot start the local subscription client | `500` | `text/plain; charset=utf-8` beginning `Failed to subscribe: failed to start ZMQ client:`. | +| `/unregister` stops the subscriber but cache cleanup fails | `500` | `text/plain; charset=utf-8` beginning `Failed to unregister prefix context:`. | -Published when all blocks for the event stream dimensions are purged. +For example, a string in `token_ids` produces an element-specific JSON error: ```json { - "event_id": 44, - "timestamp": 1739145602000, - "event_type": "cleared", - "model_name": "llama-3.1-8b", - "block_size": 64, - "additional_salt": null, - "lora_name": null, - "tenant_id": "default", - "backend_id": "worker-0", - "medium": "gpu", - "dp_rank": 0 + "error": "token_ids element must be a JSON integer", + "reason": "invalid_type", + "field": "token_ids", + "index": 0 } ``` -No additional payload fields are required. - -## Compatibility notes - -The current Conductor implementation consumes vLLM ZMQ msgpack batches and -normalizes `BlockStored` and `BlockRemoved` into the internal prefix index. -Registration metadata supplies fields such as `modelname`, `tenant_id`, -`instance_id`, `block_size`, and `additionalsalt` when the engine event does -not carry the full standardized envelope. - -### Mooncake Store master publisher - -`mooncake_master` can optionally publish RFC #1527 events when -`enable_kv_events=true`. The publisher binds a ZMQ PUB socket -(`kv_events_bind_endpoint`) and emits the same three-frame batch format used by -vLLM/SGLang: empty topic, big-endian sequence number, and a msgpack payload -`[timestamp, [events], dp_rank]`. - -**Per-block events with fixed publisher context.** Per the -[Dynamo KV Events for Custom Engines](https://docs.nvidia.com/dynamo/kv-managers/kv-events-for-custom-engines) -model, each event describes one or more **KV cache blocks** (`seq_hashes`, -`token_ids`, `parent_hash`, eviction hashes). The master emits one event per -Mooncake object key and available medium; each key is treated as one pooled -block. Block identity and available model context come from the object key. -Keys generated by the vLLM connectors encode model, parallel-rank -namespaces, KV group, and a hexadecimal prefix hash. The master extracts the -model and low 64 bits of the hash while retaining the complete digest in -`connector_block_hash` and the complete key in `object_key`. -The master always supplies per-object `tenant_id` and `medium`. A deployment -uses one Publisher for one fixed model, block-size, LoRA, salt, and DP context, -so the master fills those envelope fields from `kv_events_*` configuration. -The key-derived model takes precedence over the configured fallback. Parent -hash, token IDs, and block depth remain unset because neither the connector key -nor publisher config contains them. - -An Upsert of an existing object is represented as `removed` when the old value -becomes unreadable, followed by `stored` for every completed medium after the -new value commits. There is no separate update event type. - -Publisher-level config includes transport and stream identity plus the fixed -model context: `kv_events_bind_endpoint`, `kv_events_backend_id`, -`kv_events_model_name`, `kv_events_block_size`, -`kv_events_additional_salt`, `kv_events_lora_name`, and `kv_events_dp_rank`. -Optional compat flags are `kv_events_emit_object_key` and -`kv_events_emit_legacy_compat`. Per-object tenant identity is not replaced by -the global `kv_events_tenant_id` compatibility setting. - -Each event map uses RFC #1527 field names (`event_type`, `seq_hashes`, -`backend_id`, `medium`, and so on). When `kv_events_emit_object_key` is enabled -(default), the map also includes `object_key` with the Mooncake store key so -Dynamo and other consumers can match on `sha256` + Mooncake key format without -requiring decimal/`0x` `seq_hash` encoding. When `kv_events_emit_legacy_compat` -is enabled (default), the map also includes aliases such as `type` and -`block_hashes`. These aliases ease field mapping but do not make the master map -payload wire-compatible with vLLM's positional event arrays; a decoder adapter -is still required. - -Recognized connector keys end in a hexadecimal rolling prefix hash. Simple -object keys may instead encode a decimal or `0x`-prefixed u64. When no hash can -be parsed, events are still published if `kv_events_emit_object_key=true` (with -an empty `seq_hashes` array). Configure -`backend_id` to identify the cache owner (for example a per-node storage -daemon) and register the bind endpoint with the indexer using publisher type -`Mooncake`. - -`connector_block_hash` is a Mooncake extension containing the complete -hexadecimal connector digest; `seq_hashes` and legacy `block_hashes` remain -u64 arrays. Recognized keys may also expose `cache_prefix`, `tp_rank` or -`head_or_tp_rank`, `pcp_rank`, `dcp_rank`, `pp_rank`, and `layer_id`. -`cache_prefix` is a deployment namespace rather than request -`additional_salt`, and none of the connector rank fields is `dp_rank`. -Connector keys carry neither parent linkage nor block depth, so the master -emits both `parent_hash` and `base_block_idx` as nil. - -See the [KV Event publisher design](../kv-event/index) for connector key -formats, publication points, and medium transition rules. - -### Field provenance matrix (SGLang vs master vs indexer registration) - -In decoupled deployments (inference workers + Mooncake host/disk pool), the -global KV indexer merges **three sources of truth**. Use this table when -splitting publishers or writing PR/integration notes. - -**Legend** - -| Symbol | Meaning | -|---|---| -| **SGLang** | Inference engine ZMQ KV events (`BlockStored` / `BlockRemoved` / `AllBlocksCleared`) | -| **Master** | `mooncake_master` optional RFC #1527 publisher (`enable_kv_events`) | -| **Register** | Indexer HTTP `POST /register` (or CLI `--workers`) — not carried on the event wire | -| **S+M** | Either source may supply; must agree on value for the stream | -| **—** | Not applicable for that event type | - -#### Envelope and stream identity - -| Field | SGLang | Master | Register | Notes | -|---|---|---|---|---| -| `event_id` | Yes | Yes | — | Each publisher maintains its own monotonic counter per stream. | -| `timestamp` | Yes | Yes | — | Informational only; not used for ordering. | -| `event_type` | Yes | Yes | — | `stored` / `removed` / `cleared`. | -| `model_name` | S+M | Conditional | S+M | Master uses a recognized connector key first, then configured fixed model fallback. | -| `block_size` | Yes | Conditional | Yes | Master emits configured non-zero `kv_events_block_size`; registration must agree. | -| `additional_salt` | Yes | Conditional | S+M | Master emits non-empty fixed publisher salt; register uses `additionalsalt`. | -| `lora_name` | Yes | Conditional | S+M | Master emits configured fixed LoRA; nil means the base model. | -| `tenant_id` | S+M | Yes | Yes | Per-object on master events. Register default `default`. | -| `backend_id` | S+M | Yes | — | **Master**: storage daemon / pool owner. **SGLang**: often worker id; in decoupled mode prefer master=`daemon`, engine via **Register** `instance_id`. | -| `medium` | Yes | Partial | — | **SGLang**: `GPU`, `CPU_PINNED`, `DISK`, `EXTERNAL`, etc. **Master**: only `cpu` / `disk` (host/disk pool), never GPU. | -| `dp_rank` | Yes | Yes | Yes | Master emits configured rank in every event and the ZMQ batch trailer; registration must agree. | - -#### `stored` payload - -| Field | SGLang | Master | Register | Notes | -|---|---|---|---|---| -| `seq_hashes` | Yes | Conditional | — | **Required from SGLang** for correct prefix index. Master uses the low 64 bits of a recognized connector hash or a directly encoded decimal/`0x` u64 key. | -| `object_key` | — | Yes | — | Mooncake store key (`kv_events_emit_object_key`, default on). Used by Dynamo for sha256+key matching. | -| `connector_block_hash` | — | Conditional | — | Complete hexadecimal hash parsed from a recognized connector key; use this for full-digest matching instead of `seq_hashes`. | -| `cache_prefix` | — | Conditional | — | vLLM Mooncake deployment namespace. It is distinct from request `additional_salt`. | -| Connector rank fields | — | Conditional | — | Optional `tp_rank`/`head_or_tp_rank`, `pcp_rank`, `dcp_rank`, `pp_rank`, and `layer_id`; none are `dp_rank`. | -| `block_hashes` (legacy) | Yes | Conditional | — | Alias of `seq_hashes` when `kv_events_emit_legacy_compat` is enabled on master. | -| `parent_hash` | Yes | — | — | Connector object keys do not carry the parent link. | -| `parent_block_hash` (legacy) | Yes | — | — | Same as `parent_hash`. | -| `base_block_idx` | Yes | — | — | Connector keys do not carry block depth, so master emits nil. | -| `token_ids` | Yes | — | — | Connector object keys do not carry token IDs. | -| `block_size` (in-event) | Yes | Conditional | — | Master uses configured non-zero fixed publisher block size. | - -#### `removed` payload - -| Field | SGLang | Master | Register | Notes | -|---|---|---|---|---| -| `seq_hashes` | Yes | Conditional | — | **Required** on wire for strict RFC consumers. Master emits one hash when parseable, else empty with `object_key`. | -| `base_block_idx` | Yes | — | — | Optional but recommended for observability. | - -#### `cleared` payload - -| Field | SGLang | Master | Register | Notes | -|---|---|---|---|---| -| (no extra fields) | — | — | — | Event is envelope-only. | -| `cleared` / `AllBlocksCleared` | Yes | Yes | — | Master emits a tenant-scoped event after `RemoveAll` leaves that tenant empty. | - -#### Indexer / router plane (not in KV event JSON) - -| Field | SGLang | Master | Register | Notes | -|---|---|---|---|---| -| `instance_id` | — | — | Yes | Router-facing schedule target. Distinct from `backend_id`. | -| `endpoint` | — | — | Yes | ZMQ PUB to subscribe (SGLang or master bind address). | -| `replay_endpoint` | — | — | Yes | Optional gap replay (engine ROUTER). | -| `type` | — | — | Yes | Publisher kind: `vLLM`, `SGLang`, `Mooncake`, etc. | - -#### Recommended split for Dynamo global KV indexer - -```mermaid -flowchart LR - SGLang["SGLang ZMQ"] - Master["Mooncake master ZMQ"] - Reg["POST /register"] - Idx["Global KV indexer"] - - SGLang -->|"GPU + HiCache tiers
tokens, parent_hash, seq_hashes, lora"| Idx - Master -->|"Host/Disk pool
fixed model context, backend_id, medium=cpu|disk"| Idx - Reg -->|"instance_id, model, block_size"| Idx -``` - -| Capability | Primary source | -|---|---| -| GPU prefix hits, LoRA-aware hashes, parent chain, multi-block batches | **SGLang** | -| Pooled host/disk replica visibility | **Master** (if keys encode `seq_hash`) | -| Request routing target | **Register** (`instance_id`) | -| Tiered `/query` response (`gpu` / `cpu` / `disk`) | Merge **SGLang** + **Master** events (see RFC #1403) | +Conductor-generated JSON responses end with a newline except `/services`, +whose compact JSON body has no trailing newline. Plain-text errors shown with +`\n` above include that trailing newline. diff --git a/docs/source/design/conductor/usage.md b/docs/source/design/conductor/usage.md new file mode 100644 index 0000000000..8d32ebea10 --- /dev/null +++ b/docs/source/design/conductor/usage.md @@ -0,0 +1,399 @@ +# Run Mooncake Conductor + +[中文](../../zh/design/conductor/usage.md) + +This guide takes a Conductor deployment from a source checkout to a working +HTTP query. It covers static and dynamic event-source registration, a safe +startup order for vLLM and Mooncake, and exact cleanup commands. The examples +use the current C++ service and only fields accepted by the current parsers. + +## Before you start + +Install the repository build dependencies described in the +[build guide](../../getting_started/build.md). The Conductor component also +requires ZeroMQ, MessagePack for C++, OpenSSL, JsonCpp, glog, and +yalantinglibs, which CMake checks during configuration. + +You also need: + +- one key-value (KV) Event endpoint for each vLLM data-parallel (DP) rank you + register; +- optionally, a Mooncake Master KV Event endpoint for shared CPU or Disk + information; +- `curl` for the checks below; and +- one explicit `PYTHONHASHSEED` value shared by every compatible vLLM + process. + +The addresses, model name, and seed below are examples. Replace them with +values from the same deployment before sending cache-producing traffic. Do not +leave `PYTHONHASHSEED` unset: vLLM then creates random root bytes that Conductor +cannot reproduce from registration. + +## Build + +Configure from the repository root and build only the Conductor target: + +```bash +cmake -S . -B build -DWITH_CONDUCTOR=ON +cmake --build build --target mooncake_conductor +``` + +The resulting binary is +`build/mooncake-conductor/mooncake_conductor`. Confirm that the build produced +it: + +```bash +test -x build/mooncake-conductor/mooncake_conductor +``` + +A zero exit status means the binary is ready to run. + +## Configure + +Conductor reads two environment variables: + +| Variable | Default | When it matters | +|---|---|---| +| `CONDUCTOR_CONFIG_PATH` | `$HOME/.mooncake/conductor_config.json` | Selects the JSON file loaded at startup. A missing file starts Conductor with no static subscriptions and leaves the built-in HTTP port at `13333`. | +| `CONDUCTOR_LOG_LEVEL` | `INFO` | Sets `DEBUG`, `INFO`, `WARN`, or `ERROR`, case-insensitively. An empty or invalid value uses `INFO`. | + +Always set `http_server_port` in a readable config file. If a config file is +loaded but omits this field, the current parser sets the port to `0` rather +than using `13333`. + +The following complete static example registers two vLLM ranks for one engine +and one Mooncake shared pool. An empty `lora_name` selects the base model; a +non-empty value names a Low-Rank Adaptation (LoRA) adapter. + +```json +{ + "http_server_port": 13333, + "kvevent_instance": { + "vllm-engine-a-rank-0": { + "endpoint": "tcp://127.0.0.1:5557", + "replay_endpoint": "tcp://127.0.0.1:5558", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }, + "vllm-engine-a-rank-1": { + "endpoint": "tcp://127.0.0.1:5567", + "replay_endpoint": "tcp://127.0.0.1:5568", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 1, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }, + "mooncake-shared-pool": { + "endpoint": "tcp://127.0.0.1:6557", + "replay_endpoint": "", + "type": "Mooncake", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "shared-pool", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + } + } +} +``` + +`python_hash_seed` is the exact string in the `PYTHONHASHSEED` environment +variable on every compatible vLLM process. Conductor preserves that text, +canonical-CBOR encodes it as a text string, and calculates SHA-256 to derive the +root. For seed string `"0"`, the derived lowercase diagnostic is +`4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e`. +That digest is output in `/services` and `/global_view`; it is not registration +input or a universal default. + +Use a quoted JSON string: `"0"`, `"00"`, and numeric JSON `0` are different +inputs. The explicit literal `random` is supported and derives a root from that +exact text, but an unset environment variable makes vLLM choose unregistered +random bytes and is unsupported. vLLM `--seed` controls model and sampling +randomness; it does not select the prefix-cache root. Request `cache_salt` is +sent only to `/query`, not registration. See [how query blocks are hashed](./conductor-architecture-design.md#how-token-blocks-become-lookup-values) +for the exact rule. + +Launch every compatible vLLM process with the same explicit environment text +and canonical-CBOR prefix-hash algorithm. For the seed-zero examples: + +```bash +PYTHONHASHSEED=0 vllm serve test-model \ + --enable-prefix-caching \ + --prefix-caching-hash-algo sha256_cbor +``` + +Start Conductor with an absolute config path and the desired log level: + +```bash +export CONDUCTOR_CONFIG_PATH=/absolute/path/to/conductor_config.json +export CONDUCTOR_LOG_LEVEL=INFO +./build/mooncake-conductor/mooncake_conductor +``` + +Successful startup logs include `HTTP server listening port=13333` and the +static subscription success/failure counts. Static subscriptions start at the +same time, so the order of entries in the JSON file does not make vLLM start +before Mooncake. + +## Start with vLLM only + +Dynamic registration gives explicit control over startup order. Start +Conductor with this alternative config when you do not want static +subscriptions: + +```json +{ + "http_server_port": 13333, + "kvevent_instance": {} +} +``` + +After starting the binary, register every vLLM rank. This first request +registers rank `0`: + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5557", + "replay_endpoint": "tcp://127.0.0.1:5558", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +Register rank `1` with its own event endpoint and the same engine and hash +settings: + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5567", + "replay_endpoint": "tcp://127.0.0.1:5568", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 1, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +Each successful request returns `registered successfully`. Confirm the engine +context before allowing cache-producing traffic: + +```bash +curl -sS http://127.0.0.1:13333/global_view +``` + +The matching context must show `tenant_id` `default`, `model_name` +`test-model`, empty `lora_name`, `block_size` `16`, `python_hash_seed` `"0"`, +the derived root shown above, and `"engine-a":[0,1]` under `instances`. Also +check `/services`: it must contain both vLLM subscriptions with the same seed +and derived root. For a vLLM-only deployment, you can now start +cache-producing traffic. + +## Add Mooncake + +Keep cache-producing traffic paused while adding the shared pool. Enable and +start the Mooncake Master publisher as described in the +[Mooncake publisher guide](../kv-event/publisher-design.md), then register its +live endpoint: + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:6557", + "replay_endpoint": "", + "type": "Mooncake", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "shared-pool", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +Configure the Mooncake publisher so each event carries the same tenant, model, +LoRA name, and block size as the vLLM context. The `hash_profile` belongs to +the Mooncake registration and must match the profile already bound to that +event context. Use the same context values in the registration as well so +`/services` is easy to compare with the publisher. Mooncake `instance_id` and +`dp_rank` identify the subscription for `/services` and `/unregister`; they do +not create a query instance or override event context. A `stored` event must +also carry an object key and a usable full connector hash; see the +[subscriber compatibility guide](../kv-event/subscriber-guide.md) for the +event checks. + +Confirm that `/services` contains both vLLM ranks and the Mooncake subscription +before releasing traffic: + +```bash +curl -sS http://127.0.0.1:13333/services +``` + +For static configuration, use the other safe choice: keep cache-producing +traffic paused until `/global_view` shows the expected vLLM context, engine, +ranks, configured seed, and derived root and `/services` shows every expected +subscription with that same resolved profile. Static subscriptions start +concurrently, regardless of JSON order. + +These checks have a narrow meaning. `/services` proves that Conductor accepted +the registration and started its local subscription client; it does not prove +that the remote ZeroMQ publisher has sent an event. `/global_view` proves that +the vLLM context and ranks were registered; it does not recover Mooncake +objects cached before Conductor connected. The current Mooncake publisher does +not resend that earlier state. + +## Check it works + +Inspect the active subscriptions: + +```bash +curl -sS http://127.0.0.1:13333/services +``` + +For the dynamic example, success means `count` is `3`, the two `vLLM` entries +have `InstanceID` `engine-a` with `DPRank` `0` and `1`, and the `Mooncake` +entry has `InstanceID` `shared-pool`. All three `HashProfile` values must +match, including `python_hash_seed` `"0"` and its derived `root_digest`. + +Inspect the cache-sharing groups and registered inference ranks: + +```bash +curl -sS http://127.0.0.1:13333/global_view +``` + +Success means one matching context shows `"engine-a":[0,1]`. Mooncake does not +appear as another inference instance. After live events arrive, `prefix_count` +can increase, but that count alone does not show which cache location supplied +each block. + +## Query + +Send token IDs produced by the tokenizer for the registered model. This +example contains one complete 16-token block: + +```bash +curl -sS -X POST http://127.0.0.1:13333/query \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "test-model", + "lora_name": "", + "tenant_id": "default", + "block_size": 16, + "token_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] + }' +``` + +A successful response has an `instances.engine-a` object with DP rank keys +`"0"` and `"1"`. Hit values may be zero until matching live events arrive. +Only complete blocks are considered. If the producer hashes requests with a +cache salt, add the matching string `cache_salt` to this query; do not add it +to registrations. + +See the [HTTP API reference](./indexer-api-design.md#post-query) for the exact +response fields and the [architecture guide](./conductor-architecture-design.md#what-query-fields-mean) +for how Conductor combines GPU, CPU, and Disk availability. + +## Unregister + +Unregister the exact service key before replacing a publisher. The key is +formed from `instance_id`, normalized `tenant_id`, and `dp_rank`. + +Remove the Mooncake subscription: + +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"shared-pool","tenant_id":"default","dp_rank":0}' +``` + +Remove each vLLM rank separately: + +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","tenant_id":"default","dp_rank":1}' + +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","tenant_id":"default","dp_rank":0}' +``` + +Each successful response contains `unregistered successfully` and the exact +removed service key, for example `engine-a|default|0`. Check `/services` to +confirm that the subscriptions are absent. Unregistering one vLLM rank removes +only that rank's GPU information; unregistering Mooncake removes only objects +reported by that Mooncake endpoint. + +## Fix setup problems + +| Symptom | Check | Action | +|---|---|---| +| HTTP requests cannot connect, or the log shows port `0` | Confirm that `CONDUCTOR_CONFIG_PATH` names the file you edited and that it contains numeric `http_server_port`. | Set an absolute config path, add an explicit nonzero port, and restart Conductor. | +| A source is absent from `/services` | Read the registration response and Conductor log for an unsupported type, nonzero `cache_group`, invalid hash profile, duplicate endpoint, or conflicting service key. | Use only `vLLM` or `Mooncake`, cache group `0` or omission, a unique endpoint, and the exact supported seed-based hash fields. | +| `/services` lists a source but hits remain zero | Check whether the publisher emitted events after Conductor connected. `/services` is not proof of event delivery. | Verify the publisher status, keep traffic paused during setup, and start registration before new cache-producing traffic. Earlier Mooncake events are not resent. | +| Mooncake events do not add CPU or Disk availability | Compare `/global_view` with the tenant, model, LoRA name, and block size carried by Mooncake events; compare the registered hash profiles; check that stored events include object and full connector-hash data. | Register vLLM first, make the event context and hash profile match, and correct the Mooncake publisher/key setup. | +| `/query` returns an empty `instances` object or shorter hits than expected | Compare `model`, `tenant_id`, `lora_name`, `block_size`, optional `instance_id`, token IDs, and `cache_salt` with the producer. Check for a trailing partial block. | Query the exact registered four-field group, use the producer's salt rule, and send enough tokens for complete blocks. | +| An HTTP request returns `400` | Read the JSON `reason`, `field`, and optional `index`; malformed register or unregister JSON instead returns plain text. | Remove unsupported fields and correct the named value using the [API field tables](./indexer-api-design.md). | diff --git a/docs/source/design/kv-event/index.md b/docs/source/design/kv-event/index.md new file mode 100644 index 0000000000..42a65427c5 --- /dev/null +++ b/docs/source/design/kv-event/index.md @@ -0,0 +1,42 @@ +# KV Events + +[中文](../../zh/design/kv-event/index.md) + +KV Events tell Conductor where key-value (KV) cache blocks are currently +available. In this repository, vLLM reports GPU blocks owned by one inference +engine and rank, while Mooncake reports objects available from a shared CPU or +Disk pool. Use this page to choose the guide for the side you are configuring. + +## Choose an event source + +| Source | What it reports | Where to continue | +|---|---|---| +| vLLM | GPU cache for one registered engine and data-parallel (DP) rank. | Use the [Conductor subscriber guide](./subscriber-guide.md) to see the accepted vLLM message fields. | +| Mooncake Master | CPU or Disk objects shared by compatible registered engines. | Use the [Mooncake publisher guide](./publisher-design.md) to enable and inspect the publisher. | + +The registered source `type`, not the ZeroMQ (ZMQ) topic text, decides how +Conductor reads a message. The subscriber guide gives the detailed comparison +between `vLLM` and `Mooncake`. + +## Publish Mooncake events + +The [Mooncake KV Event publisher guide](./publisher-design.md) covers the build +option, Master settings, status counters, exact three-frame messages, +connector-key parsing, and what happens when events are missed. + +## Connect Conductor + +The [Conductor KV Event subscriber guide](./subscriber-guide.md) explains what +each registered source type may send, how hashes become lookup values, which +events change or clear cache information, and how unregister cleanup works. + +For a complete startup sequence with registration and HTTP checks, follow the +[Conductor usage guide](../conductor/usage.md). + +```{toctree} +:maxdepth: 1 +:hidden: + +publisher-design +subscriber-guide +``` diff --git a/docs/source/design/kv-event/publisher-design.md b/docs/source/design/kv-event/publisher-design.md index 66940640dd..05537b3a97 100644 --- a/docs/source/design/kv-event/publisher-design.md +++ b/docs/source/design/kv-event/publisher-design.md @@ -1,144 +1,292 @@ -# KV Event Publisher Design +# Publish Mooncake KV Events -## Goals +[中文](../../zh/design/kv-event/publisher-design.md) -The master publishes logical cache availability for external indexers while -keeping physical replica management internal. The implementation uses the -existing RFC #1527 map protocol and supports `stored`, `removed`, and -`cleared`. It does not include an indexer, replay service, or Conductor. +Mooncake Master can publish live key-value (KV) cache changes for Conductor and +other subscribers. This page shows how to enable the publisher, check whether +it is sending, and read the exact MessagePack fields it produces. It also +explains the object-key and delivery limits that affect Conductor. -## Transport +## Enable and check the publisher -The master binds a ZeroMQ PUB socket. Each multipart message is: +From the repository root, build `mooncake_master` with KV Events enabled: -1. an empty topic frame; -2. an unsigned 64-bit big-endian transport sequence; -3. a msgpack payload `[timestamp_ms, [event_maps], dp_rank]`. +```bash +cmake -S . -B build -DENABLE_KV_EVENTS=ON +cmake --build build --target mooncake_master -j +``` + +Start the Master with a usable backend name, model fallback, block size, and +ZeroMQ (ZMQ) bind address. This command uses the admin HTTP port `9003`: + +```bash +./build/mooncake-store/src/mooncake_master \ + --enable_kv_events=true \ + --kv_events_bind_endpoint=tcp://0.0.0.0:5557 \ + --kv_events_backend_id=pool-a \ + --kv_events_model_name=Qwen/Qwen2.5-7B-Instruct \ + --kv_events_block_size=16 \ + --kv_events_emit_object_key=true \ + --metrics_port=9003 +``` + +A successful start logs a line containing: + +```text +kv_events publisher enabled on tcp://0.0.0.0:5557 backend_id=pool-a +``` + +Then read the publisher status: + +```bash +curl -s http://127.0.0.1:9003/kv_events/status +``` + +Before cache traffic begins, a healthy publisher returns the following shape. +The counters can already be nonzero if operations have run: + +```json +{"enabled":true,"published_batches":0,"published_events":0,"dropped_events":0,"skipped_unparsed_keys":0,"invalid_event_hashes":0} +``` + +Check that `enabled` is `true`. After a cache operation, check that +`published_events` and normally `published_batches` increase. A subscriber is +not required for those counters to increase, so this check proves that the +Master sent messages to its ZMQ publisher (PUB) socket; it does not prove that +Conductor received them. + +## Understand the build requirement + +`-DENABLE_KV_EVENTS=ON` requires the libzmq headers and library. CMake stops +with an error if they cannot be found. A build made without this option keeps +the public Store calls available but uses a publisher stub; runtime flags +cannot turn that stub into a working publisher, and status reports +`"enabled":false`. + +The publisher also changes `enabled` to `false` if +`kv_events_bind_endpoint` or `kv_events_backend_id` is empty, or if creating or +binding the ZMQ socket fails. Check the Master log when status is false. + +## Choose Master settings + +The settings can be supplied in the Master config file or as command-line +flags. Command-line values that are explicitly set override values loaded with +`--config_path`. + +| Setting | Default | When it matters | +|---|---:|---| +| `enable_kv_events` | `false` | Must be `true` to create the publisher. | +| `kv_events_bind_endpoint` | empty | ZMQ PUB address bound by the Master, for example `tcp://0.0.0.0:5557`. Register a routable address such as `tcp://master-host:5557` in Conductor, not `0.0.0.0`. | +| `kv_events_backend_id` | empty | Required non-empty name for the Mooncake backend that reported the objects. It limits Remove and Clear cleanup in Conductor. | +| `kv_events_model_name` | empty | Fallback model for unrecognized keys and `cleared`. A model parsed from a connector key takes precedence. Conductor needs a non-empty model for `stored`. | +| `kv_events_block_size` | `0` | Fixed token count written into events. `0` is sent as `nil`; Conductor rejects a `stored` event whose block size is absent or non-positive. | +| `kv_events_lora_name` | empty | Fixed Low-Rank Adaptation (LoRA) name. Empty means the base model and is sent as `nil`. | +| `kv_events_additional_salt` | empty | Fixed string, sent as `nil` when empty. Conductor reads it but does not use it for cache sharing or lookup. | +| `kv_events_dp_rank` | `0` | Unsigned data-parallel (DP) value written into each event and the batch. Conductor keeps Mooncake DP values only for troubleshooting; they are not an inference-engine rank. | +| `kv_events_tenant_id` | `default` | Retained for config compatibility. The event tenant comes from each Store operation; an empty operation tenant becomes `default`. | +| `kv_events_emit_object_key` | `true` | Adds `object_key` to object events. Keep this `true` for Conductor, which needs the key to remove an object exactly. | +| `kv_events_emit_legacy_compat` | `true` | Adds the matching legacy `type`, `block_hashes`, and stored `parent_block_hash` fields. Conductor accepts them when they agree with the primary fields. | +| `kv_events_queue_capacity` | `65536` | Maximum pending events in the publisher queue. A positive value bounds the queue; `0` removes that bound. See [Plan for lost events](#plan-for-lost-events). | + +The `kv_events_queue_capacity` command-line help currently says the setting is +ignored, but the publisher implementation does enforce it. The table above +describes the current runtime behavior. + +## Read publisher status + +`GET /kv_events/status` is served on the Master's admin port, configured by +`metrics_port` (default `9003`). Its fields are: + +| Field | Meaning | +|---|---| +| `enabled` | The current Master has a compiled, configured, and successfully bound publisher. | +| `published_batches` | Multipart messages for which all three ZMQ sends returned success. | +| `published_events` | Events contained in those successfully sent batches. | +| `dropped_events` | Events removed because the pending queue was full, plus events in a batch whose ZMQ send failed. | +| `skipped_unparsed_keys` | Events with no sequence hash that were skipped because no object key could be sent, plus emitted events whose key had no recognized block-hash segment. The emitted form has `object_key` but no connector hash, so Conductor rejects its `stored` event. | +| `invalid_event_hashes` | Recognized connector keys whose block-hash text could not be converted to an unsigned 64-bit value. The event can still be sent with its object key, but without `connector_block_hash`. | -Publishing is asynchronous. A bounded in-process queue drops the oldest event -when full and reserves a sequence gap so subscribers can detect loss. Relevant -master flags are: +Zero counters do not prove that a subscriber is connected. Likewise, the PUB +socket can lose a message after a successful local send, so these counters are +not proof that Conductor received the events. -- `enable_kv_events` -- `kv_events_bind_endpoint` -- `kv_events_backend_id` -- `kv_events_model_name` -- `kv_events_block_size` -- `kv_events_additional_salt` -- `kv_events_lora_name` -- `kv_events_dp_rank` -- `kv_events_emit_object_key` -- `kv_events_emit_legacy_compat` -- `kv_events_queue_capacity` +## Understand the three ZMQ frames -The feature is compiled only when `ENABLE_KV_EVENTS=ON`; public client APIs -remain available and become no-ops for event metadata in builds without ZMQ. +Every publication is one multipart ZMQ message with exactly three frames: -## Object and medium state +| Frame | Exact content | +|---:|---| +| 1 | Empty topic frame. | +| 2 | One unsigned 64-bit transport sequence number in big-endian byte order. | +| 3 | One MessagePack payload: `[timestamp_ms, [event_maps], dp_rank]`. | -Event identity is based on `backend_id`, `tenant_id`, and the block identity -(`seq_hashes` and/or `object_key`). The `medium` field is one string, normally -`cpu` or `disk`. If a block is present in both tiers, the publisher emits one -event for each tier. +The publisher groups at most 64 pending events into one payload. All events in +that payload use the same signed 64-bit Unix timestamp in milliseconds. +`dp_rank` is the configured unsigned 32-bit value. The publisher assigns one +sequence value to each batch it tries to send and reserves extra values when +queue entries are dropped. It starts at `1` for each publisher process, so +queue loss deliberately leaves a gap. -Replica topology is normalized to medium availability: +The payload items are: -- the first completed replica on a medium emits `stored`; -- removing the last completed replica on a medium emits `removed`; -- changing the number or location of replicas within an available medium emits - no event; -- a successful Put or Upsert commit emits `stored` for every current medium. +| Item | MessagePack form | Meaning | +|---:|---|---| +| `timestamp_ms` | non-negative integer | Batch creation time in Unix milliseconds. | +| `event_maps` | array of maps | Events in the order the publisher dequeued them. | +| `dp_rank` | unsigned integer | Copy of `kv_events_dp_rank`; kept only for Mooncake troubleshooting in Conductor. | -The publisher keeps a compact per-process state entry for each published -object. It stores the current medium set and metadata parsed from the Mooncake -object key. A temporary zero-medium state during Upsert retains this compact -context; state is released when the object is deleted or its tenant is cleared. +## Read common event fields -## Connector key metadata +Every `stored`, `removed`, and `cleared` map contains these primary fields: -The publisher derives event metadata from keys already generated by the vLLM -connectors. No event-specific argument is added to the Mooncake client API. +| Field | MessagePack form | Source and use | +|---|---|---| +| `event_id` | unsigned 64-bit integer | Increasing event number within one publisher process. It restarts with the process. | +| `timestamp` | signed 64-bit integer | Same millisecond value as the outer `timestamp_ms`. | +| `event_type` | string | Exactly `stored`, `removed`, or `cleared`. | +| `model_name` | string or `nil` | Connector-key model when parsed; otherwise `kv_events_model_name`. | +| `block_size` | unsigned integer or `nil` | `kv_events_block_size`; `0` becomes `nil`. | +| `additional_salt` | string or `nil` | `kv_events_additional_salt`; empty becomes `nil`. | +| `lora_name` | string or `nil` | `kv_events_lora_name`; empty becomes `nil`. | +| `tenant_id` | string | Tenant on the Store operation; empty becomes `default`. | +| `backend_id` | string | `kv_events_backend_id`. | +| `medium` | string or `nil` | Object location for `stored` and `removed`; `nil` for `cleared`. | +| `dp_rank` | unsigned integer | `kv_events_dp_rank`. | -The supported vLLM key formats are: +When `kv_events_emit_legacy_compat=true`, every map also has `type`: +`BlockStored`, `BlockRemoved`, or `AllBlocksCleared`, matching `event_type`. + +`stored` and `removed` maps also contain: + +| Field | MessagePack form | Source and use | +|---|---|---| +| `group_id` | string or `nil` | Parsed connector `group:N`, if present; otherwise the Store object's group string. Empty becomes `nil`. | +| `seq_hashes` | array of zero or one unsigned integer | Low 64 bits parsed from the object key. An unparsable key produces an empty array. | +| `base_block_idx` | `nil` | Connector keys do not record the block's depth in a token chain. | +| `object_key` | string, conditionally present | Complete Mooncake key, emitted only when `kv_events_emit_object_key=true`. | +| `block_hashes` | array, conditionally present | Legacy copy of `seq_hashes`, emitted only when compatibility fields are enabled. | + +A `stored` map additionally contains `parent_hash` and `token_ids`, both +`nil`, because current connector keys do not contain either value. With legacy +fields enabled it also contains `parent_block_hash=nil`. A `removed` map does +not contain these stored-only fields. A `cleared` map contains no group, hash, +object, or topology fields. + +When a connector key is recognized, the publisher also adds the fields that +the key actually provides: + +| Field | When present | +|---|---| +| `connector_block_hash` | The complete block-hash text was valid hexadecimal and produced `seq_hashes[0]`. | +| `cache_prefix` | Text appeared before the connector model name. | +| `tp_rank` | A vLLM tensor-parallel rank was parsed. | +| `head_or_tp_rank` | A vLLM Ascend head or tensor-parallel rank was parsed. | +| `pcp_rank` | A prefill context-parallel rank was parsed. | +| `dcp_rank` | A decode context-parallel rank was parsed. It is not `dp_rank`. | +| `pp_rank` | A pipeline-parallel rank was parsed. | +| `layer_id` | An Ascend layerwise key contained a layer number. | + +## Interpret `stored`, `removed`, and `cleared` + +The three event names describe logical availability, not the number or address +of physical replicas: + +| Event | What the Mooncake publisher means | Fields current Conductor needs | +|---|---|---| +| `stored` | This object is available from the named `cpu` or `disk` medium. A repeated successful Put or Upsert commit may refresh the same availability. | `backend_id`, non-empty `tenant_id` and `model_name`, positive `block_size`, supported medium and group, `object_key`, and a usable full `connector_block_hash`. `lora_name` may be empty. `seq_hashes` may be empty; if it has a value, it must match the full hash. | +| `removed` | This object is no longer available from the named medium. The other medium can remain available. | `backend_id`, supported medium and group, and `object_key`. The full hash may be absent because Conductor follows the object record saved by the earlier `stored` event. | +| `cleared` | All objects for this publisher's `backend_id` and the event's `tenant_id` have been cleared. It applies to both CPU and Disk and uses `medium=nil`. | Non-empty `backend_id` and `tenant_id`, with no object fields. | + +The publisher can still send a `stored` event for a key it cannot fully parse +when object-key emission is enabled. Such an event has no usable +`connector_block_hash`; current Conductor logs a rejection and does not add it +to the shared-cache index. + +## Use connector keys Conductor can match + +Mooncake does not add a KV Event parameter to Store client calls. It reads +metadata from keys made by the vLLM connectors. The accepted vLLM forms are: ```text [cache_prefix@]model_name@tp_rank:N@pcpN@dcpN@pp_rank:N@group:N@block_hash_hex [cache_prefix@]model_name@tp_rank:N@pcpN@dcpN@pp_rank:N@block_hash_hex ``` -The second form supports vLLM versions predating the group field. Supported -vLLM Ascend formats are: +The second form covers connector versions from before `group:N`. The accepted +vLLM Ascend forms are: ```text model_name@pcpN@dcpN@head_or_tp_rank:N@pp_rank:N@block_hash_hex model_name@pcpN@dcpN@head_or_tp_rank:N@block_hash_hex@layer_id ``` -For recognized keys, `model_name` populates the event envelope and vLLM's -`group:N` populates `group_id`. For an unrecognized key or a `cleared` event, -`kv_events_model_name` supplies the fixed publisher model. A key-derived model -takes precedence. Mooncake also publishes the complete hexadecimal digest as -`connector_block_hash`; this preserves leading zeros and 128/256-bit hashes for -consumers that perform exact connector-hash matching. To match vLLM's default -KV Event conversion, `seq_hashes[0]` and the legacy `block_hashes[0]` remain the -low 64 bits (the rightmost 16 hex characters). The complete key remains -available as `object_key` when enabled. - -The optional vLLM `cache_prefix` is published separately as `cache_prefix`. -It is a Mooncake key namespace used to isolate deployments and is not treated -as request `additional_salt`. Parallel namespace fields encoded in the key are -also exposed as available: `tp_rank` or `head_or_tp_rank`, `pcp_rank`, -`dcp_rank`, `pp_rank`, and layerwise `layer_id`. These fields describe the -connector key and are never mapped to `dp_rank`. - -One Publisher serves one fixed model, block-size, LoRA, additional-salt, and -data-parallel context. The publisher therefore fills `block_size`, -`additional_salt`, `lora_name`, and `dp_rank` from the corresponding master -configuration. `block_size=0` remains nil, and empty salt or LoRA names are -encoded as nil. The configured `dp_rank` is emitted both in each event envelope -and in the batch trailer. Per-object `tenant_id` still comes from the Store -operation rather than the global tenant config. - -Connector key `dcpN` is decode-context parallelism and is never mapped to -`dp_rank`. Connector keys and fixed publisher config still do not provide token -IDs, parent hash, or block depth, so `token_ids`, `parent_hash`, and -`base_block_idx` remain nil. - -Connector keys use unescaped `@` separators. When vLLM's optional -`cache_prefix` is present, the segment immediately before `tp_rank` is treated -as the model name. Unknown or malformed formats are not rejected by Store; -they continue to be published through `object_key`, while `seq_hashes` remains -empty. - -Layerwise keys identify one layer but do not encode the total layer count. A -subscriber requiring whole-block availability must aggregate `layer_id` values -using model configuration; the publisher cannot infer completeness from one -key. The same rule applies to parallel namespace fields: a subscriber must use -registered TP/PCP/DCP/PP topology to decide when all required shards are -available. Connector keys also do not encode block depth, so `base_block_idx` -remains nil rather than assuming the object is a root block. - -## Publication points - -| Operation | Event behavior | +The separator is an unescaped `@`. In the first two forms, the segment just +before `tp_rank` is the model; earlier segments become `cache_prefix`. +Malformed or unknown keys are not rejected by Mooncake Store. They can still +be emitted as `object_key`, but they do not supply the full connector hash +that Conductor needs for `stored`. + +For Conductor, `block_hash_hex` must be valid hexadecimal representing at +least eight bytes: after an optional `0x` prefix it must have at least 16 +characters and an even length. The publisher preserves the full text in +`connector_block_hash` and converts the rightmost 16 hex characters to +`seq_hashes[0]`. Conductor reads those final eight bytes in big-endian order as +the 64-bit lookup value used by `/query`. + +The full hash still matters. Conductor saves the full hash and object key so +that two different objects with the same final eight bytes remain separately +removable. A query uses only the 64-bit value and therefore reports a possible +cache hit when either object is present. See the [subscriber hash +rules](./subscriber-guide.md#convert-incoming-hashes) for accepted encodings +and mismatch handling. + +`cache_prefix`, rank fields, and `layer_id` describe the connector namespace. +The publisher cannot infer how many layers or parallel parts make a complete +block, and current Conductor does not perform that completeness check. + +## Understand CPU and Disk changes + +The Master reduces replica metadata to two availability values: is the object +available in CPU, and is it available on Disk. It emits: + +| Store operation | Event result | |---|---| -| Put/BatchPut commit | `stored` for all completed media | -| Upsert of an existing block | `removed` when the old value becomes unreadable, then `stored` for all completed media on commit | -| Copy/Move completion | Medium availability delta | -| Offload/promotion completion | `stored` when the target medium first appears | -| Replica clear or eviction | `removed` when the last replica on a medium disappears | -| Stale handle/client cleanup | Medium availability delta | -| Remove/BatchRemove/regex remove | `removed` for every available medium | -| RemoveAll | Per-object `removed`, then tenant `cleared` only when empty | -| Failed uncommitted new Put | No event | - -`cleared` uses `medium=nil` and clears all media for the specified -`backend_id + tenant_id`. - -## Limitations - -The publisher is PUB-only. It does not replay missed events, publish a startup -snapshot, or persist its compact context cache. Subscribers must detect -transport sequence gaps and recover through their own reconciliation path. -Objects restored before the publisher starts can still publish tenant, backend, -fixed publisher context, object key, and any metadata parseable from that key. +| Successful Put or Upsert commit | One `stored` for every currently available medium. | +| Upsert while the old value becomes unreadable | `removed` for the old availability, followed by commit-time `stored` events. | +| Copy, move, offload, promotion, eviction, or replica cleanup | `stored` when a medium first becomes available; `removed` when its last usable replica disappears. Changing replica count or location while the medium remains available produces no availability change. | +| Remove, BatchRemove, or regex removal | One `removed` for each medium where the object was available. | +| RemoveAll | Per-object `removed` events, followed by `cleared` only when the tenant has no remaining objects. | +| Failed uncommitted new Put | No event. | + +If one object is available in both CPU and Disk, the publisher sends a +separate event for each medium. + +## Plan for lost events + +Publishing is asynchronous. A positive `kv_events_queue_capacity` bounds the +pending queue. When that queue is full, the publisher removes the oldest +pending event, increments `dropped_events`, and reserves a transport sequence +number so subscribers can see a gap. A ZMQ send failure also increments +`dropped_events` for the affected events. + +The publisher is PUB-only: it has no replay endpoint and cannot resend a +missed event. It also does not send a list of objects that were already cached +when Conductor connects. Its compact object-to-medium tracking is kept only in +the publisher process and is not persisted. Later Store changes can generate +new events for an existing object, but they do not reconstruct a complete +startup view. + +Conductor warns when the transport sequence jumps and keeps its current cache +records; the warning alone does not repair the missing change. Keep +cache-producing traffic paused until the required Conductor registrations are +visible. The complete startup checklist is in the [subscriber +guide](./subscriber-guide.md#check-mooncake-to-conductor-setup). + +## Maintainer source note + +The message builder and queue are in +`mooncake-store/src/kv_event/kv_event_publisher.cpp`; connector-key parsing is +in `mooncake-store/include/kv_event/key_util.h`. Master flags and config loading +are in `mooncake-store/src/master.cpp`, and `GET /kv_events/status` is in +`mooncake-store/src/master_admin_service.cpp`. The matching fixtures are in +`mooncake-store/tests/kv_event_publisher_test.cpp`. diff --git a/docs/source/design/kv-event/subscriber-guide.md b/docs/source/design/kv-event/subscriber-guide.md index 07ecc253bb..b10b368c86 100644 --- a/docs/source/design/kv-event/subscriber-guide.md +++ b/docs/source/design/kv-event/subscriber-guide.md @@ -1,76 +1,282 @@ -# KV Event Subscriber Guide +# Connect KV Event Sources to Conductor -## Event envelope +[中文](../../zh/design/kv-event/subscriber-guide.md) -All event maps contain: +Conductor reads two current key-value (KV) event formats: vLLM engine events +and Mooncake shared-pool events. This page helps you choose the correct +registered source type, satisfy the fields Conductor checks, and understand +exactly what Store, Remove, Clear, and unregister change. It also describes +hash conversion and the limits around missed events. -| Field | Type | +## Pick the registered source type + +Register each endpoint with exactly one case-sensitive `type`: + +- use `"vLLM"` for an inference engine that reports its GPU cache; +- use `"Mooncake"` for a Mooncake Master that reports shared CPU or Disk + objects. + +The recorded type decides which MessagePack event format Conductor reads. +Topic text does not change that choice, even when both publishers use an empty +topic. The receiving endpoint also identifies the source for cleanup and logs. +See the [HTTP API reference](../conductor/indexer-api-design.md) for registration +fields and examples. + +## Compare vLLM and Mooncake + +| Question | `vLLM` source | `Mooncake` source | +|---|---|---| +| Registered `type` | Exactly `vLLM`. | Exactly `Mooncake`. | +| Event names | `BlockStored`, `BlockRemoved`, `AllBlocksCleared` in `type`. | `stored`, `removed`, `cleared` in `event_type`; matching legacy `type` is optional. | +| Batch timestamp | Finite MessagePack float in seconds. Events have no required timestamp field. | Non-negative MessagePack integer in milliseconds. Every event has the same integer in `timestamp`. | +| Accepted cache location | `GPU`, case-insensitive. Other values are logged and ignored. | `CPU` or `Disk`, case-insensitive. Other values are logged and ignored. | +| Tenant, model, LoRA, block size | Trusted registration supplies the cache-sharing fields. Event block size and Low-Rank Adaptation (LoRA) name are checked as assertions for Store. | A Store event supplies `tenant_id`, `model_name`, `lora_name`, and `block_size`. They must match an existing vLLM cache-sharing context and the registered hash profile. | +| Data-parallel rank | Trusted registration supplies the engine rank. A non-`nil` batch rank must match it or the complete batch is rejected. | Batch and event `dp_rank` values are kept only for troubleshooting. They do not create a GPU record or a query instance. | +| Group | `group_idx` may be absent, `nil`, or `0`. | `group_id` may be `nil`, empty, or decimal string `"0"`. | +| Object fields | Uses `block_hashes`; it has no Mooncake object key. | Store needs `object_key` and `connector_block_hash`; Remove needs `object_key`; Clear carries neither. | +| `/query` `instances` | The registered engine appears under its `instance_id`, including its registered ranks. | The Mooncake source is not listed as an instance. Its CPU or Disk information appears under every compatible vLLM engine. | + +The four fields that decide whether CPU or Disk information is compatible with +an engine are `tenant_id`, model, `lora_name`, and `block_size`. The +[architecture page](../conductor/conductor-architecture-design.md#what-can-share-cache) +explains how they affect query results. + +## Match the prefix-hash profile + +KV events carry neither `PYTHONHASHSEED` nor the complete first-parent digest, +so registration is a trusted deployment assertion. Each vLLM and Mooncake +registration for a compatible context must supply the same exact +`python_hash_seed` string. Conductor canonical-CBOR encodes that text and +calculates SHA-256 to derive the `root_digest` used as the first parent; callers +do not register the digest. + +The seed must be the literal `random` or ASCII decimal text in +`0..4294967295`. Exact text matters: `"0"` and `"00"` derive different roots. +An unset `PYTHONHASHSEED` is unsupported because vLLM then selects random root +bytes that Conductor cannot reproduce. The explicit text `random` is supported +and is hashed as that exact string. vLLM `--seed` only controls model and +sampling randomness and is unrelated to this prefix-cache identity. + +For the seed-zero profile, start every compatible vLLM process with: + +```bash +PYTHONHASHSEED=0 vllm serve test-model \ + --enable-prefix-caching \ + --prefix-caching-hash-algo sha256_cbor +``` + +Register `python_hash_seed` as `"0"`. Conductor reports the resulting +`root_digest` +`4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e` +through `/services` and `/global_view` for comparison. + +## Meet the vLLM event requirements + +The vLLM payload is exactly +`[timestamp_seconds, [event_maps], data_parallel_rank]`. The last item may be a +non-negative integer or `nil`. Each event map follows these rules: + +| Event | Recognized fields and checks | Index change | +|---|---|---| +| `BlockStored` | Requires `block_hashes`, `parent_block_hash`, `token_ids`, `block_size`, `lora_id`, `medium`, and `lora_name`. `lora_id` must be `nil`; `block_size` and `lora_name` must agree with registration; `medium` must be GPU. Optional `group_idx` must be `0` or `nil`. | Adds the supplied hashes to the registered endpoint, engine, and data-parallel (DP) rank's GPU information. | +| `BlockRemoved` | Requires `block_hashes` and `medium`; optional `group_idx` must be `0` or `nil`. It must not contain recognized stored-only fields. | Removes only that registered endpoint, engine, and rank's GPU records for the supplied hashes. | +| `AllBlocksCleared` | Carries only `type` among recognized vLLM fields. | Removes all GPU information from that registered endpoint, engine, and rank. It preserves other engines and all shared CPU/Disk information. | + +`block_hashes` is an array whose items are unsigned integers or MessagePack +binary strings. Parent hashes and token IDs are read but do +not replace or rehash `block_hashes`. + +`BlockStored` also recognizes optional `extra_keys`, `kv_cache_spec_kind`, and +`kv_cache_spec_sliding_window`. If present, `extra_keys` must be `nil` or have +one `nil`/array entry per block hash. Conductor accepts only +`kv_cache_spec_kind="full_attention"` and no non-`nil` sliding-window value. +Unknown map keys are ignored; duplicate or wrongly typed recognized keys make +only that event invalid. + +## Meet the Mooncake event requirements + +The Mooncake payload is exactly +`[timestamp_ms, [event_maps], data_parallel_rank]`. The last item may be a +non-negative integer or `nil`. Every event map requires these common fields: + +| Field | Accepted form | +|---|---| +| `event_id` | Unsigned integer. It is recorded in logs, not used to sort or skip the event. | +| `timestamp` | Signed integer equal to the non-negative outer `timestamp_ms`. | +| `event_type` | `stored`, `removed`, or `cleared`. | +| `model_name` | String or `nil`. Store requires a non-empty resulting model. | +| `block_size` | Signed integer or `nil`. Store requires a positive value. | +| `additional_salt` | String or `nil`. Decoded but not used in the cache-sharing fields or hash lookup. | +| `lora_name` | String or `nil`; `nil` means the base model. | +| `tenant_id` | String. Store and Clear need a non-empty tenant for a useful mutation. | +| `backend_id` | String; Store, Remove, and Clear require it to be non-empty. | +| `medium` | String or `nil`; Store and Remove must name CPU or Disk. | +| `dp_rank` | Non-negative signed integer kept only for troubleshooting. | + +An optional legacy `type` must agree with `event_type`. The event-specific +rules are: + +| Event | Fields required by the decoder | Additional fields needed to change the index | +|---|---|---| +| `stored` | `group_id`, `seq_hashes`, `base_block_idx`, `parent_hash`, and `token_ids`. The last three may be `nil`; `seq_hashes` may be `[]`. | A supported group and medium, complete context, `object_key`, and a usable full `connector_block_hash`. A present sequence hash must match that full hash. | +| `removed` | `group_id`, `seq_hashes`, and `base_block_idx`; `seq_hashes` may be `[]`. Stored-only parent and token fields are forbidden. | A supported group and medium plus `object_key`. Conductor follows the exact object record saved by Store; optional full or sequence hash assertions must match it. | +| `cleared` | Only the common fields. All recognized object, hash, group, parent, token, and topology fields are forbidden. | Non-empty `backend_id` and `tenant_id`. The event clears matching CPU and Disk object records from this source endpoint. | + +Object events may also contain `cache_prefix`, `tp_rank`, +`head_or_tp_rank`, `pcp_rank`, `dcp_rank`, `pp_rank`, and `layer_id` with the +types documented in the [publisher guide](./publisher-design.md#read-common-event-fields). +If legacy `block_hashes` is present, it must equal `seq_hashes`; if stored +`parent_block_hash` is present, it must equal `parent_hash`. + +## Convert incoming hashes + +Conductor turns both source formats into the same unsigned 64-bit lookup +value. It does not hash an event's token or parent fields again. + +| Incoming hash | Conversion | |---|---| -| `event_id` | `u64` | -| `timestamp` | `i64` milliseconds | -| `event_type` | `stored`, `removed`, or `cleared` | -| `model_name` | string or nil | -| `block_size` | `u32` or nil | -| `additional_salt` | string or nil | -| `lora_name` | string or nil | -| `tenant_id` | string | -| `backend_id` | string | -| `medium` | string or nil | -| `dp_rank` | `u32` | - -Per-object events also include `group_id`, optional `object_key`, and -`seq_hashes`. Recognized connector keys add `connector_block_hash`, optional -`cache_prefix`, and their encoded parallel namespace fields. With legacy -compatibility enabled, `type` and `block_hashes` are also present. - -## Stored - -`stored` adds or refreshes block availability for one medium. Its additional -fields are `base_block_idx`, `parent_hash`, and `token_ids`. The Mooncake master -derives model and hash metadata from connector object keys and supplies the -fixed `model_name` fallback, `block_size`, `additional_salt`, `lora_name`, and -`dp_rank` from publisher configuration. A zero block size and empty optional -strings remain nil. `parent_hash`, `token_ids`, and `base_block_idx` remain nil. -Indexers should use `seq_hashes` and consistent registration metadata for -prefix lookup. - -`connector_block_hash` is the complete hexadecimal connector digest, while -`seq_hashes[0]` is its low 64 bits. Consumers matching full vLLM hashes must use -`connector_block_hash` (or parse `object_key`) and include `cache_prefix` in the -identity namespace. `base_block_idx` is nil because connector keys do not carry -block depth. For Ascend layerwise keys, `layer_id` does not by itself prove that -all layers are available. Likewise, one rank event does not prove that all -required TP/PCP/DCP/PP shards are present; use registered model topology when -building whole-block availability. - -Treat repeated stored events for the same block/backend/medium as idempotent. -Do not infer physical replica count from them. - -An Upsert of an existing Mooncake object publishes `removed` while the old -value is unreadable and `stored` after the replacement commits. Consumers -should not expect a separate update event type. - -## Removed - -`removed` removes availability only for the event's medium. Other media for the -same block may remain valid. Remove the whole block/backend entry only after no -media remain. - -## Cleared - -`cleared` has no block payload. Mooncake publishes it with `medium=nil` and the -fixed configured model context; it means that all media under the event's -`backend_id + tenant_id` are empty. With legacy compatibility enabled its -`type` is `AllBlocksCleared`. - -## Ordering and recovery - -Process transport frames in sequence order and use `event_id` for event order -inside the publisher stream. A transport sequence gap means at least one event -was dropped or missed. The Mooncake publisher has no replay endpoint, so a -subscriber must invalidate the affected backend state or reconcile it through -an external control-plane query. - -Subscribers joining after objects were stored do not receive historical -events. Registration and initial-state recovery are responsibilities of the -consumer deployment. +| vLLM unsigned integer | Already the lookup value; use it directly. | +| vLLM MessagePack binary string | Require at least eight bytes, then read its final eight bytes in big-endian order. | +| Mooncake `connector_block_hash` | Remove an optional `0x`, accept upper- or lower-case hex, require an even length and at least eight bytes, normalize to lower case, then read the final eight decoded bytes in big-endian order. | + +For a Mooncake Store, `seq_hashes` may be empty or contain exactly one value +equal to the lookup value from `connector_block_hash`. A different value or +more than one value rejects that event without changing its saved object +record or the cache index. For Remove, a supplied full hash or sequence value +must match the earlier Store record. + +The complete Mooncake hash and object key remain saved even though `/query` +uses only the final eight bytes. If two objects have different full hashes but +the same final eight bytes, both can provide the same possible query hit; +removing either object leaves the other object's record intact. Query-side +hash generation is documented in [How token blocks become lookup +values](../conductor/conductor-architecture-design.md#how-token-blocks-become-lookup-values). + +## Understand what each event changes + +| Event | Exact scope of the change | +|---|---| +| vLLM `BlockStored` | Adds GPU hashes for the registered source endpoint, `instance_id`, and DP rank in its registered tenant/model/LoRA/block-size context. | +| vLLM `BlockRemoved` | Removes only that engine and rank's listed GPU hashes. Repeating a remove is a no-op. | +| vLLM `AllBlocksCleared` | Removes every GPU hash from that engine and rank. It does not touch another rank, engine, or Mooncake object. | +| Mooncake `stored` | Saves one record identified by source endpoint, `backend_id`, `tenant_id`, `object_key`, and CPU-or-Disk medium. An identical repeat is harmless; a conflicting active record for that same object and medium is rejected. | +| Mooncake `removed` | Follows that exact saved object-and-medium record. An unknown object is a no-op, and removal never searches by the 64-bit value alone. | +| Mooncake `cleared` | Removes saved CPU and Disk records only for the reporting endpoint, `backend_id`, and `tenant_id`. It preserves GPU information and other endpoints, backends, and tenants. | + +Mooncake events in a valid batch run in their received order. A Clear removes +matching records established before it; a later Store in the same batch may +add availability again. Conductor does not sort by `event_id` or +automatically ignore a repeated ID. + +## Handle invalid messages + +Conductor expects exactly three ZeroMQ (ZMQ) frames: topic, an eight-byte big-endian +sequence, and one MessagePack payload. An invalid frame count, sequence frame, +MessagePack value, outer three-item shape, batch timestamp, event-array type, +or batch DP type drops the complete message without dispatching its events. A +vLLM batch whose non-`nil` DP rank conflicts with registration is also rejected +as a whole. + +Once the outer payload is valid, Conductor reads and applies each map +separately. An event with bad fields or values is skipped, while valid events +before and after it remain applied in received order. A later error does not +undo earlier changes. Unknown fields are ignored so a producer may add new +optional data; duplicate recognized fields and wrong recognized-field types +reject that one event. + +Changing the topic cannot make Conductor read one source as the other source +type. Likewise, a payload that resembles the other message format is not tried +again using that format. + +## Clean up a source + +Use `POST /unregister` with the same `instance_id`, `tenant_id`, and `dp_rank` +that identify the registration. Conductor first stops and joins that +subscriber, then removes its contributions before returning success. Events +already queued by the old subscriber cannot repopulate the index after cleanup. + +For vLLM, unregister removes that endpoint/engine/rank's GPU records and rank +registration. For Mooncake, it removes every saved CPU and Disk object record +from that registered endpoint, across all event contexts and backends, while +preserving other Mooncake endpoints and all vLLM GPU records. The Mooncake +registration's `instance_id` and `dp_rank` identify the service for +`/services` and unregister; they do not create a query instance or override +event context. + +## Plan for sequence gaps and reconnects + +Conductor records the last transport sequence received for each subscription. +When a later value jumps forward, Conductor logs a warning and keeps its +current cache records. The gap does not trigger an immediate resend request or +invalidate possibly stale records. + +After a disconnect, Conductor reconnects the live subscription. Only when +`replay_endpoint` was configured and a previous sequence is known does it ask +that endpoint for messages beginning at the next sequence. Sending the request +and receiving a response do not guarantee that every missed cache change is +recovered. An empty replay endpoint is valid and creates a live-only +subscription. + +The current Mooncake publisher has no replay endpoint and sends no startup +list of objects cached before Conductor connected. A Mooncake source should +therefore use an empty `replay_endpoint`, and its initial Conductor view must +not be treated as complete if cache traffic started earlier. Conductor also +does not reject an older/repeated transport sequence or repeated `event_id` as +a duplicate; it processes the events it receives. + +## Check Mooncake-to-Conductor setup + +Use this order before allowing requests that create cache objects: + +1. Build Mooncake Store with `-DENABLE_KV_EVENTS=ON`, enable its publisher, + keep `kv_events_emit_object_key=true`, and confirm + `GET /kv_events/status` reports `"enabled":true`. +2. Register at least one `vLLM` engine first. Its `tenant_id`, `modelname`, + `lora_name`, and `block_size` must equal the values Mooncake Store events + will carry. Start each engine with the explicit `PYTHONHASHSEED` and + `--prefix-caching-hash-algo` settings above. Check that `/global_view` shows + that engine and every expected rank. +3. Use the same complete `hash_profile` for that vLLM context and the + `Mooncake` registration: `strategy`, `algorithm`, exact + `python_hash_seed`, and `index_projection` must all match. Conductor derives + and validates the root; neither publisher's events reveal it. +4. Configure the Mooncake publisher with a non-empty `backend_id`, a usable + model fallback and positive block size, and the matching LoRA name. Remember + that a model parsed from `object_key` overrides the fallback. +5. Make every object key use a recognized connector form with a full valid hex + hash. A Conductor-compatible `stored` event must contain both `object_key` + and `connector_block_hash`. +6. Use no group or group zero, and emit only CPU or Disk availability for the + Mooncake source. Conductor ignores other media and rejects nonzero groups. +7. Register the Mooncake endpoint and check `/services` for all expected vLLM + and Mooncake subscriptions. Verify the configured seed and derived root in + both `/services` and `/global_view`. With static configuration, + subscriptions start concurrently, so keep cache-producing traffic paused + until both checks pass. +8. Allow cache traffic, confirm the Mooncake publisher counters increase, and + query a known token prefix. Shared `cpu` or `disk` counts must appear under + the compatible vLLM instance, not under the Mooncake registration name. + +`/services` shows that Conductor accepted a registration. It does not prove +that the ZMQ publisher delivered an event or that objects created before the +subscription are known. The [Conductor usage guide](../conductor/usage.md) +provides the complete registration and query commands. + +## Know the current completeness limit + +Mooncake connector keys can carry a layer number and tensor-parallel (TP), +prefill context-parallel (PCP), decode context-parallel (DCP), and +pipeline-parallel (PP) ranks. Conductor decodes those fields but currently +does not check whether all required layers or parallel parts are present before +reporting the block as available. It also does not use `cache_prefix` or +Mooncake `additional_salt` as one of the four cache-sharing fields. + +## Maintainer source note + +Message decoding is implemented in +`mooncake-conductor/src/zmq/msg_decoder.cpp`; transport sequence and reconnect +handling are in `mooncake-conductor/src/zmq/zmq_client.cpp`. Event checks and +cleanup are in `mooncake-conductor/src/kvevent/event_handler.cpp` and +`event_manager.cpp`. Current parser and end-to-end fixtures are in +`mooncake-conductor/tests/msg_decoder_test.cpp` and +`event_ingest_integration_test.cpp`. diff --git a/docs/source/zh/design/conductor/conductor-architecture-design.md b/docs/source/zh/design/conductor/conductor-architecture-design.md new file mode 100644 index 0000000000..dbaa7eaeda --- /dev/null +++ b/docs/source/zh/design/conductor/conductor-architecture-design.md @@ -0,0 +1,140 @@ +# Mooncake Conductor 架构 + +[English](../../../design/conductor/conductor-architecture-design.md) + +Mooncake Conductor 读取实时的键值（KV）缓存事件，并在内存中维护一份供路由决策使用的缓存索引。本页说明 vLLM 的 GPU 信息和 Mooncake 的 CPU 或 Disk 信息如何进入这份索引、查询中的 token 如何变成查找值，以及低秩适配（Low-Rank Adaptation，LoRA）如何隔离缓存。这里描述的是当前 C++ 服务，也会说明判断缓存命中时需要注意的限制。 + +## 从事件到查询结果 + +```mermaid +flowchart LR + V["vLLM KV 事件"] --> VR["注册 type 为 vLLM"] + M["Mooncake Master KV 事件"] --> MR["注册 type 为 Mooncake"] + VR --> VD["读取 vLLM 事件字段"] + MR --> MD["读取 Mooncake 事件字段"] + VD --> G["某个引擎及其数据并行 rank 的 GPU 块"] + MD --> S["共享的 CPU 或 Disk 对象"] + G --> I["内存中的缓存索引
租户 + 模型 + LoRA + 块大小"] + S --> I + Q["POST /query
token IDs"] --> H["计算每个完整 token 块的哈希
使用摘要的最后 8 字节"] + H --> I + I --> R["每个已注册 vLLM 引擎
及其数据并行 rank 的结果"] +``` + +注册时填写的 `type` 决定 Conductor 怎样读取消息。topic 文本和消息内容（payload）的结构都不会改变这个选择。对于同一个事件源，Conductor 按收到的先后顺序应用有效事件。 + +vLLM 事件更新已注册引擎及其数据并行（DP）rank 的 GPU 信息。Mooncake 事件更新共享的 CPU 或 Disk 信息。查询会对完整的 token 块计算哈希，按顺序逐块查找；某一项结果遇到第一个缺失块后，便不再继续累计。 + +## 哪些缓存可以共用 + +以下四个字段决定注册、事件和查询是否使用同一组缓存信息： + +| 字段 | 为什么必须一致 | +|---|---| +| `tenant_id` | 隔离不同租户的缓存信息。 | +| 模型名称 | 防止混用不同模型的缓存块。注册使用 `modelname`，查询使用 `model`，查看状态时使用 `model_name`。 | +| `lora_name` | 隔离基础模型和不同 LoRA 适配器。空字符串表示基础模型。 | +| `block_size` | 同时决定一个块包含多少个 token，以及查询计算哈希时每次处理多少个 token。该值必须为正数。 | + +引擎的 `instance_id` 和 DP rank 不会形成新的缓存共享组。它们用来确定哪些 GPU 记录和查询结果属于该引擎。请求中的 `cache_salt` 也不会改变上述四个字段；它改变的是这次查询使用的块哈希链。 + +每个四字段组合只绑定一个已注册的 `hash_profile`。之后为相同组合注册的事件源，必须使用完全相同的策略、算法、准确的 `python_hash_seed` 文本、派生根摘要和查找规则。目前只支持缓存组 `0`，也可以不填写缓存组。 + +## vLLM 的 GPU 信息与 Mooncake 的 CPU/Disk 信息 + +| 注册的事件源 | 接受的缓存位置 | 一个 stored 块表示什么 | 在 `/query` 中出现在哪里 | +|---|---|---|---| +| `vLLM` | GPU，不区分大小写 | 这条注册记录指定的 endpoint、引擎和 DP rank 报告了该块。 | 位于该引擎的 `instances` 结果和对应 DP rank 下。 | +| `Mooncake` | CPU 或 Disk，不区分大小写 | 一个 Mooncake 对象向所有四个缓存共享字段相同的已注册引擎提供该块。 | 作为共享的 `cpu` 或 `disk` 计数，出现在每个兼容的 vLLM 引擎下。 | + +Mooncake 注册项只是订阅名称，不是推理引擎，因此不会在 `/query` 中增加一行结果。即使还没有兼容的 vLLM 引擎，Conductor 也可以先接受这个订阅；但在对应的四字段组合建立前，Mooncake 事件无法加入共享缓存信息。因此，[使用指南](./usage.md)会先注册引擎，再注册共享缓存池。 + +(how-token-blocks-become-lookup-values)= +## token 块如何变成查找值 + +Conductor 每次对一个完整的 token 块计算哈希。一个块产生的完整 32 字节摘要会成为下一个块的上一个摘要（parent）输入；Conductor 不会使用较短的 64 位查找值继续计算哈希链。 + +计算块哈希前，Conductor 会按以下规则解析当前支持的 profile： + +```text +seed_text = `random` 或 0..4294967295 范围内的准确 ASCII 十进制文本 +seed_cbor = canonical-CBOR text(seed_text) +root_digest = lowercase_hex(SHA256(seed_cbor)) +``` + +来源 profile 提供 `python_hash_seed`，而不是 `root_digest`。这段准确文本必须与每个兼容 vLLM 进程中的 `PYTHONHASHSEED` 环境变量相同。Conductor 不会去除空白或规范化文本，不会读取自己的进程环境，也不会从 KV Event 推断种子。因此 `"0"` 和 `"00"` 会得到不同根摘要，数值类型的 JSON `0` 则无效。明确设置的 `"random"` 受支持；未设置 `PYTHONHASHSEED` 不受支持，因为此时 vLLM 使用注册信息无法复现的随机根字节。 + +vLLM 的 `--seed` 控制模型和采样随机数生成器，与前缀缓存的哈希标识无关。LoRA 影响每个块；非空的请求 `cache_salt` 影响第一个块及其所有后继块；Mooncake 的 `additional_salt` 仍只用于诊断。`PYTHONHASHSEED` 是兼容性元数据，不是租户隔离或安全密钥。 + +对于已经解析的哈希配置，每个块按以下步骤处理： + +1. 第一个 parent 是从 `python_hash_seed` 派生的根摘要。后续每个 parent 都是前一个块的完整 SHA-256 摘要。 +2. Conductor 编码一个包含三项的规范化简洁二进制对象表示（Concise Binary Object Representation，CBOR）数组：字节形式的 parent 摘要、该块的有符号 token 整数数组，以及 `null` 或一个有固定顺序的额外字符串数组。 +3. 非空的 `lora_name` 是每个块的第一个额外字符串。非空的查询 `cache_salt` 只放在第一个块，并排在 LoRA 之后。第一个摘要改变后，哈希链中后续的每个摘要也会随之改变。 +4. Conductor 对这些规范化 CBOR 字节计算 SHA-256。 +5. Conductor 按无符号大端序（big-endian）64 位整数读取摘要的最后 8 字节，并用这个整数查找缓存块。 + +末尾不足 `block_size` 个 token 的部分不会计算哈希，也不会计入缓存命中。Mooncake 事件中的 `additional_salt` 会被解析用于诊断，但目前既不会进入四个缓存共享字段，也不会参与这项查找计算。 + +### 仓库中的 golden vector（固定测试值） + +仓库测试数据记录的是 **vLLM 0.22.0 的 `hash_block_tokens` 语义，以及 cbor2 6.1.1 和 `canonical=True`**。其中的示例已解析 profile 记录了 `python_hash_seed` `"0"`： + +```json +{ + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" +} +``` + +规范 CBOR 把文本字符串 `"0"` 编码为十六进制 `6130`，而不是整数零的编码。对这两个字节计算 SHA-256 会得到上面的根摘要，并把它作为第一个块的 parent。 + +当 `block_size` 为 `4`、`lora_name` 为空、没有 `cache_salt`，并且 token IDs 为 `[1, 2, 3, 4, 5, 6, 7, 8]` 时，测试数据给出的结果是： + +| 块 | 完整 SHA-256 摘要 | 最后 8 字节 | 无符号十进制查找值 | +|---|---|---|---| +| 1 | `c9d58ba695280d69b243e1e0df813136ca9196b286fb1a021e0b2e028ef071cb` | `1e0b2e028ef071cb` | `2164874634404590027` | +| 2 | `24125b23e68883b5c2141db2959d48433fe6bde2f26bd914efad121d154ab2d6` | `efad121d154ab2d6` | `17270480062156288726` | + +这组种子和根摘要只是针对上述生成端和库版本的测试向量，不是所有部署的默认值。注册项提供准确的种子声明；Conductor 派生根摘要，并把它作为诊断信息返回。 + +(what-query-fields-mean)= +## 如何理解查询字段 + +所有计数的单位都是 token，而不是块。Conductor 会为每个选中的已注册 vLLM 引擎返回一项结果。 + +| 结果字段 | 含义 | +|---|---| +| `dp` | 每个已注册 DP rank 在该引擎和 rank 上连续命中的 GPU 前缀。rank key 是十进制 JSON 字符串。 | +| `gpu` | 该引擎所有 `dp` 值中的最大值。Conductor 不会把不同 rank 的块拼成更长的 GPU 前缀。 | +| `cpu` | 从查询的第一个块开始，至少有一个兼容共享 CPU 对象的连续前缀。 | +| `disk` | 从查询的第一个块开始，至少有一个兼容共享 Disk 对象的连续前缀。 | +| `longest_matched` | 某个 DP rank 使用该 rank 的 GPU 块，再加上兼容的共享 CPU 或 Disk 块后，能够覆盖的最长连续前缀。同一个块即使存在于多个位置，也只计算一次。 | + +例如，假设某个 rank 的前两个块在 GPU 中，只有第三个块位于共享 CPU 缓存，并且 `block_size` 为 `16`。这个 rank 可以报告 `longest_matched: 48` 和 `gpu: 32`；但独立计算的 `cpu` 前缀仍为 `0`，因为第一个块不在 CPU 缓存中。如果第一个和第二个 GPU 块分别属于不同 DP rank，Conductor 不会把它们合并成两个块的命中结果。 + +## 清理范围 + +清理操作只删除受影响事件源贡献的信息： + +- vLLM 的 remove 或 clear 只影响报告该事件的 endpoint、引擎和 DP rank 的 GPU 记录。其他 rank、其他引擎和共享缓存信息都会保留。 +- Mooncake 记录会保存报告事件的 endpoint、后端、租户、对象 key、完整连接器哈希，以及 CPU 或 Disk 位置。因此，即使两个对象的哈希最后 8 字节相同，删除其中一个对象也会保留另一个对象。 +- Mooncake clear 只影响报告事件的 endpoint、后端和租户下的共享对象。vLLM GPU 记录和其他 Mooncake 事件源不受影响。 +- 注销时，Conductor 会先停止选中的 `(instance_id, tenant_id, dp_rank)` 订阅，再删除该 endpoint 贡献的信息。注销 vLLM 还会从查询结果中删除对应 rank；注销 Mooncake 会删除该 endpoint 保存的全部对象绑定。 + +[Conductor 订阅端指南](../kv-event/subscriber-guide.md)详细说明事件校验和清理规则。 + +## 当前限制 + +- Conductor 连接后只能看到实时事件。当前 Mooncake 发布端不会在启动时发送此前已经缓存的对象列表。 +- 传输序号向前跳跃时，Conductor 会记录警告，但保留现有缓存记录。重新连接后，只有配置了 `replay_endpoint` 且 Conductor 已知上一个序号时，才会请求缺失事件；发出请求并不保证能够恢复全部记录。当前 Mooncake publisher 没有 replay 服务。 +- Conductor 按收到的批次顺序处理 Mooncake 事件。它不会仅仅因为 `event_id` 重复就自动忽略事件。 +- vLLM 只提供 GPU 信息，Mooncake 只提供 CPU 或 Disk 信息。其他缓存位置会被忽略，并记录警告。 +- Conductor 会读取连接器 key 中的层和并行 rank 信息，但在报告共享缓存可用之前，不会检查是否已经收到所有层，也不会检查是否已经收到所有张量并行（tensor parallel，TP）、预填充上下文并行（prefill context parallel，PCP）、解码上下文并行（decode context parallel，DCP）或流水线并行（pipeline parallel，PP）部分。 + +## 维护者参考源码 + +本页对应的主要实现位于 `mooncake-conductor/src/prefixindex/hash_strategy.cpp`、`mooncake-conductor/src/prefixindex/prefix_indexer.cpp`、`mooncake-conductor/src/kvevent/event_handler.cpp` 和 `mooncake-conductor/src/kvevent/event_manager.cpp`。准确的哈希示例来自 `mooncake-conductor/tests/fixtures/hash_golden_vectors.json`；查询结果行为由 `mooncake-conductor/tests/prefix_indexer_test.cpp` 和 `mooncake-conductor/tests/event_manager_test.cpp` 覆盖。 diff --git a/docs/source/zh/design/conductor/index.md b/docs/source/zh/design/conductor/index.md new file mode 100644 index 0000000000..da8950962f --- /dev/null +++ b/docs/source/zh/design/conductor/index.md @@ -0,0 +1,38 @@ +# Mooncake Conductor + +[English](../../../design/conductor/index.md) + +Mooncake Conductor 在内存中记录推理引擎和 Mooncake Store 上报的、可以复用的 +键值（KV）缓存块。路由器可以在选择推理引擎前查询这些信息。本页汇总了运行 +Conductor、理解查询结果和接入客户端所需的文档。 + +## 选择任务 + +### 运行 Conductor + +按照[使用指南](./usage.md)构建 C++ 服务、配置事件来源、检查注册结果、查询缓存 +可用情况，以及注销事件来源。 + +### 理解查询结果 + +阅读[架构说明](./conductor-architecture-design.md)，了解 vLLM 的 GPU 事件和 +Mooncake 共享的 CPU 或 Disk 事件如何变成每个推理引擎的查询结果。 + +### 调用 HTTP API + +查阅 [HTTP API 参考](./indexer-api-design.md)，了解当前实现的五个接口、可接受 +的字段、响应内容和错误格式。 + +### 连接事件来源 + +先阅读 [KV Event](../kv-event/index.md)，比较 vLLM 和 Mooncake 两种消息处理 +方式。该部分还会说明 Mooncake 发送什么，以及 Conductor 接受什么。 + +```{toctree} +:maxdepth: 1 +:hidden: + +conductor-architecture-design +usage +indexer-api-design +``` diff --git a/docs/source/zh/design/conductor/indexer-api-design.md b/docs/source/zh/design/conductor/indexer-api-design.md new file mode 100644 index 0000000000..ec95762968 --- /dev/null +++ b/docs/source/zh/design/conductor/indexer-api-design.md @@ -0,0 +1,373 @@ +# Mooncake Conductor HTTP API + +[English](../../../design/conductor/indexer-api-design.md) + +本参考介绍当前 C++ Conductor 服务实现的五个 HTTP 接口。你可以用这些接口注册 +实时事件来源、注销事件来源、查看 Conductor 的内存状态，以及查询可以复用的 +缓存前缀。下文中的字段名、可用值、响应字段大小写和错误格式都与当前解析和 +序列化代码一致。 + +## 选择接口 + +| 方法 | 路径 | 用途 | +|---|---|---| +| `POST` | `/register` | 启动一条 vLLM 或 Mooncake 事件订阅。 | +| `POST` | `/unregister` | 停止一条订阅，并清理该 endpoint 上报的缓存信息。 | +| `POST` | `/query` | 根据提示词的 token ID 查询缓存可用情况。 | +| `GET` | `/global_view` | 查看缓存共享范围和已注册的 vLLM rank。 | +| `GET` | `/services` | 列出正在使用的事件订阅及其完整配置。 | + +## 通用请求和响应规则 + +所有 `POST` 请求体都必须是 JSON 对象。Conductor 会拒绝未知字段。成功响应使用 +`application/json`；JSON 对象中各字段的顺序不属于接口约定。 + +大多数参数检查错误返回状态码 `400` 和一个 `application/json` 对象： + +```json +{ + "error": "unsupported request field: root_digest", + "reason": "unknown_field", + "field": "root_digest" +} +``` + +如果错误由某个字段引起，响应会包含 `field`。如果错误来自数组中的某个元素， +响应还会包含 `index`。[错误格式](#理解错误)一节列出了改为返回纯文本的情况。 + +## `POST /register` + +这个接口启动一条事件订阅。每个 vLLM 数据并行（DP）rank 都要使用不同的 +endpoint 分别注册。Mooncake 订阅提供共享 CPU 或 Disk 信息，但不会在 +`/query` 中增加一行推理实例。 + +### 请求字段 + +| 字段 | 必填 | 可接受的值和用途 | +|---|---|---| +| `endpoint` | 是 | 非空的 ZeroMQ 实时事件发布地址。该地址不能已被另一条有效注册使用。 | +| `type` | 是 | 只能是 `vLLM` 或 `Mooncake`。它决定 Conductor 按哪种事件消息格式读取数据。 | +| `modelname` | 是 | 非空的注册模型名。对 vLLM，它决定缓存范围；Mooncake 事件会自行携带模型名，决定实际写入哪个共享缓存范围。 | +| `instance_id` | 是 | 对 vLLM 是非空的推理引擎名称，对 Mooncake 是订阅名称。它是服务键的一部分；Mooncake 的这个值不会成为查询实例。 | +| `block_size` | 是 | 每个块包含的 token 数，必须为正数。对 vLLM，它决定缓存范围；Mooncake 事件会携带实际的块大小。 | +| `dp_rank` | 是 | 从 `0` 到当前平台 `int` 最大值的整数。对 vLLM，它选择 DP rank；它也是每个服务键的一部分。对 Mooncake，它只用于标识订阅。 | +| `hash_profile` | 是 | 包含下表四个哈希字段的对象。 | +| `replay_endpoint` | 否 | 重连后用来请求缺失事件的字符串地址。默认值是 `""`，表示不创建补发连接。 | +| `lora_name` | 否 | 已注册的低秩适配（LoRA）名称，默认值是 `""`。对 vLLM，它决定缓存范围；Mooncake 事件会携带实际的 LoRA 名称。 | +| `tenant_id` | 否 | 已注册的租户名。省略或传入 `""` 都会变成 `"default"`。对 vLLM，它决定缓存范围，也是服务键的一部分；Mooncake 事件会携带实际租户。 | +| `cache_group` | 否 | 整数 `0` 或 `null`。省略也表示不明确指定 group。其他值和数组都会被拒绝。 | + +目前只支持下面这一种 `hash_profile`： + +| 哈希字段 | 支持的值 | +|---|---| +| `strategy` | `vllm_v1` | +| `algorithm` | `sha256_cbor` | +| `python_hash_seed` | 字符串，内容必须是准确的 `random`，或者数值在 `0..4294967295` 范围内的 ASCII 十进制文本。Conductor 会保留前导零等原始文本。 | +| `index_projection` | `low64_be` | + +输入对象必须只包含这四个字段。空值、带正负号、两侧有空白、非字符串、无效 +UTF-8 和超出范围的种子都会被拒绝。`"0"` 与 `"00"` 不同；数值类型的 JSON +`0` 无效。注册时提供 `root_digest` 会被视为旧版未知字段并遭到拒绝。 + +由四个字段确定的同一缓存共享范围必须使用完全相同的已解析哈希配置。对 vLLM， +这些字段来自注册信息；对 Mooncake，这些字段来自每条事件，而哈希配置来自 +Mooncake 的注册信息。Conductor 对准确的种子文本进行规范 CBOR 编码，再计算 +SHA-256 派生根摘要。这段文本必须与每个兼容 vLLM 进程中的 `PYTHONHASHSEED` +相同。明确设置的字符串 `random` 受支持；未设置该环境变量时，vLLM 会选择 +Conductor 无法复现的随机根字节，因此不受支持。vLLM 的 `--seed` 控制模型和 +采样随机性，不控制前缀缓存的哈希标识。 + +关于规范 Concise Binary Object Representation（CBOR）输入顺序、LoRA 和 +`cache_salt` 的先后顺序、如何使用完整父摘要逐块计算、如何把最后八个字节按 +大端序转成查询值，以及带版本说明的测试样例，请参阅 +[token 块的哈希计算说明](./conductor-architecture-design.md)。`cache_salt` 是查询 +字段，不是注册字段。Mooncake 事件中的 `additional_salt` 只用于诊断，这个 +HTTP 接口不接受该字段。 + +### 最小请求 + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5557", + "type": "vLLM", + "modelname": "test-model", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +状态码 `200` 返回： + +```json +{ + "status": "registered successfully", + "instance_id": "engine-a" +} +``` + +完全相同的有效注册再次提交时，Conductor 仍返回相同的成功响应，但不会再启动 +一个订阅客户端。如果服务键、endpoint 或哈希配置冲突，Conductor 返回 JSON +格式的 `400`，其中 `reason` 是 `invalid_registration`。本地订阅客户端无法启动 +时，Conductor 返回纯文本 `500`。 + +## `POST /unregister` + +这个接口停止一条指定的订阅。它会等待订阅客户端停止，然后再清理该 endpoint +上报的缓存信息。 + +### 请求字段 + +| 字段 | 必填 | 可接受的值和用途 | +|---|---|---| +| `instance_id` | 是 | 注册事件来源时使用的非空字符串。 | +| `dp_rank` | 是 | 注册事件来源时使用的非负整数 rank。 | +| `tenant_id` | 否 | 注册事件来源时使用的字符串。省略或传入 `""` 都会变成 `"default"`。 | + +不接受其他字段。 + +### 最小请求 + +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","dp_rank":0}' +``` + +状态码 `200` 返回实际删除的服务键： + +```json +{ + "status": "unregistered successfully", + "removed_instances": [ + "engine-a|default|0" + ] +} +``` + +服务键不存在时返回纯文本 `404`。如果订阅客户端停止后清理失败，则返回纯文本 +`500`。此时 Conductor 会继续占用该服务键和 endpoint，二者都不能重新注册。 +请重复发送相同的 `/unregister` 请求，直到清理成功。 + +## `POST /query` + +这个接口使用已注册的哈希配置，对请求中完整的 token 块进行哈希计算。请求中的 +租户、模型、LoRA 名称和块大小必须能找到对应的注册信息。请求不能覆盖算法、 +策略、`python_hash_seed`、派生根摘要或最后八字节的查询规则。 + +### 请求字段 + +| 字段 | 必填 | 可接受的值和用途 | +|---|---|---| +| `model` | 是 | 非空的模型名，必须与注册信息中的 `modelname` 一致。 | +| `block_size` | 是 | 正整数。只有包含足够 token 的完整块会参与哈希计算。 | +| `token_ids` | 是 | 由有符号 32 位整数组成的 JSON 数组。空数组有效，并会为匹配的已注册 rank 返回零命中。 | +| `tenant_id` | 否 | 字符串。省略或传入 `""` 都会变成 `"default"`。 | +| `lora_name` | 否 | 字符串。默认值是 `""`，表示基础模型。 | +| `cache_salt` | 否 | 字符串、`null` 或省略。`null`、`""` 和省略都表示不加 salt；非空值必须与生产方一致。 | +| `instance_id` | 否 | 字符串过滤条件。匹配时只返回指定实例；未知值返回空的 `instances` 对象。 | + +请求不能包含覆盖哈希配置的字段。如果找不到对应的缓存共享范围，也会返回状态码 +`200` 和空的 `instances` 对象，并且不会创建新状态。 + +### 最小请求 + +```bash +curl -sS -X POST http://127.0.0.1:13333/query \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "test-model", + "block_size": 16, + "token_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] + }' +``` + +如果已经注册 `engine-a` 的 rank `0`，但还没有收到匹配的缓存事件，响应如下： + +```json +{ + "instances": { + "engine-a": { + "longest_matched": 0, + "dp": { + "0": 0 + }, + "gpu": 0, + "cpu": 0, + "disk": 0 + } + } +} +``` + +### 返回字段 + +| 字段 | 含义 | +|---|---| +| `instances` | 以选中的已注册 vLLM `instance_id` 为键。Mooncake 订阅不会成为返回结果中的一行。 | +| `longest_matched` | 某一个 rank 可以用自己的 GPU 缓存加上兼容的共享 CPU 或 Disk 缓存提供的最大连续 token 数。 | +| `dp` | 每个 rank 在 GPU 上连续命中的 token 数。rank 键是十进制 JSON 字符串。 | +| `gpu` | 该引擎所有 rank 中最大的连续 GPU token 数。 | +| `cpu` | 从第一个块开始，在共享 CPU 缓存中连续命中的 token 数。 | +| `disk` | 从第一个块开始，在共享 Disk 缓存中连续命中的 token 数。 | + +在当前 HTTP 测试使用的状态中，实例 `1` 的 rank `0` 有 32 个 token 的连续 GPU +缓存，实例 `2` 的 rank `1` 没有 GPU 命中，两者都能看到连续 48 个 token 的 +共享 CPU 和 Disk 缓存。准确响应如下： + +```json +{ + "instances": { + "1": { + "longest_matched": 48, + "gpu": 32, + "dp": { + "0": 32 + }, + "cpu": 48, + "disk": 48 + }, + "2": { + "longest_matched": 48, + "gpu": 0, + "dp": { + "1": 0 + }, + "cpu": 48, + "disk": 48 + } + } +} +``` + +连续命中和各 rank 的计算规则见[查询字段说明](./conductor-architecture-design.md)。 + +## `GET /global_view` + +这个接口显示当前 Conductor 进程已知的缓存共享范围。请求不需要请求体。 + +### 最小请求 + +```bash +curl -sS http://127.0.0.1:13333/global_view +``` + +注册 `engine-a` 的 rank `0` 和 `1` 后，在收到缓存事件前，单个缓存范围的响应 +格式如下： + +```json +{ + "context_count": 1, + "contexts": [ + { + "model_name": "test-model", + "lora_name": "", + "block_size": 16, + "tenant_id": "default", + "prefix_count": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" + }, + "instances": { + "engine-a": [ + 0, + 1 + ] + } + } + ] +} +``` + +`context_count` 是 `contexts` 中的条目数。`prefix_count` 统计仍有至少一条 GPU、 +CPU 或 Disk 记录的不同查询值；这里的查询值是完整摘要的最后八个字节。 +`instances` 把已注册的 vLLM 引擎映射到由数字组成的 rank 数组。Mooncake 订阅 +不会加入该映射。这个数组的顺序不固定。已解析的 `hash_profile` 会同时返回准确 +的配置种子和由它派生的小写根摘要。 + +## `GET /services` + +这个接口列出正在使用的订阅配置。请求不需要请求体。字段名特意保留下方所示的 +大小写，与注册请求不同。 + +### 最小请求 + +```bash +curl -sS http://127.0.0.1:13333/services +``` + +发送上面的最小注册请求后，状态码 `200` 返回： + +```json +{ + "count": 1, + "services": [ + { + "Endpoint": "tcp://127.0.0.1:5557", + "ReplayEndpoint": "", + "Type": "vLLM", + "ModelName": "test-model", + "LoraName": "", + "TenantID": "default", + "InstanceID": "engine-a", + "BlockSize": 16, + "DPRank": 0, + "CacheGroup": null, + "HashProfile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "root_digest": "4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e", + "index_projection": "low64_be" + } + } + ] +} +``` + +`count` 是有效服务键的数量。如果注册请求省略该字段，`CacheGroup` 就是 +`null`。服务数组的顺序不固定。出现在这个列表中只说明 Conductor 已完成 +本地订阅设置，不代表远端已经发送事件，也不代表已经找回更早的事件。开始产生 +缓存的流量前，请对照 `/global_view` 检查 `python_hash_seed` 和派生的 +`root_digest`。 + +## 理解错误 + +Conductor 会根据错误类型返回 JSON 参数错误或纯文本运行错误： + +| 情况 | 状态码 | Content type 和响应体 | +|---|---|---| +| 任意 `POST` 接口的字段检查失败 | `400` | `application/json`，包含 `error`、`reason`，以及适用时的 `field` 或 `index`。 | +| `/query` 的 JSON 格式错误，或请求体不是对象 | `400` | `application/json`，内容为 `{"error":"Invalid JSON object","reason":"invalid_json"}`。 | +| `/register` 或 `/unregister` 的 JSON 格式错误，或请求体不是对象 | `400` | `text/plain; charset=utf-8`，内容为 `Invalid JSON\n`。 | +| `/unregister` 找不到服务键 | `404` | `text/plain; charset=utf-8`，例如 `service not found: engine-a\|default\|0\n`。 | +| 使用 `GET` 请求 `/register`、`/unregister` 或 `/query`；使用 `POST` 请求 `/global_view` 或 `/services` | `405` | `text/plain; charset=utf-8`，内容为 `Method not allowed\n`。 | +| `/register` 无法启动本地订阅客户端 | `500` | `text/plain; charset=utf-8`，开头是 `Failed to subscribe: failed to start ZMQ client:`。 | +| `/unregister` 已停止订阅客户端，但缓存清理失败 | `500` | `text/plain; charset=utf-8`，开头是 `Failed to unregister prefix context:`。 | + +例如，`token_ids` 中出现字符串时，会返回精确到数组元素的 JSON 错误： + +```json +{ + "error": "token_ids element must be a JSON integer", + "reason": "invalid_type", + "field": "token_ids", + "index": 0 +} +``` + +除 `/services` 外，Conductor 生成的 JSON 响应都以换行符结尾；该紧凑 JSON +响应没有结尾换行。上表中用 `\n` 表示的纯文本错误会带有该结尾换行。 diff --git a/docs/source/zh/design/conductor/usage.md b/docs/source/zh/design/conductor/usage.md new file mode 100644 index 0000000000..838f4ec853 --- /dev/null +++ b/docs/source/zh/design/conductor/usage.md @@ -0,0 +1,315 @@ +# 运行 Mooncake Conductor + +[English](../../../design/conductor/usage.md) + +本指南从检出源码开始，一步步运行 Conductor，直到 HTTP 查询返回可用结果。内容包括静态和动态注册事件源、vLLM 与 Mooncake 的稳妥启动顺序，以及准确的清理命令。所有示例都针对当前 C++ 服务，并且只使用当前解析器接受的字段。 + +## 开始前 + +先按照[构建指南](../../../getting_started/build.md)安装仓库所需的构建依赖。Conductor 组件还需要 ZeroMQ、C++ 版 MessagePack、OpenSSL、JsonCpp、glog 和 yalantinglibs；CMake 会在配置阶段检查这些依赖。 + +还需要准备： + +- 为每个要注册的 vLLM 数据并行（DP）rank 准备一个键值（KV）事件地址（endpoint）； +- 如果需要共享 CPU 或 Disk 信息，再准备一个 Mooncake Master KV Event 地址； +- 使用 `curl` 执行下文的检查； +- 为所有兼容的 vLLM 进程设置同一个明确的 `PYTHONHASHSEED` 值。 + +下文中的地址、模型名称和种子都只是示例。开始产生缓存的业务流量前，请把它们替换为同一部署中的真实值。不要省略 `PYTHONHASHSEED`：未设置该环境变量时，vLLM 会生成随机根字节，Conductor 无法根据注册信息复现它们。 + +## 构建 + +在仓库根目录执行配置，并且只构建 Conductor 可执行程序： + +```bash +cmake -S . -B build -DWITH_CONDUCTOR=ON +cmake --build build --target mooncake_conductor +``` + +生成的可执行文件是 `build/mooncake-conductor/mooncake_conductor`。用下面的命令确认构建确实生成了该文件： + +```bash +test -x build/mooncake-conductor/mooncake_conductor +``` + +退出状态为 0，说明可执行文件已经可以运行。 + +## 配置 + +Conductor 读取两个环境变量： + +| 环境变量 | 默认值 | 何时会用到 | +|---|---|---| +| `CONDUCTOR_CONFIG_PATH` | `$HOME/.mooncake/conductor_config.json` | 指定启动时读取的 JSON 文件。文件不存在时，Conductor 不会创建静态订阅，并保留内置 HTTP 端口 `13333`。 | +| `CONDUCTOR_LOG_LEVEL` | `INFO` | 设置 `DEBUG`、`INFO`、`WARN` 或 `ERROR`，不区分大小写。值为空或无效时使用 `INFO`。 | + +在可读取的配置文件中，一定要显式设置 `http_server_port`。如果成功读取了配置文件，但其中没有这个字段，当前解析器会把端口设为 `0`，不会继续使用 `13333`。 + +下面是一份完整的静态配置示例：它为一个引擎注册两个 vLLM rank，并注册一个 Mooncake 共享缓存池。`lora_name` 为空表示基础模型；非空值表示一个低秩适配（Low-Rank Adaptation，LoRA）适配器。 + +```json +{ + "http_server_port": 13333, + "kvevent_instance": { + "vllm-engine-a-rank-0": { + "endpoint": "tcp://127.0.0.1:5557", + "replay_endpoint": "tcp://127.0.0.1:5558", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }, + "vllm-engine-a-rank-1": { + "endpoint": "tcp://127.0.0.1:5567", + "replay_endpoint": "tcp://127.0.0.1:5568", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 1, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }, + "mooncake-shared-pool": { + "endpoint": "tcp://127.0.0.1:6557", + "replay_endpoint": "", + "type": "Mooncake", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "shared-pool", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + } + } +} +``` + +`python_hash_seed` 必须与每个兼容 vLLM 进程中 `PYTHONHASHSEED` 环境变量的准确文本相同。Conductor 保留这段文本，把它编码为规范 CBOR 文本字符串，再计算 SHA-256 得到根摘要。种子字符串 `"0"` 得到的小写诊断值是 `4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e`。这个摘要会出现在 `/services` 和 `/global_view` 中；它不是注册输入，也不是通用于所有部署的默认值。 + +JSON 中必须使用带引号的字符串：`"0"`、`"00"` 和数值 `0` 是不同输入。明确设置的字面量 `random` 受支持，并根据这段准确文本派生根摘要；未设置环境变量时，vLLM 会选择无法注册复现的随机字节，因此不受支持。vLLM 的 `--seed` 控制模型和采样随机性，与前缀缓存的哈希标识无关。请求中的 `cache_salt` 只发送给 `/query`，不是注册字段。准确规则请参阅 [token 块的哈希计算](./conductor-architecture-design.md#how-token-blocks-become-lookup-values)。 + +所有兼容的 vLLM 进程都要使用相同的环境变量文本和规范 CBOR 前缀哈希算法。对于本文的种子零示例： + +```bash +PYTHONHASHSEED=0 vllm serve test-model \ + --enable-prefix-caching \ + --prefix-caching-hash-algo sha256_cbor +``` + +使用绝对配置路径和所需日志级别启动 Conductor： + +```bash +export CONDUCTOR_CONFIG_PATH=/absolute/path/to/conductor_config.json +export CONDUCTOR_LOG_LEVEL=INFO +./build/mooncake-conductor/mooncake_conductor +``` + +启动成功时，日志中会出现 `HTTP server listening port=13333`，以及静态订阅成功和失败的数量。所有静态订阅会同时启动，因此 JSON 中各项的排列顺序不能保证 vLLM 早于 Mooncake 启动。 + +## 只使用 vLLM 启动 + +动态注册可以明确控制启动顺序。如果不需要静态订阅，请使用下面这份替代配置启动 Conductor： + +```json +{ + "http_server_port": 13333, + "kvevent_instance": {} +} +``` + +启动可执行文件后，注册每个 vLLM rank。第一个请求注册 rank `0`： + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5557", + "replay_endpoint": "tcp://127.0.0.1:5558", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +用该 rank 自己的事件 endpoint 注册 rank `1`，引擎和哈希设置保持不变： + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:5567", + "replay_endpoint": "tcp://127.0.0.1:5568", + "type": "vLLM", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "engine-a", + "block_size": 16, + "dp_rank": 1, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +每个成功请求都会返回 `registered successfully`。放开会产生缓存的业务流量前，先确认引擎信息： + +```bash +curl -sS http://127.0.0.1:13333/global_view +``` + +对应结果必须显示 `tenant_id` 为 `default`、`model_name` 为 `test-model`、`lora_name` 为空、`block_size` 为 `16`、`python_hash_seed` 为 `"0"`、根摘要为上文的派生值，并且 `"engine-a":[0,1]` 位于 `instances` 下。还要检查 `/services`：其中应包含两个 vLLM 订阅，并显示相同的种子和派生根摘要。如果部署中只有 vLLM，现在可以开始产生缓存的业务流量。 + +## 添加 Mooncake + +添加共享缓存池时，先不要放开会产生缓存的业务流量。按照 [Mooncake 发布端指南](../kv-event/publisher-design.md)启用并启动 Mooncake Master 发布端，然后注册它的实时事件地址： + +```bash +curl -sS -X POST http://127.0.0.1:13333/register \ + -H 'Content-Type: application/json' \ + -d '{ + "endpoint": "tcp://127.0.0.1:6557", + "replay_endpoint": "", + "type": "Mooncake", + "modelname": "test-model", + "lora_name": "", + "tenant_id": "default", + "instance_id": "shared-pool", + "block_size": 16, + "dp_rank": 0, + "cache_group": 0, + "hash_profile": { + "strategy": "vllm_v1", + "algorithm": "sha256_cbor", + "python_hash_seed": "0", + "index_projection": "low64_be" + } + }' +``` + +配置 Mooncake 发布端，使每个事件携带的租户、模型、LoRA 名称和块大小与 vLLM 的缓存信息一致。`hash_profile` 来自 Mooncake 注册项，并且必须与该事件对应的缓存信息已经绑定的哈希配置相同。注册时也使用同样的缓存信息，这样更容易对照 `/services` 和发布端。Mooncake 的 `instance_id` 和 `dp_rank` 只用于在 `/services` 中标识订阅，以及通过 `/unregister` 注销；它们不会创建查询实例，也不会覆盖事件中的缓存信息。`stored` 事件还必须带有对象 key 和可用的完整连接器哈希；事件检查规则请参阅[订阅兼容指南](../kv-event/subscriber-guide.md)。 + +放开业务流量前，确认 `/services` 同时包含两个 vLLM rank 和 Mooncake 订阅： + +```bash +curl -sS http://127.0.0.1:13333/services +``` + +如果使用静态配置，则采用另一种稳妥做法：在 `/global_view` 显示预期的 vLLM 缓存信息、引擎、rank、配置种子和派生根摘要，并且 `/services` 显示所有预期订阅和同一份已解析 hash profile 前，保持会产生缓存的业务流量暂停。静态订阅会同时启动，与 JSON 中的排列顺序无关。 + +这些检查只能说明有限的状态。`/services` 表示 Conductor 接受了注册，并启动了本地订阅客户端；它不能证明远端 ZeroMQ 发布端已经发送事件。`/global_view` 表示 vLLM 缓存信息和 rank 已经注册；它不会恢复 Conductor 连接前已写入 Mooncake 的对象。当前 Mooncake 发布端不会重发此前的状态。 + +## 检查运行状态 + +查看当前订阅： + +```bash +curl -sS http://127.0.0.1:13333/services +``` + +对于上面的动态注册示例，`count` 为 `3` 才表示成功。两个 `vLLM` 项的 `InstanceID` 都应为 `engine-a`，`DPRank` 分别为 `0` 和 `1`；`Mooncake` 项的 `InstanceID` 应为 `shared-pool`。三个 `HashProfile` 必须相同，包括 `python_hash_seed` `"0"` 及其派生的 `root_digest`。 + +查看缓存共享组合和已注册的推理 rank： + +```bash +curl -sS http://127.0.0.1:13333/global_view +``` + +对应结果中出现 `"engine-a":[0,1]`，说明检查成功。Mooncake 不会作为另一个推理实例出现。收到实时事件后，`prefix_count` 可能增加，但仅凭这个计数无法判断每个块来自哪个缓存位置。 + +## 查询 + +发送已注册模型的分词器（tokenizer）生成的 token IDs。下面的示例包含一个完整的 16-token 块： + +```bash +curl -sS -X POST http://127.0.0.1:13333/query \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "test-model", + "lora_name": "", + "tenant_id": "default", + "block_size": 16, + "token_ids": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] + }' +``` + +成功响应中会有 `instances.engine-a` 对象，其中 DP rank key 为 `"0"` 和 `"1"`。在收到匹配的实时事件前，命中值可能为零。查询只计算完整块。如果事件生产端使用缓存盐值（cache salt）计算请求哈希，请在这次查询中添加取值相同的 `cache_salt` 字符串，不要把它加入注册项。 + +准确的响应字段请参阅 [HTTP API 参考](./indexer-api-design.md#post-query)；Conductor 如何组合 GPU、CPU 和 Disk 可用性，请参阅[架构指南](./conductor-architecture-design.md#what-query-fields-mean)。 + +## 注销 + +替换发布端前，先注销准确的服务标识（service key）。这个 key 由 `instance_id`、规范化后的 `tenant_id` 和 `dp_rank` 组成。 + +删除 Mooncake 订阅： + +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"shared-pool","tenant_id":"default","dp_rank":0}' +``` + +分别删除每个 vLLM rank： + +```bash +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","tenant_id":"default","dp_rank":1}' + +curl -sS -X POST http://127.0.0.1:13333/unregister \ + -H 'Content-Type: application/json' \ + -d '{"instance_id":"engine-a","tenant_id":"default","dp_rank":0}' +``` + +每个成功响应都包含 `unregistered successfully` 和准确的已删除 service key，例如 `engine-a|default|0`。检查 `/services`，确认对应订阅已经消失。注销一个 vLLM rank 只删除该 rank 的 GPU 信息；注销 Mooncake 只删除由该 Mooncake endpoint 报告的对象。 + +## 处理配置问题 + +| 现象 | 检查 | 处理方式 | +|---|---|---| +| HTTP 请求无法连接，或日志显示端口为 `0` | 确认 `CONDUCTOR_CONFIG_PATH` 指向刚才编辑的文件，并且文件中包含数值类型的 `http_server_port`。 | 设置绝对配置路径，添加明确的非零端口，然后重启 Conductor。 | +| `/services` 中没有某个事件源 | 查看注册响应和 Conductor 日志，检查是否使用了不支持的 type、非零 `cache_group`、无效哈希配置、重复 endpoint 或冲突的 service key。 | type 只使用 `vLLM` 或 `Mooncake`；缓存组使用 `0` 或省略；使用唯一 endpoint 和准确的种子式哈希字段。 | +| `/services` 中有事件源，但命中值一直为零 | 检查发布端是否在 Conductor 连接后发出过事件。`/services` 不能证明事件已经送达。 | 检查发布端状态，配置期间保持业务流量暂停，并在产生新缓存前完成注册。此前的 Mooncake 事件不会重发。 | +| Mooncake 事件没有增加 CPU 或 Disk 可用性 | 对照 `/global_view`，检查 Mooncake 事件携带的租户、模型、LoRA 名称和块大小；比较注册的哈希配置；确认 stored 事件包含对象信息和完整连接器哈希数据。 | 先注册 vLLM，确保事件中的缓存信息和哈希配置一致，并修正 Mooncake 发布端或 key 配置。 | +| `/query` 返回空的 `instances` 对象，或命中长度比预期短 | 对照事件生产端检查 `model`、`tenant_id`、`lora_name`、`block_size`、可选 `instance_id`、token IDs 和 `cache_salt`。同时检查末尾是否是不完整块。 | 查询准确的已注册四字段组合，使用生产端的盐值规则，并发送足够组成完整块的 token。 | +| HTTP 请求返回 `400` | 查看 JSON 中的 `reason`、`field` 和可选 `index`；注册或注销请求的 JSON 格式错误会改为返回纯文本。 | 删除不支持的字段，并按照 [API 字段表](./indexer-api-design.md)修正错误中指出的值。 | diff --git a/docs/source/zh/design/kv-event/index.md b/docs/source/zh/design/kv-event/index.md new file mode 100644 index 0000000000..b02d0afcbb --- /dev/null +++ b/docs/source/zh/design/kv-event/index.md @@ -0,0 +1,40 @@ +# KV Event + +[English](../../../design/kv-event/index.md) + +KV Event 用来告诉 Conductor：键值（KV）缓存块当前存放在哪里。在本项目中， +vLLM 上报某个推理引擎及其 rank 持有的 GPU 缓存块，Mooncake 则上报共享 +CPU 或 Disk 缓存池中的对象。请根据你要配置的一端选择对应指南。 + +## 选择事件来源 + +| 来源 | 上报内容 | 后续文档 | +|---|---|---| +| vLLM | 某个已注册引擎及其数据并行（DP）rank 的 GPU 缓存。 | 查看 [Conductor 订阅指南](./subscriber-guide.md)，了解 Conductor 接受哪些 vLLM 消息字段。 | +| Mooncake Master | 可由兼容的已注册引擎共享的 CPU 或 Disk 对象。 | 查看 [Mooncake 发布指南](./publisher-design.md)，启用发布端并检查其状态。 | + +Conductor 根据注册来源的 `type` 选择消息读取方式，而不是根据 ZeroMQ +（ZMQ）的 topic 文本选择。订阅指南详细对比了 `vLLM` 和 `Mooncake`。 + +## 发布 Mooncake 事件 + +[Mooncake KV Event 发布指南](./publisher-design.md)介绍构建选项、Master +设置、状态计数、准确的三帧消息格式、connector key 解析以及事件丢失后的 +行为。 + +## 连接 Conductor + +[Conductor KV Event 订阅指南](./subscriber-guide.md)说明每种注册来源可以 +发送什么、哈希如何转成查询值、哪些事件会更新或清除缓存信息，以及注销时 +如何清理。 + +如需完整的启动、注册和 HTTP 检查步骤，请参阅 +[Conductor 使用指南](../conductor/usage.md)。 + +```{toctree} +:maxdepth: 1 +:hidden: + +publisher-design +subscriber-guide +``` diff --git a/docs/source/zh/design/kv-event/publisher-design.md b/docs/source/zh/design/kv-event/publisher-design.md new file mode 100644 index 0000000000..447d7151a4 --- /dev/null +++ b/docs/source/zh/design/kv-event/publisher-design.md @@ -0,0 +1,271 @@ +# 发布 Mooncake KV Event + +[English](../../../design/kv-event/publisher-design.md) + +Mooncake Master 可以向 Conductor 和其他订阅端实时发布键值（KV）缓存变化。 +本页介绍如何启用发布端、确认它是否正在发送消息，以及如何理解它实际生成的 +MessagePack 字段。本文也会说明影响 Conductor 的对象 key 要求和消息传递限制。 + +## 启用并检查发布端 + +在项目根目录中构建启用了 KV Event 的 `mooncake_master`： + +```bash +cmake -S . -B build -DENABLE_KV_EVENTS=ON +cmake --build build --target mooncake_master -j +``` + +启动 Master 时，需要提供可用的后端名称、模型备用值、块大小和 ZeroMQ +（ZMQ）绑定地址。下面的命令使用管理 HTTP 端口 `9003`： + +```bash +./build/mooncake-store/src/mooncake_master \ + --enable_kv_events=true \ + --kv_events_bind_endpoint=tcp://0.0.0.0:5557 \ + --kv_events_backend_id=pool-a \ + --kv_events_model_name=Qwen/Qwen2.5-7B-Instruct \ + --kv_events_block_size=16 \ + --kv_events_emit_object_key=true \ + --metrics_port=9003 +``` + +启动成功后，日志中会出现包含以下内容的一行： + +```text +kv_events publisher enabled on tcp://0.0.0.0:5557 backend_id=pool-a +``` + +然后读取发布端状态： + +```bash +curl -s http://127.0.0.1:9003/kv_events/status +``` + +在缓存流量开始前，健康的发布端会返回下面这种结构。如果此前已经执行过缓存 +操作，计数也可能已经不是零： + +```json +{"enabled":true,"published_batches":0,"published_events":0,"dropped_events":0,"skipped_unparsed_keys":0,"invalid_event_hashes":0} +``` + +先确认 `enabled` 为 `true`。执行缓存操作后，再确认 `published_events` 以及 +通常情况下的 `published_batches` 有所增加。即使没有订阅端，这些计数也会 +增加，因此该检查只能证明 Master 已向自己的 ZMQ 发布（PUB）socket 发送 +消息，不能证明 Conductor 已经收到消息。 + +## 了解构建要求 + +`-DENABLE_KV_EVENTS=ON` 需要 libzmq 头文件和库。如果 CMake 找不到它们， +配置阶段会直接报错。未开启该选项的构建仍保留 Store 公共调用，但使用的是 +空实现；运行时参数无法把这个空实现变成真正的发布端，状态会显示 +`"enabled":false`。 + +如果 `kv_events_bind_endpoint` 或 `kv_events_backend_id` 为空，或者创建、 +绑定 ZMQ socket 失败，发布端同样会把 `enabled` 设为 `false`。此时应查看 +Master 日志。 + +## 选择 Master 设置 + +这些设置既可以写进 Master 配置文件，也可以通过命令行参数传入。显式传入的 +命令行值会覆盖 `--config_path` 所加载的值。 + +| 设置 | 默认值 | 何时会用到 | +|---|---:|---| +| `enable_kv_events` | `false` | 必须设为 `true` 才会创建发布端。 | +| `kv_events_bind_endpoint` | 空 | Master 绑定的 ZMQ PUB 地址，例如 `tcp://0.0.0.0:5557`。向 Conductor 注册时应使用 `tcp://master-host:5557` 这类可连接的地址，不能使用 `0.0.0.0`。 | +| `kv_events_backend_id` | 空 | 必须是非空的 Mooncake 后端名称，用于标识上报这些对象的后端。Conductor 会用它限制 Remove 和 Clear 的清理范围。 | +| `kv_events_model_name` | 空 | 无法识别 key 以及 `cleared` 事件所用的备用模型名。从 connector key 解析出的模型名优先。Conductor 要求 `stored` 事件具有非空模型名。 | +| `kv_events_block_size` | `0` | 写入事件的固定 token 数。`0` 会发送为 `nil`；如果 `stored` 的块大小缺失或不是正数，Conductor 会拒绝该事件。 | +| `kv_events_lora_name` | 空 | 固定的低秩适配（LoRA）名称。空值表示基础模型，并发送为 `nil`。 | +| `kv_events_additional_salt` | 空 | 固定字符串，空值发送为 `nil`。Conductor 会读取它，但不会把它用于缓存共享或查询。 | +| `kv_events_dp_rank` | `0` | 写入每个事件和批次的无符号数据并行（DP）值。Conductor 只保留 Mooncake DP 值用于排查问题，不把它当作推理引擎的 rank。 | +| `kv_events_tenant_id` | `default` | 为兼容配置而保留。事件中的 tenant 来自每次 Store 操作；操作中的空 tenant 会变成 `default`。 | +| `kv_events_emit_object_key` | `true` | 在对象事件中加入 `object_key`。与 Conductor 配合时应保持为 `true`，因为 Conductor 需要该 key 才能准确删除对象。 | +| `kv_events_emit_legacy_compat` | `true` | 加入对应的旧字段 `type`、`block_hashes`，以及 stored 事件的 `parent_block_hash`。只要它们与主要字段一致，Conductor 就会接受。 | +| `kv_events_queue_capacity` | `65536` | 发布端队列中最多等待处理的事件数。正数会限制队列长度；`0` 表示不设该限制。参见[规划事件丢失](#规划事件丢失)。 | + +`kv_events_queue_capacity` 的命令行帮助目前声称该设置会被忽略，但发布端实现 +确实会执行这个限制。上表描述的是当前实际运行行为。 + +## 读取发布端状态 + +`GET /kv_events/status` 由 Master 的管理端口提供。该端口通过 +`metrics_port` 配置，默认值为 `9003`。响应字段如下： + +| 字段 | 含义 | +|---|---| +| `enabled` | 当前 Master 已编译发布功能、完成配置并成功绑定发布地址。 | +| `published_batches` | 三次 ZMQ 帧发送均返回成功的多帧消息数。 | +| `published_events` | 上述成功发送批次中包含的事件数。 | +| `dropped_events` | 等待队列已满时被移除的事件数，加上 ZMQ 发送失败批次中的事件数。 | +| `skipped_unparsed_keys` | 没有序列哈希、并且由于无法发送对象 key 而被跳过的事件，加上 key 中没有可识别块哈希部分但仍被发送的事件。后一种消息包含 `object_key`，却没有 connector hash，因此 Conductor 会拒绝其中的 `stored` 事件。 | +| `invalid_event_hashes` | connector key 已被识别，但其中的块哈希文本无法转成无符号 64 位值。事件仍可能带着对象 key 发出，但不会包含 `connector_block_hash`。 | + +计数为零不能证明订阅端已连接。同样，PUB socket 在本地发送成功后仍可能丢失 +消息，因此这些计数不能证明 Conductor 已经收到事件。 + +## 了解三帧 ZMQ 消息 + +每次发布都是一条恰好包含三帧的 ZMQ 多帧消息： + +| 帧 | 准确内容 | +|---:|---| +| 1 | 空 topic 帧。 | +| 2 | 一个按大端序编码的无符号 64 位传输序号。 | +| 3 | 一个 MessagePack payload：`[timestamp_ms, [event_maps], dp_rank]`。 | + +发布端最多把 64 个等待事件放进一个 payload。同一 payload 中的所有事件使用 +同一个有符号 64 位 Unix 毫秒时间戳。`dp_rank` 是配置的无符号 32 位值。 +发布端为每个尝试发送的批次分配一个序号；队列丢弃事件时还会额外保留序号。 +每次发布端进程启动后都从 `1` 开始，因此队列丢失会有意留下序号空缺。 + +payload 中三项的含义如下： + +| 项 | MessagePack 形式 | 含义 | +|---:|---|---| +| `timestamp_ms` | 非负整数 | 批次创建时的 Unix 毫秒时间。 | +| `event_maps` | map 数组 | 事件顺序与发布端从队列取出的顺序相同。 | +| `dp_rank` | 无符号整数 | `kv_events_dp_rank` 的副本；Conductor 只将它用于 Mooncake 路径的问题排查。 | + +## 读取通用事件字段 + +每个 `stored`、`removed` 和 `cleared` map 都包含以下主要字段： + +| 字段 | MessagePack 形式 | 来源和用途 | +|---|---|---| +| `event_id` | 无符号 64 位整数 | 当前发布端进程中递增的事件编号。进程重启后会重新开始。 | +| `timestamp` | 有符号 64 位整数 | 与外层 `timestamp_ms` 相同的毫秒值。 | +| `event_type` | 字符串 | 只能是 `stored`、`removed` 或 `cleared`。 | +| `model_name` | 字符串或 `nil` | 如果成功解析 connector key，则使用其中的模型；否则使用 `kv_events_model_name`。 | +| `block_size` | 无符号整数或 `nil` | 来自 `kv_events_block_size`；`0` 会变成 `nil`。 | +| `additional_salt` | 字符串或 `nil` | 来自 `kv_events_additional_salt`；空值会变成 `nil`。 | +| `lora_name` | 字符串或 `nil` | 来自 `kv_events_lora_name`；空值会变成 `nil`。 | +| `tenant_id` | 字符串 | Store 操作使用的 tenant；空值会变成 `default`。 | +| `backend_id` | 字符串 | 来自 `kv_events_backend_id`。 | +| `medium` | 字符串或 `nil` | `stored` 和 `removed` 的对象位置；`cleared` 使用 `nil`。 | +| `dp_rank` | 无符号整数 | 来自 `kv_events_dp_rank`。 | + +当 `kv_events_emit_legacy_compat=true` 时，每个 map 还会包含 `type`： +`BlockStored`、`BlockRemoved` 或 `AllBlocksCleared`，并与 `event_type` +对应。 + +`stored` 和 `removed` map 还包含： + +| 字段 | MessagePack 形式 | 来源和用途 | +|---|---|---| +| `group_id` | 字符串或 `nil` | 如果 connector 中带有 `group:N`，则使用解析结果；否则使用 Store 对象的 group 字符串。空值会变成 `nil`。 | +| `seq_hashes` | 包含零个或一个无符号整数的数组 | 从对象 key 解析出的低 64 位。无法解析的 key 会生成空数组。 | +| `base_block_idx` | `nil` | connector key 不记录该块在 token 链中的深度。 | +| `object_key` | 字符串，按配置决定是否存在 | 完整 Mooncake key，仅当 `kv_events_emit_object_key=true` 时发送。 | +| `block_hashes` | 数组，按配置决定是否存在 | `seq_hashes` 的旧版副本，仅当兼容字段开启时发送。 | + +`stored` map 还包含 `parent_hash` 和 `token_ids`，两者都是 `nil`，因为 +当前 connector key 不含这两个值。开启旧版字段时，它还包含 +`parent_block_hash=nil`。`removed` map 不包含这些只属于 stored 的字段。 +`cleared` map 不包含 group、哈希、对象或并行拓扑字段。 + +如果成功识别 connector key，发布端还会加入该 key 实际提供的字段： + +| 字段 | 何时存在 | +|---|---| +| `connector_block_hash` | 完整块哈希文本是有效十六进制，并成功生成了 `seq_hashes[0]`。 | +| `cache_prefix` | connector 模型名前存在文本。 | +| `tp_rank` | 解析出了 vLLM 张量并行 rank。 | +| `head_or_tp_rank` | 解析出了 vLLM Ascend head 或张量并行 rank。 | +| `pcp_rank` | 解析出了 prefill 上下文并行 rank。 | +| `dcp_rank` | 解析出了 decode 上下文并行 rank。它不是 `dp_rank`。 | +| `pp_rank` | 解析出了流水线并行 rank。 | +| `layer_id` | Ascend 按层 key 中包含层编号。 | + +## 理解 `stored`、`removed` 和 `cleared` + +这三个事件名表示逻辑上的可用性，不表示物理副本的数量或地址： + +| 事件 | Mooncake 发布端表达的含义 | 当前 Conductor 所需字段 | +|---|---|---| +| `stored` | 该对象可从指定的 `cpu` 或 `disk` 介质读取。重复成功的 Put 或 Upsert 提交可能再次上报相同可用性。 | `backend_id`、非空的 `tenant_id` 和 `model_name`、正数 `block_size`、受支持的介质和 group、`object_key`，以及可用的完整 `connector_block_hash`。`lora_name` 可以为空。`seq_hashes` 可以为空；如果有值，则必须与完整哈希一致。 | +| `removed` | 该对象已不能从指定介质读取。它在另一种介质上仍可能可用。 | `backend_id`、受支持的介质和 group，以及 `object_key`。可以不带完整哈希，因为 Conductor 会查找此前 `stored` 保存的对象记录。 | +| `cleared` | 该发布端的 `backend_id` 与事件 `tenant_id` 下的所有对象都已清空。它同时作用于 CPU 和 Disk，并使用 `medium=nil`。 | 非空的 `backend_id` 和 `tenant_id`，且不能带对象字段。 | + +开启对象 key 发送后，即使发布端无法完整解析某个 key，也仍可能发送对应的 +`stored` 事件。这种事件没有可用的 `connector_block_hash`；当前 Conductor +会记录拒绝日志，不会把它加入共享缓存索引。 + +## 使用 Conductor 可匹配的 connector key + +Mooncake 不会给 Store 客户端调用新增 KV Event 参数，而是从 vLLM connector +生成的 key 中读取信息。可识别的 vLLM 格式如下： + +```text +[cache_prefix@]model_name@tp_rank:N@pcpN@dcpN@pp_rank:N@group:N@block_hash_hex +[cache_prefix@]model_name@tp_rank:N@pcpN@dcpN@pp_rank:N@block_hash_hex +``` + +第二种格式适用于还没有 `group:N` 字段的旧 connector 版本。可识别的 vLLM +Ascend 格式如下： + +```text +model_name@pcpN@dcpN@head_or_tp_rank:N@pp_rank:N@block_hash_hex +model_name@pcpN@dcpN@head_or_tp_rank:N@block_hash_hex@layer_id +``` + +分隔符是没有转义机制的 `@`。在前两种格式中，`tp_rank` 前一个片段会被 +当作模型名，更前面的片段会成为 `cache_prefix`。Mooncake Store 不会拒绝 +未知或格式错误的 key。发布端仍可将它们作为 `object_key` 发出，但它们无法 +提供 Conductor 接受 `stored` 时所需的完整 connector hash。 + +对 Conductor 而言，`block_hash_hex` 必须是表示至少八个字节的有效十六进制 +文本：去掉可选的 `0x` 前缀后，长度至少为 16 个字符，并且字符数必须为 +偶数。发布端在 `connector_block_hash` 中保留完整文本，并把最右侧 16 个 +十六进制字符转成 `seq_hashes[0]`。Conductor 按大端序读取最后八个字节， +得到 `/query` 使用的 64 位查询值。 + +完整哈希仍然有用。Conductor 同时保存完整哈希和对象 key，因此两个完整哈希 +不同、但最后八个字节相同的对象仍可以分别删除。查询只使用 64 位值，所以只要 +其中任一对象存在，就会报告可能命中。可接受的编码和不一致处理方式见 +[订阅端哈希规则](./subscriber-guide.md#转换收到的哈希)。 + +`cache_prefix`、各种 rank 字段和 `layer_id` 用来描述 connector 的 key +空间。发布端无法判断多少层或多少个并行部分才组成一个完整块；当前 Conductor +也不会检查这种完整性。 + +## 了解 CPU 和 Disk 变化 + +Master 会把副本信息归纳成两个可用性结果：对象能否从 CPU 读取，以及能否从 +Disk 读取。它会按以下规则发出事件： + +| Store 操作 | 事件结果 | +|---|---| +| Put 或 Upsert 成功提交 | 针对当前每种可用介质各发送一个 `stored`。 | +| Upsert 期间旧值变得不可读 | 先针对旧可用性发送 `removed`，提交时再发送 `stored`。 | +| Copy、Move、offload、promotion、eviction 或副本清理 | 某种介质首次可用时发送 `stored`；该介质最后一个可用副本消失时发送 `removed`。只要介质仍可用，仅改变副本数量或位置就不会产生可用性变化。 | +| Remove、BatchRemove 或正则删除 | 针对对象此前可用的每种介质各发送一个 `removed`。 | +| RemoveAll | 先逐对象发送 `removed`；仅当 tenant 中没有剩余对象时再发送 `cleared`。 | +| 新 Put 未提交且失败 | 不发送事件。 | + +如果一个对象同时可从 CPU 和 Disk 读取，发布端会针对两种介质分别发送事件。 + +## 规划事件丢失 + +发布过程是异步的。正数 `kv_events_queue_capacity` 会限制等待队列长度。队列 +已满时，发布端会移除最旧的等待事件、增加 `dropped_events`，并保留一个 +传输序号，让订阅端能够看到序号空缺。ZMQ 发送失败也会按受影响的事件数增加 +`dropped_events`。 + +发布端只提供 PUB socket：它没有 replay endpoint，无法重发漏掉的事件； +Conductor 连接时，它也不会发送此前已缓存对象的清单。发布端进程只在内存中 +保存精简的对象与介质对应关系，不会持久化。后续 Store 变化可以为已有对象 +生成新事件，但不能借此还原完整的启动状态。 + +传输序号跳跃时，Conductor 会警告并保留当前缓存记录；警告本身不会补回漏掉的 +变化。在 Conductor 中看到所需注册之前，应暂停产生缓存的流量。完整启动清单 +见[订阅指南](./subscriber-guide.md#检查-mooncake-到-conductor-的配置)。 + +## 维护者源码索引 + +消息构建和队列位于 `mooncake-store/src/kv_event/kv_event_publisher.cpp`， +connector key 解析位于 `mooncake-store/include/kv_event/key_util.h`。 +Master 参数和配置加载位于 `mooncake-store/src/master.cpp`， +`GET /kv_events/status` 位于 +`mooncake-store/src/master_admin_service.cpp`。对应测试数据在 +`mooncake-store/tests/kv_event_publisher_test.cpp` 中。 diff --git a/docs/source/zh/design/kv-event/subscriber-guide.md b/docs/source/zh/design/kv-event/subscriber-guide.md new file mode 100644 index 0000000000..66d452ec0f --- /dev/null +++ b/docs/source/zh/design/kv-event/subscriber-guide.md @@ -0,0 +1,252 @@ +# 将 KV Event 来源连接到 Conductor + +[English](../../../design/kv-event/subscriber-guide.md) + +Conductor 目前可以读取两种键值（KV）事件格式：vLLM 引擎事件和 Mooncake +共享缓存池事件。本页帮助你选择正确的注册来源类型、满足 Conductor 检查的 +字段，并准确理解 Store、Remove、Clear 和 unregister 会修改什么。本文也会 +介绍哈希转换和漏掉事件时的限制。 + +## 选择注册来源类型 + +每个 endpoint 必须且只能注册为下面一种区分大小写的 `type`： + +- 推理引擎上报自身 GPU 缓存时使用 `"vLLM"`； +- Mooncake Master 上报共享 CPU 或 Disk 对象时使用 `"Mooncake"`。 + +Conductor 根据注册时记录的 type 选择 MessagePack 解析方式。即使两种发布端 +都使用空 topic，topic 文本也不能选择或覆盖解析方式。接收消息的 endpoint +同时用于标识消息来源、限制清理范围和记录日志。注册字段与示例见 +[HTTP API 参考](../conductor/indexer-api-design.md)。 + +## 对比 vLLM 和 Mooncake + +| 问题 | `vLLM` 来源 | `Mooncake` 来源 | +|---|---|---| +| 注册 `type` | 必须是 `vLLM`。 | 必须是 `Mooncake`。 | +| 事件名 | `type` 中的 `BlockStored`、`BlockRemoved`、`AllBlocksCleared`。 | `event_type` 中的 `stored`、`removed`、`cleared`；可以带对应的旧版 `type`。 | +| 批次时间戳 | 以秒为单位的有限 MessagePack 浮点数。事件中不要求单独的时间戳字段。 | 以毫秒为单位的非负 MessagePack 整数。每个事件的 `timestamp` 都必须是同一个整数。 | +| 接受的缓存位置 | `GPU`，不区分大小写。其他值会记录警告并被忽略。 | `CPU` 或 `Disk`，不区分大小写。其他值会记录警告并被忽略。 | +| Tenant、模型、LoRA、块大小 | 可信的注册信息提供这些缓存共享字段。Store 事件中的块大小和低秩适配（LoRA）名称只用于核对。 | Store 事件提供 `tenant_id`、`model_name`、`lora_name` 和 `block_size`。它们必须对应一个已有的 vLLM 缓存共享范围，并使用已注册的 hash profile。 | +| 数据并行 rank | 可信的注册信息提供引擎 rank。批次中的非 `nil` rank 必须与它一致，否则整批都会被拒绝。 | 批次和事件中的 `dp_rank` 只用于排查问题，不会生成 GPU 记录或查询实例。 | +| Group | `group_idx` 可以不存在、为 `nil` 或为 `0`。 | `group_id` 可以是 `nil`、空字符串或十进制字符串 `"0"`。 | +| 对象字段 | 使用 `block_hashes`，没有 Mooncake 对象 key。 | Store 需要 `object_key` 和 `connector_block_hash`；Remove 需要 `object_key`；Clear 不携带这两个字段。 | +| `/query` 的 `instances` | 已注册引擎按 `instance_id` 出现，其中包含已注册的 rank。 | Mooncake 来源不会成为一个 instance。它的 CPU 或 Disk 信息显示在每个兼容的 vLLM 引擎下。 | + +决定 CPU 或 Disk 信息能否被某个引擎共用的四个字段是 `tenant_id`、模型、 +`lora_name` 和 `block_size`。它们如何影响查询结果，见 +[架构页](../conductor/conductor-architecture-design.md#哪些缓存可以共用)。 + +## 匹配前缀哈希配置 + +KV Event 不携带 `PYTHONHASHSEED`，也不携带完整的第一个 parent 摘要，因此注册 +信息是可信的部署声明。同一兼容缓存范围内的每个 vLLM 和 Mooncake 注册项都必须 +提供同一段准确的 `python_hash_seed` 字符串。Conductor 把这段文本编码为规范 +CBOR，再计算 SHA-256 派生第一个 parent 使用的 `root_digest`；调用方不注册这个 +摘要。 + +种子必须是字面量 `random`，或者数值在 `0..4294967295` 范围内的 ASCII 十进制 +文本。准确文本会影响结果：`"0"` 和 `"00"` 派生不同的根摘要。未设置 +`PYTHONHASHSEED` 不受支持，因为此时 vLLM 会选择 Conductor 无法复现的随机根 +字节。明确设置的文本 `random` 受支持，并按这段准确字符串计算哈希。vLLM 的 +`--seed` 只控制模型和采样随机性，与这个前缀缓存标识无关。 + +对于种子零配置，所有兼容的 vLLM 进程都使用： + +```bash +PYTHONHASHSEED=0 vllm serve test-model \ + --enable-prefix-caching \ + --prefix-caching-hash-algo sha256_cbor +``` + +注册时把 `python_hash_seed` 设为 `"0"`。Conductor 会通过 `/services` 和 +`/global_view` 返回派生的 `root_digest` +`4e1195df020de59e0d65a33a4279f1183e7ae4e5d980e309f8b55adff2e61c3e`， +供部署对照检查。 + +## 满足 vLLM 事件要求 + +vLLM payload 必须是 +`[timestamp_seconds, [event_maps], data_parallel_rank]`。最后一项可以是 +非负整数或 `nil`。每个事件 map 遵循以下规则： + +| 事件 | 可识别字段与检查 | 对索引的修改 | +|---|---|---| +| `BlockStored` | 必须包含 `block_hashes`、`parent_block_hash`、`token_ids`、`block_size`、`lora_id`、`medium` 和 `lora_name`。`lora_id` 必须是 `nil`；`block_size` 和 `lora_name` 必须与注册信息一致；`medium` 必须是 GPU。可选的 `group_idx` 必须是 `0` 或 `nil`。 | 把给定哈希加入已注册 endpoint、引擎及其数据并行（DP）rank 的 GPU 信息。 | +| `BlockRemoved` | 必须包含 `block_hashes` 和 `medium`；可选的 `group_idx` 必须是 `0` 或 `nil`。不能带已识别的 stored 专用字段。 | 只删除该已注册 endpoint、引擎和 rank 对这些哈希的 GPU 记录。 | +| `AllBlocksCleared` | 在已识别的 vLLM 字段中只能携带 `type`。 | 删除该已注册 endpoint、引擎和 rank 的全部 GPU 信息，保留其他引擎以及所有共享 CPU/Disk 信息。 | + +`block_hashes` 是一个数组，每一项可以是无符号整数或 MessagePack 二进制 +字符串。Conductor 会读取 parent hash 和 token ID，但不会用它们替换或重新 +计算 `block_hashes`。 + +`BlockStored` 还识别可选的 `extra_keys`、`kv_cache_spec_kind` 和 +`kv_cache_spec_sliding_window`。如果提供 `extra_keys`，它必须是 `nil`， +或者为每个 block hash 提供一项 `nil`/数组。Conductor 只接受 +`kv_cache_spec_kind="full_attention"`，并且不接受非 `nil` 的 sliding-window +值。未知 map key 会被忽略；已识别 key 重复或类型错误时，只会使该事件无效。 + +## 满足 Mooncake 事件要求 + +Mooncake payload 必须是 +`[timestamp_ms, [event_maps], data_parallel_rank]`。最后一项可以是非负整数 +或 `nil`。每个事件 map 都必须包含以下通用字段： + +| 字段 | 接受的形式 | +|---|---| +| `event_id` | 无符号整数。它会写入日志，但不用于排序或跳过事件。 | +| `timestamp` | 与外层非负 `timestamp_ms` 相同的有符号整数。 | +| `event_type` | `stored`、`removed` 或 `cleared`。 | +| `model_name` | 字符串或 `nil`。Store 最终必须得到非空模型名。 | +| `block_size` | 有符号整数或 `nil`。Store 要求它是正数。 | +| `additional_salt` | 字符串或 `nil`。Conductor 会读取，但不会用于缓存共享字段或哈希查询。 | +| `lora_name` | 字符串或 `nil`；`nil` 表示基础模型。 | +| `tenant_id` | 字符串。Store 和 Clear 要真正修改记录时，需要非空 tenant。 | +| `backend_id` | 字符串；Store、Remove 和 Clear 都要求它非空。 | +| `medium` | 字符串或 `nil`；Store 和 Remove 必须指定 CPU 或 Disk。 | +| `dp_rank` | 非负有符号整数，只用于排查问题。 | + +如果带有旧版 `type`，它必须与 `event_type` 一致。每种事件还有以下规则： + +| 事件 | 解析时必需的字段 | 真正修改索引还需要什么 | +|---|---|---| +| `stored` | `group_id`、`seq_hashes`、`base_block_idx`、`parent_hash` 和 `token_ids`。后三个字段可以是 `nil`；`seq_hashes` 可以是 `[]`。 | 受支持的 group 和介质、完整的 tenant/model/LoRA/block size、`object_key`，以及可用的完整 `connector_block_hash`。如果有 sequence hash，它必须与完整哈希一致。 | +| `removed` | `group_id`、`seq_hashes` 和 `base_block_idx`；`seq_hashes` 可以是 `[]`。不能包含 stored 专用的 parent 和 token 字段。 | 受支持的 group 和介质，以及 `object_key`。Conductor 会查找 Store 保存的准确对象记录；可选的完整哈希或 sequence hash 必须与该记录一致。 | +| `cleared` | 只包含通用字段。不能带任何已识别的对象、哈希、group、parent、token 或并行拓扑字段。 | 非空的 `backend_id` 和 `tenant_id`。该事件会清除这个来源 endpoint 下匹配的 CPU 和 Disk 对象记录。 | + +对象事件还可以包含 `cache_prefix`、`tp_rank`、`head_or_tp_rank`、 +`pcp_rank`、`dcp_rank`、`pp_rank` 和 `layer_id`。这些字段的类型见 +[发布指南](./publisher-design.md#读取通用事件字段)。如果带有旧版 +`block_hashes`，它必须等于 `seq_hashes`；如果 stored 事件带有 +`parent_block_hash`，它必须等于 `parent_hash`。 + +## 转换收到的哈希 + +Conductor 会把两种来源的哈希转成同一种无符号 64 位查询值。它不会再次计算 +事件中的 token 或 parent 字段。 + +| 收到的哈希 | 转换方式 | +|---|---| +| vLLM 无符号整数 | 它已经是查询值，直接使用。 | +| vLLM MessagePack 二进制字符串 | 至少需要八个字节，然后按大端序读取最后八个字节。 | +| Mooncake `connector_block_hash` | 去掉可选的 `0x`，接受大小写十六进制，要求长度为偶数且至少表示八个字节，将文本统一为小写，然后按大端序读取最后八个解码后的字节。 | + +Mooncake Store 的 `seq_hashes` 可以为空，也可以只包含一个值；该值必须等于 +从 `connector_block_hash` 得到的查询值。值不同或多于一个时，Conductor 会 +拒绝该事件，不修改已保存的对象记录或缓存索引。Remove 如果带有完整哈希或 +sequence 值，也必须与此前的 Store 记录一致。 + +虽然 `/query` 只使用最后八个字节，Conductor 仍会保存完整 Mooncake 哈希和 +对象 key。如果两个对象的完整哈希不同，但最后八个字节相同，它们都可以提供 +同一个可能的查询命中；删除其中一个不会删除另一个对象的记录。查询端如何生成 +哈希，见 [token 块如何变成查找值](../conductor/conductor-architecture-design.md#token-块如何变成查找值)。 + +## 理解每种事件会修改什么 + +| 事件 | 准确的修改范围 | +|---|---| +| vLLM `BlockStored` | 在已注册的 tenant/model/LoRA/block size 范围内，为注册来源 endpoint、`instance_id` 和 DP rank 加入 GPU 哈希。 | +| vLLM `BlockRemoved` | 只删除该引擎和 rank 上列出的 GPU 哈希。重复 Remove 不会产生额外影响。 | +| vLLM `AllBlocksCleared` | 删除该引擎和 rank 的全部 GPU 哈希，不会影响其他 rank、引擎或 Mooncake 对象。 | +| Mooncake `stored` | 保存一条由来源 endpoint、`backend_id`、`tenant_id`、`object_key` 和 CPU/Disk 介质共同确定的记录。完全相同的重复事件不会产生额外影响；同一对象和介质上的冲突记录会被拒绝。 | +| Mooncake `removed` | 查找上述准确的对象与介质记录。未知对象不会产生影响，删除时绝不会只按 64 位值搜索。 | +| Mooncake `cleared` | 只删除上报 endpoint、`backend_id` 和 `tenant_id` 下保存的 CPU 和 Disk 记录，保留 GPU 信息以及其他 endpoint、backend 和 tenant。 | + +有效批次中的 Mooncake 事件按接收顺序执行。Clear 会删除它之前建立的匹配记录； +同一批次中后续的 Store 仍可重新加入可用性。Conductor 不会按 `event_id` 排序， +也不会因为 ID 重复而自动忽略事件。 + +## 处理无效消息 + +Conductor 要求恰好三帧 ZeroMQ（ZMQ）消息：topic、八字节大端序 sequence， +以及一个 MessagePack payload。帧数错误、sequence 帧错误、MessagePack 值 +无效、外层不是三项结构、批次时间戳无效、事件不是数组，或者批次 DP 类型错误 +时，Conductor 会丢弃整条消息，不处理其中任何事件。vLLM 批次中的非 `nil` +DP rank 如果与注册信息冲突，同样会使整批被拒绝。 + +外层 payload 有效后，每个 map 会分别解析和执行。格式错误或未通过内容检查的 +事件会被跳过，但它前后的有效事件仍按接收顺序生效；不会因为后面的错误撤销 +前面已完成的修改。未知字段会被忽略，方便发布端增加新的可选数据；已识别字段 +重复或类型错误时，只会拒绝该事件。 + +修改 topic 不能让某个来源切换到另一种解析方式。即使 payload 的形状看起来像 +另一种来源，Conductor 也不会改用另一种方式重试。 + +## 清理一个来源 + +调用 `POST /unregister` 时，应使用标识该注册的同一组 `instance_id`、 +`tenant_id` 和 `dp_rank`。Conductor 会先停止该订阅并等待其退出，再删除它 +贡献的记录，然后才返回成功。旧订阅中已经排队的事件不能在清理完成后重新写入 +索引。 + +对 vLLM 而言，unregister 会删除该 endpoint/引擎/rank 的 GPU 记录和 rank +注册。对 Mooncake 而言，它会删除该注册 endpoint 在所有事件上下文和 backend +中保存的 CPU、Disk 对象记录，同时保留其他 Mooncake endpoint 和全部 vLLM +GPU 记录。Mooncake 注册中的 `instance_id` 和 `dp_rank` 仅用于在 `/services` +中标识服务以及 unregister；它们不会生成查询实例，也不会覆盖事件中的 +tenant/model/LoRA/block size。 + +## 规划序号空缺和重连 + +Conductor 会记录每个订阅最后收到的传输序号。如果后续序号向前跳跃，Conductor +会记录警告并保留当前缓存记录。这个空缺不会立即触发重发请求，也不会让 +Conductor 自动删除可能已经过期的记录。 + +断开连接后，Conductor 会重新连接实时订阅。只有配置了 `replay_endpoint` 且 +已知此前序号时，它才会请求从下一个序号开始的消息。发出请求并收到响应，也不 +保证每个漏掉的缓存变化都能恢复。空 replay endpoint 是有效配置，此时只创建 +实时订阅。 + +当前 Mooncake 发布端没有 replay endpoint，也不会发送 Conductor 连接前已经 +缓存的对象清单。因此 Mooncake 来源应使用空 `replay_endpoint`；如果缓存流量 +更早开始，就不能认为 Conductor 的初始视图是完整的。Conductor 也不会把较旧或 +重复的传输序号、重复的 `event_id` 当成重复消息而拒绝，而是会处理收到的事件。 + +## 检查 Mooncake 到 Conductor 的配置 + +在允许请求创建缓存对象前，请按以下顺序操作： + +1. 使用 `-DENABLE_KV_EVENTS=ON` 构建 Mooncake Store，启用发布端，保持 + `kv_events_emit_object_key=true`，并确认 `GET /kv_events/status` 返回 + `"enabled":true`。 +2. 先注册至少一个 `vLLM` 引擎。它的 `tenant_id`、`modelname`、 + `lora_name` 和 `block_size` 必须等于 Mooncake Store 事件将携带的值。 + 使用上文明确的 `PYTHONHASHSEED` 和 `--prefix-caching-hash-algo` 设置启动 + 每个引擎。确认 `/global_view` 中能看到该引擎和每个预期 rank。 +3. 该 vLLM 范围和 `Mooncake` 注册必须使用同一份完整 `hash_profile`： + `strategy`、`algorithm`、准确的 `python_hash_seed` 和 `index_projection` + 必须完全相同。Conductor 会派生并验证根摘要；任一发布端的事件都不会提供它。 +4. 为 Mooncake 发布端配置非空 `backend_id`、可用的模型备用值、正数块大小， + 以及相同的 LoRA 名称。注意，从 `object_key` 解析出的模型会覆盖备用值。 +5. 每个对象 key 都应使用可识别的 connector 格式，并包含完整、有效的十六进制 + 哈希。Conductor 能接受的 `stored` 必须同时包含 `object_key` 和 + `connector_block_hash`。 +6. 不使用 group 或只使用 group zero，并且 Mooncake 来源只上报 CPU 或 Disk + 可用性。Conductor 会忽略其他介质，并拒绝非零 group。 +7. 注册 Mooncake endpoint，然后在 `/services` 中检查所有预期的 vLLM 和 + Mooncake 订阅。在 `/services` 和 `/global_view` 中同时核对配置种子和派生 + 根摘要。使用静态配置时，各订阅会同时启动；两项检查都通过前，应暂停产生 + 缓存的流量。 +8. 放开缓存流量，确认 Mooncake 发布端计数增加，再查询一个已知 token 前缀。 + 共享 `cpu` 或 `disk` 计数必须出现在兼容的 vLLM instance 下，而不是 + Mooncake 注册名下。 + +`/services` 只能说明 Conductor 接受了注册，不能证明 ZMQ 发布端已经送达事件， +也不能证明订阅前创建的对象已经被记录。完整的注册和查询命令见 +[Conductor 使用指南](../conductor/usage.md)。 + +## 了解当前完整性限制 + +Mooncake connector key 可以携带层编号，以及张量并行（TP）、prefill 上下文 +并行（PCP）、decode 上下文并行（DCP）和流水线并行（PP）rank。Conductor +会读取这些字段，但目前不会先检查是否已经收到所有必需的层或并行部分，再报告 +块可用。它也不会把 `cache_prefix` 或 Mooncake `additional_salt` 作为四个 +缓存共享字段之一。 + +## 维护者源码索引 + +消息解析实现在 `mooncake-conductor/src/zmq/msg_decoder.cpp` 中；传输序号和 +重连处理位于 `mooncake-conductor/src/zmq/zmq_client.cpp`。事件检查和清理位于 +`mooncake-conductor/src/kvevent/event_handler.cpp` 和 `event_manager.cpp`。 +当前解析和端到端测试数据位于 `mooncake-conductor/tests/msg_decoder_test.cpp` +和 `event_ingest_integration_test.cpp`。