[Store] Add opt-in replica placement shadow evaluator

[catyans]

What this draft does

This adds an opt-in, non-actuating Dynamic Replica placement SHADOW path for Mooncake Store. It turns client-facing Get observations plus bounded Master tier signals into placement intents and fixed-cardinality metrics, but deliberately does not submit Copy/Move tasks or mutate replica metadata.

Motivation: the Store V3 roadmap (#1035) lists cache scheduling with migration/promotion/demotion, hot-data detection with more replicas, and Data Replica: Dynamic Replication. This draft isolates the decision and observability layer so its safety and signal semantics can be reviewed before any actuator is introduced.

The stack is split into five reviewable commits:

1. tier-aware pure placement policy with a COMPLETE-replica safety floor;
2. standalone CountMinSketch-based SHADOW evaluator and immutable signal snapshots;
3. explicit profile configuration, config propagation, and client Get integration;
4. bounded collectors for MEMORY, LOCAL_DISK, NoF SSD, and REMOTE_STORE signals;
5. fixed-cardinality Prometheus metrics and integration/fault tests.

Safety and compatibility

• Default is fully off: an empty --replica_placement_shadow_config does not construct the evaluator.
• The profile must explicitly define COLD/WARM/HOT × MEMORY/LOCAL_DISK/NOF_SSD/REMOTE_STORE targets.
• Missing, source-down, stale, unknown-capacity, full, and unhealthy signals fail conservatively.
• Pending replicas/add intents suppress duplicate additions but never satisfy min_complete_replicas for deletion safety.
• External snapshots become authoritative after the first accepted publication; the automatic collector cannot overwrite them.
• Client-facing single/batch Get is observed after copying descriptors; admin Get is excluded.
• Metric labels are enum-only. No key, tenant, or endpoint can enter the public metric API.
• This PR contains no reconciler, scheduler, target/source endpoint selection, Copy/Move submission, or metadata mutation.

Validation

All product tests were built and run on two 192-vCPU Lingjun H20 cluster nodes from detached exact product HEAD 174e2240429f6e0a573e05c2d47c88394425ba1d (upstream base 3bbd9b2c36924626207b088b525d036b1982ca05). No local-to-remote scp was used.

• final Release smoke on node100 and node102: 35 cases per node
• Release 100× on node100: 3,500 cases
• Release 100× on node102: 3,500 cases
• ASan+LSan 20× on node100: 700 cases, 0 finding
• TSan concurrent filters 10× on node100: 30 tests, 0 product finding
• startup-level default-off / valid / invalid profile checks: rc 124 / 124 / 1
• experiment-only in-process Get benchmark, 3 trials / 45M measured Gets, every run rc 0

The TSan run uses one narrow suppression for a pre-existing libglog lazy-init race: race:google::LogMessageTime::CalcGmtOffset. The suppression and limitation will be included with the detailed evidence comment.

Scope boundary

This draft proves configuration, signal semantics, intent generation, bounded metrics, concurrency behavior, and control-plane overhead. It does not claim improved hit rate, TTFT, bandwidth, recovery time, or storage cost. Those require the later bounded reconciler/executor and end-to-end workload experiments.

Follow-up work (not implemented here)

• deterministic target/source selection;
• reconciliation ownership and HA behavior;
• task admission, atomic reserve, submit, and exact rollback;
• cooldown/hysteresis and anti-oscillation policy;
• real NoF/SPDK health/capacity integration;
• end-to-end scale-out/scale-in/failure scorecard across DRAM, SSD, and remote storage.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[catyans]

Exact-head cluster validation report

This comment records the final post-rebase evidence for product HEAD 174e2240429f6e0a573e05c2d47c88394425ba1d on upstream base 3bbd9b2c36924626207b088b525d036b1982ca05. Older 85004d65 and f620fafc runs are archived separately and are not used below.

Environment

	node100	node102
host	lingjun-100 (8.130.214.133 / 192.168.0.40)	lingjun-102 (8.130.10.114 / 192.168.0.41)
CPU	2 sockets × 48 cores × 2 threads, Intel Xeon Platinum 8575C, 192 vCPU	same
GPU	8 × NVIDIA H20-3e (not used by these control-plane tests)	same cluster SKU
kernel	5.10.134-16.3.al8.x86_64	same
container	Ubuntu 24.04.3 LTS, codex_mooncake_pr_eval	Ubuntu 24.04.3 LTS, yanshu_bwa
toolchain	GCC 13.3.0 / CMake 3.28.3	same

Both nodes fetched the public fork branch directly and detached-checkout the exact product SHA. No local-to-remote scp was used.

Test matrix

One full suite contains 14 pure-policy, 7 standalone-evaluator, and 14 Master integration cases (35 cases).

run	repeats	cases	result
node100 Release (-O3, metrics on)	100	3,500	PASS
node102 Release (-O3, metrics on)	100	3,500	PASS
node100 ASan+LSan	20	700	PASS, 0 finding
node100 TSan concurrent filters	10 each	30	PASS, 0 product finding
final Release smoke, both nodes	1 each	70	PASS
total	—	7,800	0 failed

TSan stress volume:

• policy: 32 threads × 10,000 Plan × 10 repeats = 3.2M calls;
• evaluator: 8 threads × 2,000 Observe × 10 repeats = 160k calls;
• publisher: 100 publishes × 10 repeats = 1,000 publishes;
• Master: 8 threads × 500 Get × 10 repeats = 40k client-facing Gets.

The only suppression is a narrow pre-existing libglog lazy-init race: race:google::LogMessageTime::CalcGmtOffset.

Startup and fault behavior

scenario	result
default-off, no profile	starts normally, no SHADOW log; timeout rc 124
valid explicit 12-target profile	starts normally with auto_collect_master_signals=1; timeout rc 124
invalid profile missing one target	rejected before service startup; rc 1

signal/fault	add intent	behavior
missing snapshot	0	conservative
source unavailable	0	conservative
stale snapshot	0	conservative
LOCAL_DISK capacity unknown	0	REQUIRED_TIER_SIGNAL_UNKNOWN
LOCAL_DISK known full	0	REQUIRED_TIER_UNAVAILABLE
REMOTE_STORE path missing	0	REQUIRED_TIER_UNHEALTHY
LOCAL_DISK has reported free capacity	1	expansion opportunity detected
REMOTE_STORE path reachable with free space	1	expansion opportunity detected

Fixed cardinality

The public metrics have a hard maximum of 40 series:

• observations {temperature,status}: 3 × 4 = 12;
• add intents {temperature,tier}: 3 × 4 = 12;
• remove intents {temperature,tier}: 3 × 4 = 12;
• degraded {reason}: 4.

The serialization test injects a unique secret key and verifies that neither it nor tenant/endpoint labels appear in Prometheus output.

In-process Get overhead

The benchmark is experiment-only (16332f00) and is not part of the product commit stack. It runs on node100 Release -O3, pinned to CPU 150, with 20k warmup Gets per mode, 25 × 100k measured Gets per mode, six rotating profile/mode contexts, and three independent trials (45M measured Gets total). Every Get must succeed.

profile	mode	trial 1 p50	trial 2 p50	trial 3 p50	median-of-trials p50	absolute delta vs off
quiescent	off	350.980 ns	354.962 ns	354.135 ns	354.135 ns	baseline
quiescent	external snapshot	725.130 ns	724.443 ns	726.253 ns	725.130 ns	+370.995 ns
quiescent	auto collector	749.926 ns	750.375 ns	752.342 ns	750.375 ns	+396.240 ns
active intents	off	349.701 ns	353.582 ns	352.958 ns	352.958 ns	baseline
active intents	external snapshot	848.023 ns	851.726 ns	849.269 ns	849.269 ns	+496.311 ns
active intents	auto collector	799.633 ns	801.836 ns	802.371 ns	801.836 ns	+448.878 ns

xychart-beta
    title "Replica placement SHADOW in-process Get p50"
    x-axis [quiet-off, quiet-external, quiet-auto, active-off, active-external, active-auto]
    y-axis "ns/Get" 0 --> 900
    bar [354.135, 725.130, 750.375, 352.958, 849.269, 801.836]

Loading

The absolute measured increment is 0.371–0.496 µs/Get. Relative deltas are +104.8% (quiescent external), +111.9% (quiescent auto), +140.6% (active external), and +127.2% (active auto), because the in-process off baseline is only ~0.35 µs/Get. active external does more work than active auto in this cluster setup because the all-AVAILABLE external snapshot emits two add-intent metric increments, while the auto collector conservatively marks the unmounted LOCAL_DISK tier unavailable and emits one. The quiescent rows are the fair collector-gate comparison.

The benchmark isolates control-plane cost; it does not include RPC, network, LLM serving, or TTFT and therefore is not an end-to-end performance claim.

Evidence integrity and scope

evidence	bytes / rows	passing summaries	SHA-256
node100 Release100 log	1,354,338 bytes	300	98b801893db1213c63c8f2948a6a4880e567ffb27a956015a7a09de4782e2017
node102 Release100 log	1,358,938 bytes	300	1f3a4aeaaee864ab9e6cb133ea3f2aa20d9d306e730ce25f7a81f3e0eff5f213
node100 ASan+LSan20 log	273,341 bytes	60	28d0794164aa6639db5528cd22ab6c1cddf918720fcb7ad9d07c0e95d101ffcc
node100 TSan10 final log	22,992 bytes	30	7df9e060abca8eaffcf0e30d019c7ef4e33075bcb35179659a92634332bebe08
node100 benchmark raw log	38,955 bytes	3 × 15M Get	dbf0d67e1d81129e2880989ce537ae80d279dd535137e344b354ec233ae3ec06
benchmark samples CSV	451 rows / 450 samples	—	c23b4edb61e245030f2f2f3db0944e5464e5aec16f54893ed7f71b6c62ccb085
benchmark summary CSV	19 rows / 18 summaries	—	034f6f5bf2178ffec3feba6ccc87d29e4e4c8a99323b7df750c996673ba36ba7

The complete 25-file checksum manifest has SHA-256 90d055c9dc1bac54170ee9d8ab7ed24de53834603904457df69570b05b61a21a. All product/benchmark rc files are 0; startup rc files are 124/124/1 as documented. Both final environment snapshots report TRACKED_DIFF_FILES=0, and node100 was restored from experiment HEAD 16332f00faccf2c7ed40d00aa8e954ca218dffbd to product HEAD 174e2240429f6e0a573e05c2d47c88394425ba1d after the benchmark. An earlier TSan orchestration attempt exited with rc 66 before starting any test because its temporary suppression file was absent; it is excluded. Only the complete tsan10-final run is counted above.

The results prove opt-in semantics, conservative signal handling, Get-path integration, exact concurrent counters, bounded metric cardinality, sanitizer cleanliness within the stated suppression, and measured in-process overhead. They do not prove active scale-out/scale-in, real NoF/SPDK behavior, HA ownership, hit-rate/TTFT/bandwidth/recovery improvements, or storage-cost reduction; those remain follow-up work.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.49147% with 88 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mooncake-store/src/master.cpp	11.84%	67 Missing ⚠️
mooncake-store/src/master_service.cpp	95.20%	7 Missing ⚠️
...cake-store/tests/replica_placement_policy_test.cpp	97.68%	4 Missing ⚠️
mooncake-store/src/replica_placement_policy.cpp	95.89%	3 Missing ⚠️
mooncake-store/include/master_config.h	50.00%	2 Missing ⚠️
mooncake-store/src/replica_placement_shadow.cpp	98.31%	2 Missing ⚠️
mooncake-store/src/master_metric_manager.cpp	96.29%	1 Missing ⚠️
...ore/tests/replica_placement_master_shadow_test.cpp	99.72%	1 Missing ⚠️
...cake-store/tests/replica_placement_shadow_test.cpp	99.47%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!