[Bug]: Bucket LRU may evict newly offloaded buckets before older recently read buckets

[ygr9237]

Summary

When the bucket-based L4 backend uses the lru eviction policy, a newly offloaded bucket is inserted into the LRU index with last_access_ns_ = 0. The timestamp is updated only when the bucket is later read through BatchLoad().

As a result, a newly written but not-yet-read bucket can be selected for eviction before an older bucket that has been read recently. In write-heavy workloads, this can cause newly offloaded KV cache data to be evicted on the next capacity-triggered eviction.

The current behavior is effectively least recently read, rather than LRU over successful reads and writes/admissions.

Environment

• Mooncake version: 0.3.10.post2
• GPU：B300
• Model：GLM-5.1
• Deployment mode: standalone mooncake_master + mooncake_client
• L4 backend: bucket_storage_backend
• L4 storage: local filesystem using .bucket and .meta files
• Transport: RDMA
• Bucket eviction policy: lru
• Configured per-node bucket capacity: 52 TiB
• Bucket size limit: 256 MiB
• Bucket key limit: 500
• io_uring: disabled

Affected component

• BucketStorageBackend
• Bucket-based L4 offload (.bucket / .meta)
• MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY=lru
• MOONCAKE_BUCKET_EVICTION_POLICY=lru

Relevant source files:

• mooncake-store/src/storage_backend.cpp
• mooncake-store/include/storage_backend.h

Deployment configuration

Master

nohup numactl --interleave=0-3 mooncake_master \
    --rpc_port=53011 \
    --enable_http_metadata_server=true \
    --http_metadata_server_port=8011 \
    --metrics_port=9011 \
    --eviction_high_watermark_ratio=0.95 \
    --allocation_strategy=free_ratio_first \
    --enable_offload=true \
    --pending_task_timeout_sec=1800 \
    --processing_task_timeout_sec=1800 \
    --logtostderr \
    > /data/logs/mooncake_remote_l3_l4/mooncake_master_l3_l4_remote.log 2>&1 &

Client / L4 bucket backend

ulimit -l unlimited || true

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/python3.12/dist-packages/mooncake:${LD_LIBRARY_PATH:-}

MC_MIN_RPC_PORT=16011 \
MC_MAX_RPC_PORT=16011 \
MOONCAKE_DEBUG_L4_READ_MD5=0 \
MOONCAKE_OFFLOAD_FILE_STORAGE_PATH="/data/mooncake_l4" \
MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend \
MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES=$((256 * 1024 * 1024)) \
MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT=500 \
MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE=$((52 * 1024 * 1024 * 1024 * 1024)) \
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=$((100 * 1024 * 1024 * 1024)) \
MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY=lru \
MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_TTL_MS=5000 \
MOONCAKE_OFFLOAD_USE_URING=0 \
nohup numactl --interleave=0-3 mooncake_client \
    --host=<client-ip> \
    --port=52011 \
    --global_segment_size=250GB \
    --master_server_address=<master-ip>:53011 \
    --metadata_server=http://<master-ip>:8011/metadata \
    --protocol=rdma \
    --device_names=<rdma-device-list> \
    --enable_offload=true \
    --logtostderr \
    > /data/logs/mooncake_remote_l3_l4/mooncake_client_l3_l4_remote.log 2>&1 &

MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE is the bucket-backend quota relevant to this report.

Current behavior

A newly committed bucket is added to the bucket map and LRU index with an initial timestamp of zero:

buckets_.emplace(bucket_id, std::move(bucket));
lru_index_.emplace(0LL, bucket_id);

last_access_ns_ is updated only when BatchLoad() reads the bucket:

const auto now = std::chrono::steady_clock::now()
                     .time_since_epoch()
                     .count();

bucket_it->second->last_access_ns_.store(
    now, std::memory_order_relaxed);

The eviction index is ordered by {last_access_ns_, bucket_id}, and the smallest entry is selected first.

This produces the following deterministic ordering:

1. Bucket A is written.
2. Bucket A is read through BatchLoad(), so A.last_access_ns_ > 0.
3. Bucket B is written but has not yet been read, so B.last_access_ns_ = 0.
4. A later offload triggers L4 eviction.
5. Bucket B is selected before bucket A, even though B was written more recently.

If multiple buckets have never been read, they all have last_access_ns_ = 0. Selection among them is then determined by bucket_id, so smaller/earlier bucket IDs are evicted first.

There is only a short implicit protection during the current write: PrepareEviction() runs before the new bucket has been inserted into buckets_, so that write cannot evict itself. Once the commit completes, the new bucket is immediately eligible for the next eviction.

inflight_reads_ does not provide admission protection. It only prevents deletion from racing with reads that were already in progress after a bucket was selected for eviction.

Production observation

The issue was noticed after the configured L4 bucket capacity was reached.

In the observed four-node cluster:

• each node reached the configured 52 TiB L4 bucket capacity limit (approximately 52.00 TiB shown on the dashboard);
• aggregate L4 usage reached approximately 207.98 TiB;
• after reaching the limit, observed L4 bucket-read activity became much lower;
• on July 17, the page-cache/read snapshot for one node was concentrated on buckets written on July 7;
• newer buckets were rarely visible in the observed read/page-cache samples.

Capacity screenshot

The dashboard showed that each node had reached the configured 52 TiB L4 bucket capacity limit, with approximately 52.00 TiB displayed per node and 207.98 TiB in aggregate.

Bucket write-time distribution screenshot

The snapshot collected on July 17 showed page-cache residency/read activity concentrated on buckets whose displayed write time was July 7.

This observation is consistent with the current LRU ordering:

• older buckets that have been read receive a positive last_access_ns_;
• newly written buckets remain at 0 until their first successful BatchLoad();
• once capacity is full, newly written but unread buckets can therefore be removed before older read-hot buckets.

However, the dashboard observation alone is not intended as complete proof that every newly written bucket is immediately evicted. The source-code ordering itself is deterministic, while the production evidence motivated this report and shows the operational impact that may result from it.

Why this is problematic

A successful write normally constitutes admission into an LRU-managed cache and should place the new entry at, or near, the MRU end.

With the current implementation:

• newly offloaded KV cache data receives no recency protection;
• a bucket can become an eviction candidate immediately after its write completes;
• read-hot old buckets are preferred over newly admitted buckets regardless of write recency;
• write-heavy or streaming workloads may experience avoidable cache churn;
• increasing MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE only reduces eviction frequency and does not guarantee protection for newly written buckets.

Additional production evidence from a four-client deployment with 5 TiB L4 capacity per client

To further verify the runtime impact, I analyzed a deployment consisting of four independent mooncake_client instances. Each client used the same bucket-based L4 backend with the lru eviction policy and had an independently configured L4 bucket capacity limit of 5 TiB, for an aggregate configured capacity of 20 TiB.

Test window and capacity timeline

• Workload period: 2026-07-17 01:54:00 to 2026-07-17 06:01:00

• L4 directory:

  /glm5-2_mooncake_l4/l4_data/ssd_data/mooncake_remote_l3_l4_clean
  

• The storage reached its configured limit at approximately:

  2026-07-17 04:38:25
  

• The first reconstructed capacity-triggered write/eviction event appeared at:

  2026-07-17 04:38:27.526500
  

Therefore, continuous bucket eviction started after the storage reached the configured limit.

For the specific mooncake_client instance analyzed below, the eviction log reported:

max_total_size = 5497558138880 bytes = 5 TiB

The dashboard panel displayed a node-level file-storage plateau of 20 TiB. The backend log value above is used as the source of truth for the quota of the specific bucket backend instance analyzed in this reconstruction.

During this period, the client repeatedly logged:

[Evict] triggered
[Evict] prepared: buckets=1
[Evict] finalized: deleted 1 bucket(s)

Historical bucket reconstruction

The eviction log does not print the selected bucket ID. To reconstruct the write history, I used a known bucket ID, timestamp, and exact logged required size as an anchor:

bucket_id: 7308125764896
write_time: 2026-07-17 05:56:03.903993
required: 268187449 bytes

Using this anchor, I reconstructed the following range:

first bucket_id: 7308125761626
last bucket_id:  7308125764896
total writes:    3271

The reconstructed results were:

Result	Count
Buckets still present	1,481
Buckets completely absent	1,790
Partial .bucket-only or .meta-only states	0
Surviving .bucket files whose exact size matched the logged required value	1,481 / 1,481
Exact size-match rate	100.00%

The 1481/1481 exact size match validates the mapping between the logged write events, reconstructed bucket IDs, and surviving files.

The fact that all missing entries lost both their .bucket and .meta files, with no partial states, is also consistent with complete bucket removal rather than an incomplete write or isolated file corruption.

Recently written bucket removed while much older buckets remained

One of the newest reconstructed buckets that is no longer present was:

bucket_id:        7308125764830
write_time:       2026-07-17 05:53:54.389101
bucket_data_size: 266549599 bytes
current state:    both .bucket and .meta are absent

However:

older buckets still present: 1415
newer buckets still present: 66

For example, the following much older bucket was still present:

bucket_id:  7308125761626
write_time: 2026-07-17 04:38:27.526500
state:      present

It was written approximately 1 hour, 15 minutes, and 26.86 seconds before the missing bucket.

The immediately following bucket was also still present:

bucket_id:  7308125764831
write_time: 2026-07-17 05:53:54.649519
state:      present

It was written only approximately 260.42 milliseconds after the missing bucket.

The observed retention order was therefore:

many much older buckets remained present
a recently written bucket was absent
later-written buckets remained present

This ordering is not FIFO and does not protect newly admitted buckets based on write recency.

It is consistent with the current implementation:

buckets_.emplace(bucket_id, std::move(bucket));
lru_index_.emplace(0LL, bucket_id);

A newly admitted but unread bucket enters the LRU index with a recency value of 0, while an older bucket that has previously been read has a positive last_access_ns_ and is therefore ranked as newer.

Evidence boundary

The historical client log does not include the bucket ID selected by each individual eviction. Therefore, this reconstruction cannot associate bucket 7308125764830 with one specific [Evict] finalized line.

However, the following evidence is available:

1. the backend had reached its configured 5 TiB limit;
2. the backend was continuously performing capacity-triggered eviction;
3. the reconstructed write-to-bucket mapping was validated against all surviving files with a 100.00% exact size-match rate;
4. 1,790 reconstructed buckets were absent as complete .bucket / .meta pairs;
5. a bucket written at 05:53:54.389101 was absent;
6. 1,415 buckets written earlier were still present;
7. 66 buckets written later were still present;
8. the source code admits newly written buckets into the LRU index with last_access_ns_ = 0.

Taken together, this provides strong production evidence of recency inversion and premature eviction of newly admitted buckets under capacity pressure.

Expected behavior

When the eviction policy is lru, a successfully committed bucket should be treated as recently used/admitted.

A newly written bucket should not rank as older than every bucket that has ever been read.

Proposed fix

Initialize the new bucket's recency timestamp during a successful BatchOffload commit and insert the same timestamp into lru_index_:

const auto now = std::chrono::steady_clock::now()
                     .time_since_epoch()
                     .count();

bucket->last_access_ns_.store(
    now, std::memory_order_relaxed);

buckets_.emplace(bucket_id, std::move(bucket));
lru_index_.emplace(now, bucket_id);

This changes the policy semantics to least recently read or written/admitted, which is closer to conventional LRU cache behavior.

Before submitting...

• Ensure you searched for relevant issues and read the documentation.

[github-actions]

Thanks for opening this issue, @ygr9237!

Field	Value
Issue	#2976
GitHub user ID	152075046
Reporter	@ygr9237

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.