[Bug]: ssd total capacity shows 0 after master restart with snapshot and offload enabled

[liangxu2000]

Bug Report

Environment

• Mooncake version: master branch
• Master config: enable_snapshot=true, enable_offloading=true
• Client: 1 client, writing 100 KV pairs with SSD offload

Description

After master restarts with snapshot restore, the SSD total capacity (denominator)
in metrics shows 0, while the used space (numerator) is correct:

Before restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 2TB
After restart: Mem Storage: 400MB / 128GB | SSD Storage: 400MB / 0 B
^^^

The issue persists until the client also restarts (re-executes FileStorage::Init).

Root Cause (from source code analysis)

Two things happen:

1. ssd_total_capacity_bytes is not serialized in snapshot

In mooncake-store/src/segment.cpp, the SegmentSerializer serializes LocalDiskSegment with only enable_offloading and the offloading objects list (line ~589-591). The field ssd_total_capacity_bytes is not included.

After deserialize (line ~927-928), a new LocalDiskSegment is created with ssd_total_capacity_bytes = 0 (default value, segment.h:88).

2. file_total_capacity_ gauge is not rebuilt after restore

In TryRestoreStateFromSnapshot() (master_service.cpp), memory capacity is rebuilt after restore (lines 5001-5034), but there is no equivalent logic for file capacity. The file_total_capacity_ gauge remains at its initial value 0.

Note: ReportSsdCapacity RPC is the only runtime path that sets file_total_capacity_, but the client only calls it once at FileStorage::Init() and does not re-report after master restart.

Steps to Reproduce

1. Start master with enable_snapshot=true and enable_offloading=true
2. Start 1 client with SSD offload configured (e.g. 2TB capacity)
3. Write 100 KV pairs to trigger offloading to SSD
4. Observe metrics: SSD Storage: X / 2TB (correct)
5. Restart the master process (client continues running)
6. Master restores from snapshot
7. Observe metrics: SSD Storage: X / 0 B (denominator is 0)

Suggested Fix

1. Add ssd_total_capacity_bytes to LocalDiskSegment serialization/deserialization in segment.cpp
2. After snapshot restore in master_service.cpp, rebuild file_total_capacity_ gauge from restored segments, similar to the existing mem capacity rebuild logic (lines 5001-5034)

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

[github-actions]

Thanks for opening this issue, @liangxu2000!

Field	Value
Issue	#2783
GitHub user ID	258133064
Reporter	@liangxu2000

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[he-yufeng]

Thanks for the precise root-cause analysis — the missing field in the LocalDiskSegment serializer was exactly it. Opened #2927, which appends ssd_total_capacity_bytes to the packed array and reads it back backward-compatibly (old snapshots keep the current behavior until the client re-reports).