Skip to content

[Store] Serialize ssd_total_capacity_bytes in local disk snapshot (#2783)#2927

Merged
stmatengss merged 1 commit into
kvcache-ai:mainfrom
he-yufeng:fix/2783-serialize-ssd-capacity
Jul 18, 2026
Merged

[Store] Serialize ssd_total_capacity_bytes in local disk snapshot (#2783)#2927
stmatengss merged 1 commit into
kvcache-ai:mainfrom
he-yufeng:fix/2783-serialize-ssd-capacity

Conversation

@he-yufeng

@he-yufeng he-yufeng commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Description

Fixes #2783. After a master restart with snapshot restore, the SSD total capacity in metrics drops to 0 B (the used-bytes numerator stays correct), and it stays wrong until the client also restarts and re-runs FileStorage::Init:

Before restart: SSD Storage: 400MB / 2TB
After restart:  SSD Storage: 400MB / 0 B

SegmentSerializer packs a LocalDiskSegment as [enable_offloading, count, key1, task1, ...] and never includes ssd_total_capacity_bytes, so a segment rebuilt from a snapshot keeps its default 0 until the next client heartbeat that carries capacity (which today only happens on client re-init).

This appends ssd_total_capacity_bytes to the packed array and reads it back when present. Snapshots written before this change lack the trailing field, so Deserialize reads it only when array.size extends past the offloading objects; older snapshots deserialize exactly as before (capacity stays 0 until the client re-reports it). The format bump is therefore backward compatible — this mirrors the existing legacy-format handling already in Deserialize.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

The change is confined to the msgpack serialize/deserialize path, so I verified it with a standalone program that replicates the exact LocalDiskSegment pack/unpack logic:

  • New format round-trips capacity: a segment serialized with the trailing field deserializes back with ssd_total_capacity_bytes intact (2 TiB in the test).
  • Old format is backward compatible: a segment serialized without the field (pre-fix snapshot) deserializes with enable_offloading and the offloading objects intact and capacity 0 — reproducing the pre-fix behavior without crashing on the shorter array.
  • Empty-objects edge case: with no offloading objects the capacity sits at index 2 and still round-trips.

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (standalone serialize/deserialize round-trip, described above)

I didn't wire a repo test through the full snapshot child-process harness (it needs a Linux/etcd environment I can't run locally), but I'm happy to add a SegmentSerializer round-trip test asserting capacity survives if you'd like one in-tree.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.shclang-format isn't available in my environment; I matched .clang-format (Google, 4-space, 80 cols) by hand
  • I have run pre-commit run --all-files and all hooks pass — toolchain unavailable locally; please let CI run it
  • I have updated the documentation (if applicable) — no doc change needed
  • I have added tests to prove my changes are effective — see the standalone verification note above
  • For changes >500 LOC: I have filed an RFC issue — N/A (12 lines)

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Claude Code assisted while writing this fix. I've reviewed every changed line, traced the serialize/deserialize indices myself, and verified the round-trip and backward-compatibility with the standalone described above.

After a master restart with snapshot restore, the SSD total capacity in
metrics dropped to 0 (used bytes stayed correct) until the client restarted
and re-ran FileStorage::Init (kvcache-ai#2783). SegmentSerializer packed a
LocalDiskSegment as [enable_offloading, count, key/task pairs...] and never
included ssd_total_capacity_bytes, so the restored segment kept its default 0.

Append ssd_total_capacity_bytes to the packed array and read it back when
present. Snapshots written before this change lack the trailing field and
deserialize unchanged (capacity stays 0 until the client re-reports it), so
the format bump is backward compatible.

Signed-off-by: Yufeng He <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@stmatengss
stmatengss merged commit 5d87a43 into kvcache-ai:main Jul 18, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: ssd total capacity shows 0 after master restart with snapshot and offload enabled

2 participants