[RFC]: [Store] Entity Ownership Decoupling, SegmentView, and Recoverable Lifecycle

[Icedcoco]

Changes proposed

Summary

This RFC proposes a first-version redesign of Mooncake Store Master metadata ownership and resource lifecycle. The proposal separates logical object metadata, replica state, allocation handles, and client/segment liveness while keeping the implementation deliberately small:

• Object owns its Replica instances.
• Replica owns one AllocatedBuffer.
• AllocatedBuffer points to its containing Segment.
• Segment owns the allocator and publishes its lifecycle through an independent SegmentView.
• Raw pointers express ownership or observation; no global Registry and no shared_ptr ownership graph are introduced.
• Object has no lifecycle state. When its replica collection becomes empty, the object is removed from the object map and then destroyed.
• Replica changes are represented by a small set of primitive events and replayed in order. Composite operations such as Move are not new oplog operations.

The design is intended to complement RFC #2953 rather than replace its client liveness semantics. RFC #2953 supplies the client-level recovery window; this RFC defines how that signal affects Segments, Replicas, allocation handles, metadata visibility, and HA promotion.

Motivation

The current Master implementation keeps resource ownership and lifecycle information spread across Object, Replica, Segment, allocator, task, and HA paths. This creates several practical risks:

1. A Replica can retain information that is owned by another entity, making deletion and recovery ordering difficult to reason about.
2. Asynchronous callbacks may outlive the structure they reference if they capture entity pointers directly.
3. Segment state, connection information, and allocator state can be restored from different persistence paths and therefore disagree.
4. Composite operations write business-specific oplog entries in several places, making ordering and replay behavior difficult to prove.
5. Promotion currently needs an allocator even when the complete Segment description is not available yet.

The first version should solve these problems without introducing a general object registry, a new Replica state machine framework, or a new generation protocol for Objects.

Goals

1. Make ownership and pointer lifetime explicit for Object, Replica, AllocatedBuffer, Segment, and allocator.
2. Keep Replica identifiers stable enough for replay and asynchronous lookup, while allowing callers to use ObjectKey and ReplicaId without retaining a Replica*.
3. Define a fixed Segment lifecycle and its event transitions.
4. Centralize primitive Replica mutations and their oplog mapping.
5. Make removal safe: an AllocatedBuffer must be released before its Replica is removed from Object.
6. Make SegmentView the only persistent authority for Segment information.
7. Define a deterministic standby/promotion order that works before a real allocator is available.
8. Preserve the existing behavior where possible and keep the first version reviewable.

Non-goals

• No global Registry for Replica ownership or lookup.
• No shared_ptr ownership graph for core entities.
• No formal Replica state-machine framework in the first version.
• No Object generation field.
• No SegmentId reuse. New mounts receive a new client-generated UUID.
• No periodic forced deletion of PROCESSING Replicas. They complete naturally; cleanup is triggered by mutation callbacks or explicit removal paths.
• No change to the RPC schema in this RFC.
• No claim that a non-transactional backend can provide the same HA ordering guarantees.

Entity model

Object
  owns vector<Replica*>

Replica
  owns AllocatedBuffer*

AllocatedBuffer
  observes Segment*

Segment
  owns allocator

Object

Object retains ObjectKey, logical size, data type, tenant/group metadata, lease/pin/quota/accounting information, a per-object monotonic ReplicaId counter, and vector<Replica*>.

Object does not retain client id or Segment id. It has no independent lifecycle state. A Replica is removed from the vector only after its buffer has been released. Once the vector is empty, the Object is first erased from the object map and then destroyed.

group_id is immutable while the Object exists, matching the current code's validation. size may change during destructive Upsert; cleanup and quota accounting must use the sizes recorded in each AllocatedBuffer rather than the current Object size.

Replica

Each Replica retains its ObjectKey and a 32-bit ReplicaId. IDs are allocated monotonically per Object and are not reused while the Object exists. Copy and Move create a new target Replica with a new ID; the source ID does not change.

The first version keeps the current status values:

UNDEFINED, INITIALIZED, PROCESSING, COMPLETE, REMOVED, FAILED

It does not add a generic Replica state-machine abstraction. Replica methods should own common getters, descriptor access, read/write eligibility checks, reference-count changes, and the small set of legal status changes. They do not write HA storage directly.

AllocatedBuffer

AllocatedBuffer is a logical allocation handle, not ownership of the whole client memory region. It stores its containing Segment pointer, requested and reserved sizes, and the medium-specific allocation descriptor (offset, address, file locator, or equivalent).

Its destruction synchronously returns the allocation to the local allocator and decrements Segment.active_allocations. A separate released_ flag is not required. Local release is expected to be infallible; an error is an invariant failure rather than an ordinary business result.

Segment and SegmentView

Segment owns the allocator and runtime connection object. Its frequently read lifecycle state is an atomic value. The fixed states are:

INIT
ACTIVE
DRAINING
GRACEFUL_UNMOUNTING
SUSPECTED
UNMOUNTING
UNMOUNTED

DRAINED is intentionally not a state; a drained Segment proceeds to UNMOUNTING.

All durable Segment information is stored in an independent SegmentView store. SegmentView is not part of Object/Replica oplog or snapshot state and is the sole authority for Segment identity, medium, endpoint/connection information, lifecycle state, deadlines, reason, and connection generation.

The primary writes SegmentView first and publishes the new local atomic state only after the backend write succeeds. A failed write leaves the old local state in place and is retried. Ordinary GET/PUT/Copy/Move continue using the previous state; they do not receive an artificial retry result merely because the state publication backend is temporarily unavailable.

Segment lifecycle

The normal transitions are:

INIT --registration/heartbeat--> ACTIVE
ACTIVE --active drain request--> DRAINING
DRAINING --drain complete--> UNMOUNTING
ACTIVE --graceful unmount--> GRACEFUL_UNMOUNTING
GRACEFUL_UNMOUNTING --deadline--> UNMOUNTING
GRACEFUL_UNMOUNTING --temporary heartbeat loss--> SUSPECTED
INIT --recovered graceful registration--> GRACEFUL_UNMOUNTING
INIT --registration timeout--> SUSPECTED
SUSPECTED --heartbeat recovery--> ACTIVE or GRACEFUL_UNMOUNTING
SUSPECTED --long disconnect--> UNMOUNTING
UNMOUNTING --active_allocations == 0 and cleanup complete--> UNMOUNTED

GRACEFUL_UNMOUNTING never silently returns to ACTIVE. Reconnection keeps the Segment in graceful unmounting; cancelling that process requires an explicit future management event.

SUSPECTED rejects new reads, writes, allocation, Copy, Move, and task dispatch. Already-running tasks may finish and report their result so that Replica status, references, and allocation accounting converge.

UNMOUNTING restores existing Object/Replica/task metadata but rejects new work. PROCESSING Replicas are allowed to return normally. The first version does not periodically cancel them or force cleanup after a timeout.

UNMOUNTED means the allocator has been destroyed, every AllocatedBuffer has been released, active_allocations == 0, and the final SegmentView state has been persisted. A future mount uses a new Segment UUID and is unrelated to the old Segment.

Primitive Replica events and oplog

The system uses a finite set of primitive events:

ALLOCATE
WRITE_SUCCEEDED
WRITE_RETRY
WRITE_FAILED
WRITE_FAILED_REMOVE
REMOVE
REMOVE_DURABLE
FINALIZE
INC_REF
DEC_REF

The durable oplog mapping is:

ALLOCATE success       -> REPLICA_CREATE
WRITE_SUCCEEDED        -> REPLICA_COMPLETE
WRITE_FAILED          -> REPLICA_FAILED
WRITE_FAILED_REMOVE   -> REPLICA_REMOVE
REMOVE                -> REPLICA_REMOVE

REMOVE_DURABLE, FINALIZE, and reference-count changes are local cleanup steps and do not create another logical oplog entry.

Object/Replica methods validate and mutate in-memory state. A shared Master mutation helper owns HA/non-HA persistence, ordered submission, retry policy, durable callbacks, and finalization. This avoids scattering oplog writes through Put, Copy, Move, Evict, and cleanup implementations.

Composite operations are expressed using primitive events:

• Put: allocate, write, complete.
• Copy: allocate and complete a new Replica, then adjust references.
• Move: complete the target Replica, then remove the source Replica.
• Evict: remove the selected Replica.

Move must submit REPLICA_COMPLETE(target) before REPLICA_REMOVE(source) to the same ordered writer. It need not wait for the first durable callback. The ordered writer guarantees that a durable remove implies that the preceding complete was accepted.

Upsert is intentionally destructive in the first version. Regardless of size, old Replicas are marked removed and their remove events are submitted first; only then is the Object's logical size updated and a new processing Replica created. This permits a short interval with no readable Replica and accepts the corresponding data-loss risk on failure. The old Object structure is not deleted before old Replica cleanup completes.

HA recovery and promotion

SegmentView is a strict recovery prerequisite. Standby/promotion must not recover Object/Replica resources from a missing or incomplete SegmentView. When SegmentView cannot be read, the standby retries with backoff. If the old Master lease is lost during this period, the partial recovery is discarded and the process returns to standby instead of entering serving.

The promotion sequence is:

1. Replay Object/Replica oplog into memory.
2. Use DummyAllocator while real Segment allocators are unavailable. It stores logical allocation records in original allocation order.
3. Acquire leader fencing.
4. Read a complete consistent SegmentView set.
5. Create real Segments and allocators.
6. Pause subsequent oplog catch-up.
7. Traverse all DummyAllocator records and perform real allocation in order.
8. Resume oplog catch-up only after the existing allocation rebuild completes.
9. Catch up to the current oplog head and validate pointer chains and active_allocations.
10. Enter serving only after both allocation rebuild and oplog catch-up have completed.

Allocation conflicts, capacity mismatches, or descriptor mismatches are treated as program/link errors in this first version. The affected Replica and the conflicting Replica are marked for removal and cleaned. The design relies on recorded SegmentId, ReplicaId, requested/reserved size, descriptor, and ordered replay to make such conflicts unreachable in normal operation.

The long-term direction is hot-standby promotion. Snapshot allocator internals are compatibility data only and should eventually be removed from snapshots.

Concurrency and pointer safety

Version one keeps metadata shard locks for Object/Replica lifecycle, status, reference counts, vector mutation, and object accounting. SegmentManager's lock protects Segment and allocator lifetime. The lock order is:

metadata shard -> SegmentManager -> allocator

Async tasks do not retain entity pointers. They carry ObjectKey, ReplicaId, SegmentId, and task-specific descriptors, then re-lookup the entity under the appropriate lock. Replica* is therefore not an asynchronous ownership mechanism.

Relationship to RFC #2953

RFC #2953 proposes a useful client-level Active -> Suspected -> Offline recovery window and a common serving-eligibility decision. This RFC keeps that concept, but places the resource lifecycle boundary at Segment and Replica:

Concern	RFC #2953 emphasis	This RFC
Recovery identity	Client incarnation	Segment UUID plus client liveness
Temporary failure	Client Suspected	Segment Suspected, propagated to admission checks
Terminal cleanup	ClientOffboardingJob	Segment Unmounting plus Replica/buffer convergence
Routing decision	ServingEligibility seam	Entity predicates backed by Segment state and Replica status
Persistence	Snapshot/oplog compatibility	Independent SegmentView; primitive Replica oplog
Async safety	Job object	ID-based task relookup and owner-first destruction

The two designs can coexist: a client liveness record can drive Segment state transitions, while SegmentView and Replica events define the exact resource cleanup and replay behavior.

Rollout

1. Introduce entity ownership and ID-based asynchronous lookup without changing external APIs.
2. Centralize Replica mutation and oplog mapping.
3. Add Segment atomic lifecycle and SegmentView persistence.
4. Add DummyAllocator-based promotion ordering and remove dependence on snapshot allocator internals.
5. Migrate old duplicate Segment persistence fields and retain compatibility readers only for the transition period.

Open questions

These are implementation details, not blockers for the architecture:

1. Should SegmentView use an independent HA backend interface or a dedicated keyspace in the existing backend?
2. Is a cross-key version barrier required between SegmentView and Replica oplog commits, or is ordered replay sufficient for the first version?
3. Should standby watch SegmentView continuously or read it only during promotion/recovery?
4. Which compatibility release can remove allocator internals from snapshots?

Any feedback is appreciated!

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[github-actions]

Thanks for opening this issue, @Icedcoco!

Field	Value
Issue	#2979
GitHub user ID	102317026
Reporter	@Icedcoco

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.