Changes proposed
Motivation
Mooncake Store currently treats one client heartbeat timeout as a permanent failure. Temporary network jitter, a client pause, or a delayed Master monitor loop can therefore irreversibly unmount recoverable resources and delete their metadata.
We propose adding a recovery window: after the first TTL expires, the Master retains the client's resources but stops routing new requests and work to them; a valid liveness signal makes them serving again. Terminal cleanup starts only if the client remains unavailable for a second TTL.
Resource availability checks are currently distributed across allocation, replica lookup, Copy/Move, task dispatch, drain, offload, and promotion. This proposal also centralizes that decision so the new liveness semantics are applied consistently.
Changes proposed
- Introduce an Active → Suspected → Offline state machine for each client incarnation, with separate active and suspicion TTLs.
- Add a typed
ServingEligibility seam for all client-owned resource routing, while retaining Suspected resources for possible recovery.
- Separate client liveness from remount readiness and segment lifecycle.
- Express terminal cleanup as a retryable
ClientOffboardingJob; complete it synchronously first, then move the same job to an asynchronous executor.
- Preserve the existing Ping RPC schema and legacy TTL configuration.
- Rebuild process-local liveness records when restoring Master snapshots; keep NoF on its independent health model.
Proposed design
Client liveness
Each client incarnation has a ClientLivenessRecord:
Active --active TTL--> Suspected --suspicion TTL--> Offline
^ |
+---- liveness signal --+
- Active resources may serve new requests.
- Suspected resources are retained but cannot receive new allocations, descriptors, or work. A valid liveness signal recovers the same incarnation.
- Offline is terminal for that incarnation and starts offboarding.
The suspicion TTL starts when the Master actually commits Active -> Suspected, so a delayed monitor does not shorten the recovery window. Ping and successful segment mounts are liveness signals; ordinary Put/Get traffic and task/offload/promotion heartbeats are not.
Ping remains wire-compatible: a ready Active client receives OK; an unknown or Offline client receives NEED_REMOUNT; Ping may recover a Suspected client before applying the existing remount-readiness check.
Client liveness, remount readiness (ok_client_), and segment lifecycle remain independent. In particular, ok_client_ will no longer determine whether resources may serve or must be retained.
ServingEligibility
Allocation, metadata reads, task routing, offload, promotion, and drain will use a small typed interface with the following common semantics:
resource can serve
= resource is intrinsically valid
&& segment lifecycle permits serving
&& owning ClientLivenessRecord is Active
ShouldRetainResources
= ClientLivenessRecord is Active or Suspected
This supplements existing segment checks rather than replacing them. NoF resources continue to use their own probe and lifecycle rules.
While Suspected, the Master blocks new dependencies on the client, including allocations, descriptors, Copy/Move routing, task delivery, offload/promotion work, and drain selection. End, Revoke, failure, and completion callbacks for already-started work remain accepted so refcounts, quota, and task state can converge.
Memory allocator registrations and buffers retain the specific ClientLivenessRecord that created them, preventing a later incarnation from reactivating old memory resources.
Offboarding
Terminal retirement uses one job interface:
PrepareClientOffboarding(record) -> ClientOffboardingJob
CompleteClientOffboarding(job)
Prepare commits Offline and immediately isolates the incarnation. Complete performs the expensive work: metadata cleanup, memory segment unmount, LocalDisk/offload/promotion cleanup, quota convergence, and external metadata cleanup. The old record is removed only after all steps succeed; until then it acts as a Mount/ReMount barrier.
The first implementation calls Complete synchronously from the client monitor. A follow-up submits the same job to an asynchronous executor with batching, retry, shutdown draining, and queue observability. This keeps state and cleanup semantics unchanged when execution becomes asynchronous.
HA restore and compatibility
After snapshot restore, each restored client starts as Active with a fresh active TTL, preserving the current recovery semantics. The Master rebuilds the liveness associations needed by restored resources, while readiness still follows the existing ReMount protocol.
Snapshot creation must not capture partially completed offboarding. The synchronous phase preserves the current exclusion behavior; asynchronous offboarding will add explicit snapshot/offboarding coordination.
New configuration:
client_active_ttl_sec: 10
client_suspicion_ttl_sec: 20
client_live_ttl_sec and --client_ttl remain deprecated aliases for the active TTL. If an active/legacy TTL is explicitly configured but suspicion TTL is omitted, suspicion inherits the resolved active TTL. No RPC schema change is required.
LocalDisk and task metadata currently identify ownership by client ID. Initially, complete cleanup plus the Offline remount barrier prevents old resources from overlapping a new incarnation. An explicit incarnation token would provide stricter isolation for artificially reused client IDs, but requires a wire/schema change and is outside the initial scope.
Development plan
- Add
ClientLivenessRecord, dual-TTL configuration, Ping/Mount integration, and the three-state monitor.
- Introduce
ServingEligibility across allocation, descriptors, Copy/Move, task/drain, LocalDisk/offload, and promotion.
- Add synchronous
ClientOffboardingJob, retry semantics, HA rebinding, metrics, and regression coverage.
- Move job completion to the asynchronous executor without changing state semantics.
Implementation may proceed while this RFC is discussed, in reviewable commits that can be adjusted before merge.
Open questions
- Should Admin descriptor queries hide Suspected resources while diagnostic status queries still expose them?
- Is it acceptable for new offload/promotion work rejected during Suspected to require a later rescan rather than automatic replay initially?
- Should snapshots wait for pending asynchronous offboarding jobs, or should offboarding state eventually be serialized?
- Is preserving the current RPC schema preferable to adding an explicit incarnation token in the first version?
Feedback on the state semantics, ServingEligibility seam, and staged synchronous-to-asynchronous offboarding plan is especially welcome.
Before submitting a new issue...
Changes proposed
Motivation
Mooncake Store currently treats one client heartbeat timeout as a permanent failure. Temporary network jitter, a client pause, or a delayed Master monitor loop can therefore irreversibly unmount recoverable resources and delete their metadata.
We propose adding a recovery window: after the first TTL expires, the Master retains the client's resources but stops routing new requests and work to them; a valid liveness signal makes them serving again. Terminal cleanup starts only if the client remains unavailable for a second TTL.
Resource availability checks are currently distributed across allocation, replica lookup, Copy/Move, task dispatch, drain, offload, and promotion. This proposal also centralizes that decision so the new liveness semantics are applied consistently.
Changes proposed
ServingEligibilityseam for all client-owned resource routing, while retaining Suspected resources for possible recovery.ClientOffboardingJob; complete it synchronously first, then move the same job to an asynchronous executor.Proposed design
Client liveness
Each client incarnation has a
ClientLivenessRecord:The suspicion TTL starts when the Master actually commits
Active -> Suspected, so a delayed monitor does not shorten the recovery window. Ping and successful segment mounts are liveness signals; ordinary Put/Get traffic and task/offload/promotion heartbeats are not.Ping remains wire-compatible: a ready Active client receives
OK; an unknown or Offline client receivesNEED_REMOUNT; Ping may recover a Suspected client before applying the existing remount-readiness check.Client liveness, remount readiness (
ok_client_), and segment lifecycle remain independent. In particular,ok_client_will no longer determine whether resources may serve or must be retained.ServingEligibilityAllocation, metadata reads, task routing, offload, promotion, and drain will use a small typed interface with the following common semantics:
This supplements existing segment checks rather than replacing them. NoF resources continue to use their own probe and lifecycle rules.
While Suspected, the Master blocks new dependencies on the client, including allocations, descriptors, Copy/Move routing, task delivery, offload/promotion work, and drain selection. End, Revoke, failure, and completion callbacks for already-started work remain accepted so refcounts, quota, and task state can converge.
Memory allocator registrations and buffers retain the specific
ClientLivenessRecordthat created them, preventing a later incarnation from reactivating old memory resources.Offboarding
Terminal retirement uses one job interface:
Prepare commits Offline and immediately isolates the incarnation. Complete performs the expensive work: metadata cleanup, memory segment unmount, LocalDisk/offload/promotion cleanup, quota convergence, and external metadata cleanup. The old record is removed only after all steps succeed; until then it acts as a Mount/ReMount barrier.
The first implementation calls Complete synchronously from the client monitor. A follow-up submits the same job to an asynchronous executor with batching, retry, shutdown draining, and queue observability. This keeps state and cleanup semantics unchanged when execution becomes asynchronous.
HA restore and compatibility
After snapshot restore, each restored client starts as Active with a fresh active TTL, preserving the current recovery semantics. The Master rebuilds the liveness associations needed by restored resources, while readiness still follows the existing ReMount protocol.
Snapshot creation must not capture partially completed offboarding. The synchronous phase preserves the current exclusion behavior; asynchronous offboarding will add explicit snapshot/offboarding coordination.
New configuration:
client_live_ttl_secand--client_ttlremain deprecated aliases for the active TTL. If an active/legacy TTL is explicitly configured but suspicion TTL is omitted, suspicion inherits the resolved active TTL. No RPC schema change is required.LocalDisk and task metadata currently identify ownership by client ID. Initially, complete cleanup plus the Offline remount barrier prevents old resources from overlapping a new incarnation. An explicit incarnation token would provide stricter isolation for artificially reused client IDs, but requires a wire/schema change and is outside the initial scope.
Development plan
ClientLivenessRecord, dual-TTL configuration, Ping/Mount integration, and the three-state monitor.ServingEligibilityacross allocation, descriptors, Copy/Move, task/drain, LocalDisk/offload, and promotion.ClientOffboardingJob, retry semantics, HA rebinding, metrics, and regression coverage.Implementation may proceed while this RFC is discussed, in reviewable commits that can be adjusted before merge.
Open questions
Feedback on the state semantics,
ServingEligibilityseam, and staged synchronous-to-asynchronous offboarding plan is especially welcome.Before submitting a new issue...