[Bug]: ETCD OpLog is enabled implicitly for etcd HA but lacks retention and complete metadata replication

[no-teet-on]

Bug Report

Description

Before the commit that introduced the MC_ENABLE_HA_METADATA_REPLICATION feature gate, selecting etcd as the HA backend automatically enabled OpLog following:

capabilities.has_oplog_following =
spec.type == HABackendType::ETCD;

As a result, every deployment using etcd for leader election also entered the etcd OpLog replication path, with no way to use etcd only for leadership coordination.

Current problems

1. Leader election is tightly coupled with metadata replication

   Using ha_backend_type=etcd implicitly enables OpLog following. Users cannot independently enable etcd leader election while disabling the experimental metadata replication path.

2. OpLog entries do not have a TTL

   Entries under the following prefix are written using regular etcd Put or BatchCreate operations without an attached lease:

   /oplog/<cluster_id>/<sequence_id>

   The following metadata keys also have no TTL:

   /oplog/<cluster_id>/latest
   /oplog/<cluster_id>/snapshot/<snapshot_id>/sequence_id

3. There is no automatic OpLog cleanup

   EtcdOpLogStore::CleanupOpLogBefore() can delete entries below a sequence boundary, but it is not called by any production path. It is currently only exercised by tests.

   Therefore, once production OpLog writes are enabled, the number of keys and the etcd database size can grow indefinitely.

4. The primary write path is not fully integrated

   The production request path does not appear to:

   • Create an etcd OpLog store in writer mode.
   • Attach it through OpLogManager::SetOpLogStore().
   • Append production metadata mutations to the OpLog.

   Consequently, a standby may start an OpLog watcher even though the active primary is not publishing the required records.

5. Promotion does not hand replicated metadata to the serving master

   HotStandbyService maintains an internal metadata store and provides ExportMetadataSnapshot(). However, the promotion path creates a new WrappedMasterService without transferring the replicated
   standby metadata or its last applied sequence ID.

   This means that successfully applying OpLog records in the standby does not necessarily make that state available to the newly promoted serving master.

Impact

• etcd HA deployments unexpectedly depend on an incomplete replication path.
• Enabling OpLog may give users a false expectation of lossless metadata failover.
• Stale or empty metadata may be served after failover.
• OpLog keys may accumulate indefinitely and eventually consume the etcd storage quota.
• A failure to initialize the OpLog reader can prevent a standby from being safely promoted.

Expected behavior

Leader election and metadata replication should be independently configurable.

Until the metadata replication path is complete, etcd OpLog following should be disabled by default and enabled only through an explicit feature gate such as:

MC_ENABLE_HA_METADATA_REPLICATION=1

The complete implementation should also:

• Wire all primary metadata mutations into an etcd OpLog writer.
• Transfer replicated standby state into the promoted master.
• Preserve and restore the last applied sequence ID.
• Introduce a safe retention policy based on snapshot boundaries and standby progress.
• Periodically invoke OpLog cleanup.
• Clean obsolete snapshot sequence markers.
• Expose metrics for OpLog key count, retained sequence range, and cleanup failures.
• Document the etcd storage and permission requirements.

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

[github-actions]

Thanks for opening this issue, @no-teet-on!

Field	Value
Issue	#2971
GitHub user ID	305664233
Reporter	@no-teet-on

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[Icedcoco]

Thanks for the detailed write-up. I checked this against the current main branch, and the concerns in points 1 and 4 are valid. Leadership and metadata replication are still coupled: choosing ha_backend_type=etcd also enables OpLog following, without an independent option to turn replication off. At the same time, the primary write path is not fully wired while the standby reader can already be started. We’ll get these two issues fixed as soon as possible.

For point 2, the lack of TTL is intentional. OpLog entries are durable recovery data, so expiring them through etcd leases could leave gaps in the replay history. They should instead be removed through an explicit retention policy once a durable snapshot covers the relevant sequence range.

Point 3 is also valid: automatic cleanup has not been implemented yet. The intended approach is to clean up old OpLog entries after a snapshot is safely persisted, while making sure lagging standbys can either continue replaying or bootstrap from a retained snapshot.

Point 5 is being addressed in #2826, which passes the replicated metadata, segment state, and last applied sequence ID into the promoted master. Please feel free to take a look and leave comments if you spot anything we missed.