Skip to content

MeshSync: periodic reconciliation to self-heal missed watch events (design) #581

Description

@leecalcote

Summary

Add a periodic reconcile that re-lists watched resources and publishes only genuine drift, to self-heal missed watch events. Today the informer resync period is 0 (pure watch); a watch gap silently drifts MeshSync's snapshot from cluster truth until an external resync is triggered.

Key decisions

  • A dedicated LIST-vs-Store diff loop, not a non-zero informer resync period - a client-go resync only replays the local cache and cannot detect events missed entirely (including deletes). The loop diffs a fresh LIST against the informer's cache.Store: LIST minus Store = missed add, higher resourceVersion = missed update, Store minus LIST = missed delete (reconstruct the tombstone through the existing DeleteFunc path).
  • Publishes only drift (reimplements resourceVersion suppression at the reconcile layer) so it never spams Server; steady state = zero publishes.
  • User-tunable interval via the MeshSync CRD, default off, with an enforced floor; jittered and staggered across GVRs to avoid re-list storms.

Risks flagged

A partial/failed LIST must abort that GVR's pass for the tick (never diff a truncated list as a mass-delete). Fixes a pre-existing h.stores unguarded-map race that the loop's concurrent read is the first to expose.

Complementary to JetStream durability (companion blueprint).

Full design

docs/design/fd5-periodic-reconciliation.md (added in #576).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions