fix(queue): recover fast-policy partial records

[smiggleworth]

Summary

Prevent fast queue recovery from turning a policy-permitted partial split record into a permanent startup outage. Fast-policy startup now durably removes incomplete header/body remnants and invalidates affected derived indexes; buffered and strict policies remain fail-closed.

Fixes #78

What Changed

• Added fast-policy startup reconciliation for missing bodies and orphan bodies in one durable transaction.
• Kept complete split records and embedded legacy records byte-for-byte unchanged.
• Required sync or cloud-strict write options for recovery cleanup and wired boot to the broker recovery policy.
• Made header/body pairing realm-aware and covered cross-realm collisions.
• Audited KV, Stream, and Schedule persistence paths; documented why Queue fast mode is the only policy that can admit this partial state.
• Updated recovery, domain-boundary, and cloud-operations documentation.

Validation

• Red: cargo test --lib should_reconcile_ -- --nocapture failed on missing-body and orphan-body startup cases before the implementation.
• Green: cargo test --lib domains::queue passed 113 tests.
• Green: cargo fmt --all -- --check.
• Green: cargo test --workspace.
• Green: cargo clippy --locked --workspace --all-targets --all-features -- -D warnings -D clippy::pedantic.

Domain Boundary Review

• The change remains inside Queue startup persistence handling; no async was introduced into core code.
• The recovery guarantee is limited to fast-policy split-row remnants already permitted by the configured loss window. It does not imply recovery of lost messages.
• KV, Stream, and Schedule retain their existing transactional and fail-closed behavior.
• Relevant development and operations docs were updated in this change.

Notes

• No UI or OpenAPI surface changed, so adapter and UI validation were not applicable.

[Copilot]

Pull request overview

This PR hardens Queue startup recovery under the fast write policy by durably reconciling policy-permitted partial split-record remnants (header-only or body-only) so they don’t cause permanent startup failure loops, while keeping buffered/strict policies fail-closed and leaving complete/legacy records untouched.

Changes:

• Introduces fast-policy startup preparation that scans persisted queue state, deletes incomplete header/body remnants, and invalidates derived queue indexes for rebuild.
• Wires boot-time Queue initialization to use broker-level durable write options for reconciliation, and adjusts sink initialization to call the new “prepare” routine.
• Adds regression tests for missing-body/orphan-body reconciliation, durable-write requirement, realm-aware pairing, and persistence across restart; updates ops/dev documentation.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/domains/queue/sink/domain_sink_impl.rs	Calls queue “prepare persisted state” on sink initialization and threads recovery write options into construction.
src/domains/queue/actor/constructors_validation.rs	Implements persisted-state scanning, fast-policy reconciliation (durable delete + index invalidation), and realm-aware header/body pairing.
src/domains/queue/actor/storage_keys/keys.rs	Removes now-unused body-suffix helper (replaced by full storage-key construction).
src/boot/domains.rs	Passes broker durable write options into Queue sink creation; small refactor for RPC timeout variable.
src/domains/queue/actor/tests/startup_reconciliation.rs	Adds reconciliation-focused regression tests including realm collision and restart persistence.
src/domains/queue/actor/tests.rs	Registers the new startup reconciliation test module.
docs/operations/cloud-setup.md	Documents fast-policy startup reconciliation and durable cleanup behavior.
docs/development/recovery-internals.md	Updates recovery steps and adds a persisted partial-state policy table including Queue fast-mode reconciliation.
docs/development/domain-boundaries-spec.md	Documents fast-policy startup discard/rebuild behavior and buffered/strict fail-closed contract.

Comments suppressed due to low confidence (1)

src/domains/queue/actor/constructors_validation.rs:448

• The durable-policy failure here also says queue validation failed, but it is emitted from the reconciliation step. Aligning the prefix with queue reconciliation failed makes it clearer that the operator action is about the cleanup write policy, not corrupted state validation.

        if !recovery_write_options.is_sync() && !recovery_write_options.is_cloud_strict() {
            return Err(format!(
                "queue validation failed: family={family} key_category=reconciliation error=durable recovery write policy required"
            ));

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.