bug: durable MemTable writes can be acknowledged by an older generation

[u70b3]

This was generated by AI during triage.

Description

Durable MemTable writes can be acknowledged by a WAL flush from an older MemTable generation.

Each BatchStore generation numbers batches from zero, but the previous BatchDurableWatcher compared those generation-local positions against a writer-global durability watermark. After rotation, a successful flush from generation N could therefore satisfy a durable write at the same local position in generation N+1 before the newer write's own WAL append completed.

This is a durability correctness issue: the caller can receive success even though the request has not been made durable. A crash or a concurrent writer claiming a higher epoch can then expose the false acknowledgement.

Failure sequence

sequenceDiagram
    autonumber
    participant C as Client
    participant G0 as Writer A / generation N
    participant M as Shared durability watermark
    participant G1 as Writer A / generation N+1
    participant B as Writer B
    participant W as WAL

    C->>G0: durable put, local range 0..1
    G0->>W: flush generation N
    W-->>G0: success
    G0->>M: publish watermark 1
    Note over G0,G1: MemTable rotates and local positions restart at 0
    B->>W: claim a higher writer epoch
    C->>G1: durable put, local range 0..1
    G1->>M: wait for watermark >= 1
    M-->>G1: already satisfied by generation N
    G1-->>C: incorrect success
    G1->>W: flush generation N+1
    W-->>G1: PeerClaimedEpoch

Loading

Correct completion ownership

sequenceDiagram
    autonumber
    participant C as Client
    participant G as Accepting MemTable generation
    participant R as Request completion cell
    participant H as WalFlushHandler
    participant W as WAL

    C->>G: durable put
    G->>G: capture exact BatchStore and end range
    G->>R: create request-scoped completion
    G->>H: trigger(store, range, completion)
    H->>W: append exact captured range
    W-->>H: success or typed failure
    H->>R: write exact flush result
    R-->>C: complete this request's watcher
    Note over R,H: A different generation uses a different completion cell

Loading

Steps to reproduce

1. Open writer A with durable MemTable writes enabled.
2. Write and fully flush generation N so its local range 0..1 publishes durability.
3. Rotate writer A to generation N+1.
4. Open writer B on the same shard with a higher epoch so writer A's next WAL append is fenced.
5. Issue writer A's first durable put in generation N+1, also with local range 0..1.
6. Before the fix, the call returns success from generation N's watermark instead of returning PeerClaimedEpoch from its own flush.

The public regression is:

cargo test -p lance --test integration_tests mem_wal::durable_put_does_not_alias_across_memtable_generations -- --exact

Expected behavior

durable_write=true must return success only after the WAL handler reports success for a flush covering the exact BatchStore range accepted for that request. Flushes from other generations must not satisfy the request. Typed fencing and persistence failures must be returned to the caller.

Acceptance criteria

• Bind each durable MemTable write to a request-scoped flush completion for its exact accepting BatchStore and end range.
• Preserve public API signatures and generation-local WriteResult.batch_positions semantics.
• Preserve WAL-only behavior and persistent formats.
• Cover public two-writer fencing, injected persistence failure, handler closure, and coalesced already-covered empty-flush behavior.

Tracking

The fix is implemented in PR #7759.

Maintainer note

The automated issue labeler added performance, but this is a durability correctness bug rather than a performance regression. Please remove performance and apply critical-fix.

Environment

• Language binding: Rust
• Storage backend used by the regression: local object store
• Affected area: rust/lance/src/dataset/mem_wal