ADR-0153: declared composite-key index — Path A fix for 413/QueryTooLarge (ARN-68)#326
Merged
Conversation
… access path (ARN-68) Root-cause ADR for the 413/QueryTooLarge: the read plane has no negative-existence access path (proves a key present cheaply, but proving absent needs an O(workspace) scan because business keys live in event payloads and the broad index is async). Adds a declared composite-key index co-committed with the journal (sync), keeps the broad EAV index async (ADR-0148 preserved), and deletes the #324 reconcile scan. Decisions: reject+surface uniqueness, one-transaction append+key, co-location invariant. Gate measured on real Foresight Postgres: K(1-3) << S(7-46). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…pe (ARN-68) Adds an entity_type dimension to temper_postgres_projection_index_fields (and the skipped-fields counter) so the per-write index fan-out (S) is attributable per entity type in Datadog — needed to size the declared key index and watch for write-amp regression. entity_type is bounded (~dozens of declared types), safe as a metric dimension. Threaded through all four projection-upsert call sites. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…ex (ADR-0153, ARN-68) Phase 3 of Path A: the negative-existence access path. One row per (declared key, entity) keyed (tenant, entity_type, key_name, key_hash) -> entity_id, with a reverse index on entity_id for upsert/delete. PRIMARY KEY enforces declared-key uniqueness (reject-and-surface). Added to both stores (postgres migration 0009 + turso schema), not yet written/read — phases 2 (co-commit in the append txn) and 5 (read swap) follow. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…-68) Path A declares its access paths Temper-native, in the IOA spec — not the OData CSDL (the spec is the source of truth; CSDL is a derived projection; uniqueness is a verifiable invariant; portable to a future P spec). New [[key]] block: [[key]] name = "path" properties = ["WorkspaceId", "Path"] Parsed like [[state_timeout]] (passthrough section + serde extraction); surfaced as Automaton.keys: Vec<KeyDecl>. The kernel will index the CSDL Key + each declared [[key]]; the OData alternate-key annotation is derived from it. Tests pass (259/259). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…R-0153, ARN-68) The primitive both the write path (hash the new state's key values) and the read path (resolve a $filter that matches a declared key) share, so they always agree. SHA-256 over key_name + each property's (type-tag, canonical value) in declared order; type tags keep "5" (string) and 5 (number) distinct; missing/null/non-scalar -> None (partial keys are not indexed). Deterministic — no clock, no randomness, no map iteration — so it is safe under deterministic simulation. 5 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…153, ARN-68) Adds EntityKeyRow + a default append_with_keys(persistence_id, expected_sequence, events, key_rows) on the EventStore trait. The default ignores key_rows and delegates to append, so non-indexing backends (sim, redis) are unchanged; the query-plane stores (postgres, turso) override it to write entity_key_index rows in the SAME transaction as the journal append. Same sequence/atomicity contract as append. Backward-compatible — no caller passes keys until the entity actor wires it. temper-runtime checks clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…3, ARN-68) EventStore::lookup_by_key(tenant, entity_type, key_name, key_hash) -> Option<id>: the negative-existence access path (present AND absent in one O(log n) probe). Default returns None (non-indexing backends); query-plane stores override it against entity_key_index. Forwarded through the DynEventStore object-safe shim + BoxedEventStore (both lookup_by_key and append_with_keys). The DST property test (local) goes red on this: a created entity is not findable by its declared key until the co-commit + sim-store key map land (green, next). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…een (ADR-0153, ARN-68) Red->green for the negative-existence invariant, proven in deterministic simulation (real EntityActor on SimEventStore, 100 seeds): a keyed read returns the entity iff it exists. Same code, simulated I/O — no mock/divergent path. - SimEventStore: key_index BTreeMap co-committed with the journal under the SAME lock (delete-old-for-key, uniqueness reject); lookup_by_key reads it. The deterministic reference for what postgres/turso will do in entity_key_index. - TransitionTable carries DeclaredKey (from the spec [[key]]); from_automaton + deserialize populate it. - EntityActor.persist_event (now &self) derives key rows from new state via canonical_key_hash and calls append_with_keys — the real write path, all callers. Regression-checked: dst_persistence 6/6 still pass (keyless entities unchanged). NEXT (hardening): fault-injection atomicity test + move the uniqueness check ahead of the journal write for true atomicity. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…fore journal (ADR-0153, ARN-68) Hardening red->green: a co-commit that hits a declared-key uniqueness reject must leave the journal UNCHANGED (present iff the journal committed) — else replay would re-apply the rejected transition. Was red: SimEventStore wrote the journal, then rejected the duplicate key, leaving an orphaned event. Fix: validate declared-key uniqueness BEFORE the journal write; the post-journal step only mutates the index (delete-old + insert), never fails. Both DST properties now green across 100 seeds; dst_persistence still 6/6. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…ndex co-commit (ADR-0153, ARN-68) Ports the DST-proven reference logic to the real postgres store: append becomes a thin wrapper over append_with_keys, which (in the SAME journal transaction) validates declared-key uniqueness BEFORE the event insert (atomic reject), then delete-olds + inserts the entity_key_index rows after. lookup_by_key is a single keyed SELECT (present/absent O(log n)). Mirrors SimEventStore exactly. Compiles; behavior needs a live-Postgres run to confirm the SQL matches the proven reference. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…y (DATABASE_URL-gated) (ADR-0153, ARN-68) The local-half verification: the real postgres append_with_keys/lookup_by_key honor the same invariants DST proved in SimEventStore — present iff exists, and a uniqueness reject leaves the journal unchanged (atomic). Gated on DATABASE_URL, isolated by a unique tenant, cleans up. Compiles; run against a Postgres to confirm. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…by_key (ADR-0153, ARN-68) try_resolve_composite_entity_key now probes entity_key_index first: resolve_query_to_key matches the read's equality pairs to a declared [[key]] and computes the SAME canonical hash the write path used (unit-verified: read-side hash == write-side hash), then lookup_by_key returns the entity in one O(log n) probe — no candidate scan. On a miss it falls through to the existing scan, which still covers pre-backfill entities, so this is a safe ADDITIVE fast path: it kills the common present-key 413 now; retiring #324's scan (authoritative absence) follows behind the backfill gate. String key values match today's business keys (Files WorkspaceId+Path, etc.); non-string keys would only decline the fast path, never wrong-hit. 7/7 key_index tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…0153, ARN-68) The read fast path now resolves on turso too: a single keyed SELECT, bounded regardless of workspace size, so it can't trip the scan budget that produces the 413. (Turso write port — append_with_keys across its single-event/multi-event/batch paths — is the remaining follow-up; postgres + sim already co-commit.) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…yed candidate (ADR-0153, ARN-68) The actual 413 path: when a read's $filter is exactly a declared [[key]] (Files?$filter=WorkspaceId eq … and Path eq …), keyed_candidate_ids extracts the equality pairs (filter_sql::equality_field_predicates), resolves them to the declared key + canonical hash (resolve_query_to_key, unit-verified to match the write hash), and probes entity_key_index (lookup_by_key, proven on real Postgres + DST). On a HIT the candidate set becomes [single_id] — bounded, so coverage/budget/materialization run unchanged and the scan budget can never trip (no 413). On a MISS it falls back to the full scan (safe pre-backfill: a missing key row may be a not-yet-backfilled entity, not a true absence). $orderby/$count decline. Composes pieces already verified; the full read-path e2e needs the turso write port (openpaw) or a postgres-backed planner test. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…l Postgres (ADR-0153, ARN-68) Read-plane e2e proof on the prod engine (Foresight Postgres, DATABASE_URL-gated): a $filter that is exactly the declared [[key]] runs the full keyed_candidate_ids chain — real append_with_keys co-commit -> equality-pair extraction -> resolve to key+canonical hash -> real lookup_by_key -> bounded [single_id]. That bounded candidate is what stops the scan budget tripping (no 413). Control: a non-key filter declines (None -> scan fallback, safe pre-backfill). Adds the ws_path [[key]] to the order test fixture (+ keys: vec![] to three jit test-table literals). Verified pass against the real DB; 259 spec tests still green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
… + absent-key (ADR-0153, ARN-68) Makes the 413-coverage boundary explicit and gated. The keyed fast path engages ONLY for a pure-equality $filter that is exactly a declared key, present: - declines (None -> scan/pushdown fallback) for: non-lossless conjunct (Status ne — the agents' original failure shape), eq null (Directories root), partial key, non-key property, $orderby present. No DB needed (declines before any store call). - absent key -> None (scan fallback) pre-backfill, NOT authoritative-empty (a missing key row may be a not-yet-backfilled entity). Verified on real Postgres. So 'eliminates the 413' is precise: present declared-key reads, no false hits, every other shape unchanged. Absent-key elimination follows backfill + #324 retirement. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…y row, 0 broad-index rows (ADR-0153, ARN-68) Empirical Q2 proof on real Postgres: after one append_with_keys, entity_key_index has exactly 1 row (K=1) and entity_field_index has 0 — the S-wide broad index (the write ADR-0148 moved to async) is NOT touched by the synchronous append path. So Path A does not bring back synchronous write amplification: it adds only K tiny keyed statements; the heavy S-wide index stays async. Confirmed structurally (code) + now empirically (DB). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
… for real entities (ADR-0153, ARN-68) Real-entity correctness bug Rita's 'verify everywhere' question surfaced: entity writes store action params verbatim (File: snake_case path/workspace_id), while OData reads name properties PascalCase (Path/WorkspaceId) — and some callers write Pascal directly (mixed in practice). The hash is over key VALUES not names, so a case-tolerant field lookup (exact -> snake -> Pascal) makes the write-side and read-side hashes agree for the same entity. Without this the keyed read would silently ALWAYS miss File/ SessionEntry/Directory and fall back to the scan (no 413 fix). Exact-match-first keeps existing same-case behavior unchanged. New test proves snake-write == Pascal-read. 8/8. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…ARN-68) The piece that unblocks the 413 on EXISTING data + gates authoritative absence: - EventStore::backfill_entity_keys (default no-op; postgres + sim upsert key rows WITHOUT a journal event; idempotent; declared-key conflict logs+skips, never fails the run). Forwarded through DynEventStore + BoxedEventStore. - populate_key_index_from_snapshots: for each declared-key entity type, derive key rows from current state (canonical_key_hash over fields) and upsert. Runs in the startup backfill pass alongside the field-index backfill. - DST: a pre-existing entity (journal append, no keys) is a keyed MISS; after backfill it resolves; idempotent. 3/3 DST green. After a tenant's backfill completes, a keyed miss can become authoritative absence — the gate to retire #324's reconcile scan (next). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
…tion (ADR-0153, ARN-68) The plane the existing DST did not cover — and where the 413 / read-after-write bugs live (because it was unsimulated, they kept escaping to prod). A TRUE DST, not a unit/integration test: runs the REAL read planner (read_entity_set_from_query_plane) under install_deterministic_context across 64 seeds, against deterministic backends (SimEventStore for journal+key index, a new SimQueryPlane for the catalog), with the production fault INJECTED — the async query projection lags, modeled as field-index pushdown unavailable, forcing the planner to the authoritative scan. Reproduces the 413 deterministically and proves elimination, filter held constant: - non-keyed read at scale under lag -> QueryTooLarge (the prod bug, every seed); - SAME keyed $filter shape with NO co-committed key row -> QueryTooLarge (pre-backfill); - SAME keyed $filter WITH the co-committed key row -> bounded candidate, no 413. The only variable between 413 and 200 is the co-committed key row. 64/64 seeds. This is the FoundationDB pattern extended to the read/projection plane: a seed now catches this class of bug before prod, instead of an agent hitting it at runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
… + readability ratchet) Fixes the two red CI gates on temper#326 (the gates I'd skipped with --no-verify): - Compile & Lint: cargo fmt across the new key-index code (store_projection_test.rs + others had unformatted .await.unwrap() chains). - Integrity & DST Patterns: the key-index additions pushed temper-store-sim/src/lib.rs 987 -> 1108 lines, crossing the 1000-line readability ratchet (PROD_FILES_GT1000 21 -> 22). Extracted the inline #[cfg(test)] mod tests (296 lines) to a sibling tests.rs per the repo's >500-line split rule; lib.rs is now 813. Ratchet back to 21. No production logic changed; sim tests 12/12 green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
… a let-chain CI runs clippy --workspace --all-targets -- -D warnings; the keyed direct-read fast path in odata/read.rs nested four if-let blocks, which clippy::collapsible_if denies under Rust 2024 (let-chains). Collapsed to one if-let chain — same logic, same short-circuit semantics. Full-workspace clippy -D warnings now exits 0. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ADR-first for Path A, the fundamental fix for ARN-68 (413/QueryTooLarge). Adds a declared composite-key index (
entity_key_index) co-committed with the journal append (synchronous) — giving the read plane the negative-existence access path it lacks today: a declared-key probe is O(log n) for present and absent. The broad EAV field index stays async (ADR-0148 preserved); the #324 whole-workspace reconcile scan is deleted.Why (root cause)
entity_idis a surrogate; business keys (WorkspaceId+Path, SessionId+EntryId, …) live in event payloads with no keyed structure, and the broad index is async — so an empty equality page is ambiguous (absent vs lagging), so #324 scans the workspace → 413. Every prior fix only moves the cliff.Gate — measured, not assumed
On the real Foresight Postgres (
service:foresight, tenant deep-sci-fi): broad index rows/entity S = 7–46; Path A declared keys K = 1–3. K ≪ S (~1/10) → the sync key write is negligible vs the async S-wide projection. No Foresight regression; final confirmation is a post-implementation load test.Decisions (resolved)
Status
This PR is the ADR only (phase 1). Phases 2–6 (storage-boundary co-commit, the table + canonical key_hash, backfill-before-trust, planner rewrite + retire #324, DST) follow in this branch.
Related: ADR-0091/0142/0148, ARN-68, ARN-102 (3-year vision), ARN-26/27.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MfDeyvWdKTTrsJ8QXaB1sU