feat: [[vector]] declared access paths + Temper.Nearest kNN (ARN-159, RFC-0003)#341
Merged
Conversation
…(ARN-159) Slice 1 of ARN-159 (kernel-native vector similarity, ADR-0155). Adds the `[[vector]]` spec declaration — name, property, model_property, dims, metric — as a sibling of `[[key]]`, parsed via the same serde second pass and carried onto the JIT TransitionTable as `DeclaredVector`. The verification cascade rejects a vector path whose property/model_property is not a declared state variable, whose dims is 0, whose metric is not one of cosine/dot/l2, or whose name repeats. No index or read surface yet — those are the following slices. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
Slice 2 of ARN-159 (ADR-0155). Makes vector similarity a live, budgeted, deterministic read on the sim backend end to end: - runtime EventStore gains EntityVectorRow / EntityVectorCandidate and a single co-commit entry point `append_with_index_rows` (keys + vectors in one tx); append_with_keys now forwards to it, so existing callers are unchanged. Plus vector_candidates / backfill_entity_vectors / mark_vector_index_backfilled / vector_index_backfilled_types / vectored_entity_ids_for_type (defaults inert). - sim store co-commits vector rows under the journal lock and serves candidates in deterministic entity-id order; the entity actor parses each declared [[vector]] from post-transition state and co-commits it. - new temper-server::vector_index: little-endian f32 packing, tolerant vector parsing (JSON array or JSON string), exact-scan ranking (cosine/dot/l2) with the determinism contract — f32 accumulation in candidate order, ties by entity id. - Temper.Nearest: a collection-bound GET function /<Set>/Temper.Nearest(decl,to|vector,k,model,filter). Named-arg URL parsing in temper-odata; the handler resolves the model partition, charges the candidate budget (413 on overflow), ranks, then materializes + equality-filters + read-authorizes lazily in score order until k accepted (== filter then top-k), emitting the OData list shape with a per-row @temper.score. - DST: seeded writes + Nearest ranking reproduce across 100 seeds; model partitioning and self-exclusion hold. 9 vector_index unit tests, path-parse tests. Postgres + Turso backends and the backfill driver are the following slices. Co-Authored-By: Claude Fable 5 <[email protected]>
…RN-159) Slice 3 of ARN-159 (ADR-0155). Postgres now maintains entity_vector_index: - migration 0012 adds entity_vector_index (packed-LE-f32 BYTEA blob, PK on tenant/type/decl/model_tag/entity, partition + reverse indexes) and vector_index_backfill_watermark. - the event append co-commits vector rows in the SAME transaction as the journal (delete-then-insert per decl so a changed vector/model tag replaces cleanly), mirroring the declared-key co-commit — so a kNN read is consistent with committed state without a scan. - backfill_entity_vectors / vector_candidates (ORDER BY entity_id for deterministic ranking) / mark_vector_index_backfilled / vector_index_backfilled_types / vectored_entity_ids_for_type implemented against the new tables. - the LE-f32 blob encoders move to temper-runtime beside EntityVectorRow so the stores and the kernel ranking share one byte layout (server re-exports them). Co-Authored-By: Claude Fable 5 <[email protected]>
Slice 4 of ARN-159 (ADR-0155). Turso now maintains entity_vector_index: - schema adds entity_vector_index (BLOB vector, partition + reverse indexes) and vector_index_backfill_watermark, created at store init. - append_with_index_rows is overridden write-behind: the event append runs first (with its retry logic), then the derived vector rows follow in a separate write. Safe for vectors (no uniqueness constraint) where it is not for keys — a lagging or failed index write only makes a ranking temporarily incomplete, which the backfill watermark accounts for; it can never corrupt a keyed absence. - vector_candidates (ORDER BY entity_id) / backfill_entity_vectors (upsert) / mark_vector_index_backfilled / vector_index_backfilled_types / vectored_entity_ids_for_type implemented; TenantStoreRouter forwards them all so kNN works on the routed production deployment. - a local-SQLite turso test proves write-behind candidates, model partitioning, upsert-replace, and the watermark/id-listing roundtrip. Co-Authored-By: Claude Fable 5 <[email protected]>
…RN-159) Slice 5 of ARN-159 (ADR-0155), completing kernel v1: - populate_vector_index_from_snapshots: the declared-vector backfill, mirroring the declared-key one — authoritative enumeration, strict per-entity state load, parse each declared [[vector]] to a row, idempotent upsert, and a per-(tenant,type) watermark set only when every existing entity was indexed or is skippable. Reuses a shared EntityLoadOutcome/load_entity_current_fields helper factored out of the key backfill (no duplication). - wired into boot (bootstrap spawns it per tenant after the key backfill) and into the post-spec-change reconcile (so a newly declared [[vector]] path re-indexes existing entities immediately), watermark-gated so unchanged types skip. - Temper.Nearest reads reference/filter fields from the temper OData body's nested `fields` object (fixed via a `fields`-aware, snake/Pascal-tolerant lookup). - HTTP end-to-end test through the real axum router: create entities (vectors co-commit), GET Temper.Nearest by reference (self-excluded, ordered, @temper.score) and by raw vector, plus a 400 for an unknown decl. Kernel v1 is complete across sim/postgres/turso; consumer cutovers are a separate effort. Co-Authored-By: Claude Fable 5 <[email protected]>
… ratchet (ARN-159) The vector EventStore surface adds six methods, each needing the dyn-adapter forwarding trio (DynEventStore trait decl + blanket impl + BoxedEventStore wrapper) in storage/mod.rs — unavoidable boilerplate parallel to the existing key-index methods. This grows the file past the readability ratchet's file-size thresholds, so the baseline is intentionally updated (the sanctioned path); the snapshot also locks in pre-existing improvements (lower println / dead-code / unwrap counts). `append_with_index_rows` has six args, under clippy's threshold, so its `#[allow(clippy::too_many_arguments)]` was removed. Co-Authored-By: Claude Fable 5 <[email protected]>
…budget (ARN-159) Fixes all findings from the PR #341 review (ADR-0155 updated with the details): F1/F4 — deleted/cleared entities were ranked forever and ate the candidate budget. The write path now RECONCILES an entity's vector rows (delete all, insert current) whenever its type declares vectors, gated by a new `reconcile_vectors` flag on `append_with_index_rows`; a soft-deleted entity (`status="Deleted"`) emits no rows so it is purged, and a cleared vector/model property drops that path. `backfill_entity_vectors` is now a reconcile (empty rows purge); the backfill purges deleted/phantom entities it loads. Defense in depth: the Nearest walk skips any `Deleted` body and `to='<deleted>'` returns 404. F2 — the `to=` reference entity is now read-authorized (same Cedar gate as a normal read) before its existence/embedding or the ranking it seeds is disclosed. F3 — Turso vector maintenance is retried (the event-append retry primitives), not a warn-once one-shot; on exhaustion it logs loudly. Reconcile handles the turso-side delete. Kept in the EventStore layer (not the field-index queue) because that queue is spec-agnostic while vector parsing needs the spec, and routing there would double-maintain on Postgres. Fixed the trait doc drift. F5 — metric accumulation moved to f64 and the score is dropped if the narrowed f32 is non-finite; the blob decoder rejects non-finite components. An overflowing/corrupt vector declines instead of producing a NaN that would sort first. F6 — `vector_candidates` applies `LIMIT budget+1` in the store, so an over-budget partition 413s without loading it all; the N-at-most walk posture is documented. F7 — `Temper.Nearest` rejects any OData system query option ($top/$select/$filter/…) with 400 instead of silently ignoring it. Suggestions: reject duplicate named args; dispatch by the fully-qualified name (`Temper.Nearest`) so a same-named function elsewhere can't route here; the backfill signature now includes property/model_property/dims/metric so an in-place edit re-indexes; `$metadata` advertisement deferred (documented). Tests added: delete→Nearest exclusion + to=deleted 404; equality-filter e2e (filter before top-k); reference-authz-denied 403 + walk-row read-denied skipped; query-option and duplicate-arg rejection; turso delete/purge reconcile; overflow/NaN ranking. Full temper-server suite green under workspace features (the 2 spec-validate cases need the observe feature the workspace unifies). Co-Authored-By: Claude Fable 5 <[email protected]>
The review fixes add correctness logic (reconcile/authz/status handling) and two more trait-forwarder parameters (reconcile_vectors, limit) to the dyn-adapter boilerplate, nudging the file-size ratchet. Intentional baseline update (the sanctioned path). Co-Authored-By: Claude Fable 5 <[email protected]>
Collaborator
Author
|
Ready for review. Independent review: 2 rounds, final PASS. Round 1 found 7 findings (1 critical: deleted entities served forever; 1 security: unauthorized Live-local server run (Rita's standing rule — beyond the suite): booted temper-server on a local libSQL store, loaded a real
Suites: nearest_odata e2e 7/7, DST 2/2 (100 seeds), vector_index 12/12, turso/sim store green; clippy/fmt clean. Residuals (tracked, non-blocking for merge): postgres co-commit wants one live-Postgres integration run before production cutover; ARN-205 (pre-existing append_batch index gap, shared with declared keys); consumer cutovers are the separate phase-C effort. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+213
to
+229
| if failed == 0 { | ||
| if let Some((store, _)) = state.event_journal() | ||
| && let Err(e) = store | ||
| .mark_vector_index_backfilled(tenant.as_str(), entity_type, ¤t_set) | ||
| .await | ||
| { | ||
| tracing::error!( | ||
| tenant = %tenant, entity_type = %entity_type, error = %e, | ||
| "vector index backfill: failed to persist watermark" | ||
| ); | ||
| } | ||
| tracing::info!( | ||
| tenant = %tenant, entity_type = %entity_type, vector_set = %current_set, | ||
| total, newly_indexed, already, skipped, | ||
| "entity_vector_index backfill complete; type watermarked" | ||
| ); | ||
| } else { |
There was a problem hiding this comment.
tracing::info! fires unconditionally within if failed == 0, even when mark_vector_index_backfilled returns an error. Guard the info log so it only runs when the watermark write actually succeeded.
Suggested change
| if failed == 0 { | |
| if let Some((store, _)) = state.event_journal() | |
| && let Err(e) = store | |
| .mark_vector_index_backfilled(tenant.as_str(), entity_type, ¤t_set) | |
| .await | |
| { | |
| tracing::error!( | |
| tenant = %tenant, entity_type = %entity_type, error = %e, | |
| "vector index backfill: failed to persist watermark" | |
| ); | |
| } | |
| tracing::info!( | |
| tenant = %tenant, entity_type = %entity_type, vector_set = %current_set, | |
| total, newly_indexed, already, skipped, | |
| "entity_vector_index backfill complete; type watermarked" | |
| ); | |
| } else { | |
| if failed == 0 { | |
| let watermarked = if let Some((store, _)) = state.event_journal() { | |
| match store | |
| .mark_vector_index_backfilled(tenant.as_str(), entity_type, ¤t_set) | |
| .await | |
| { | |
| Ok(()) => true, | |
| Err(e) => { | |
| tracing::error!( | |
| tenant = %tenant, entity_type = %entity_type, error = %e, | |
| "vector index backfill: failed to persist watermark" | |
| ); | |
| false | |
| } | |
| } | |
| } else { | |
| false | |
| }; | |
| if watermarked { | |
| tracing::info!( | |
| tenant = %tenant, entity_type = %entity_type, vector_set = %current_set, | |
| total, newly_indexed, already, skipped, | |
| "entity_vector_index backfill complete; type watermarked" | |
| ); | |
| } | |
| } else { |
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/temper-server/src/state/projection_backfill/vector_index.rs
Line: 213-229
Comment:
The success `tracing::info!` fires unconditionally within `if failed == 0`, even when `mark_vector_index_backfilled` returns an error. Guard the info log so it only runs when the watermark write actually succeeded.
```suggestion
if failed == 0 {
let watermarked = if let Some((store, _)) = state.event_journal() {
match store
.mark_vector_index_backfilled(tenant.as_str(), entity_type, ¤t_set)
.await
{
Ok(()) => true,
Err(e) => {
tracing::error!(
tenant = %tenant, entity_type = %entity_type, error = %e,
"vector index backfill: failed to persist watermark"
);
false
}
}
} else {
false
};
if watermarked {
tracing::info!(
tenant = %tenant, entity_type = %entity_type, vector_set = %current_set,
total, newly_indexed, already, skipped,
"entity_vector_index backfill complete; type watermarked"
);
}
} else {
```
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+249
to
+262
| async { Ok(Vec::new()) } | ||
| } | ||
|
|
||
| /// Record that `entity_vector_index` is **complete** for `(tenant, entity_type)` | ||
| /// — every existing entity has had its declared vectors indexed by the backfill | ||
| /// (ADR-0155 watermark, mirroring `mark_key_index_backfilled`). `vector_set` is | ||
| /// the sorted, comma-joined declared vector-path NAMES the backfill covered, so a | ||
| /// later declaration of an ADDITIONAL path is detected as a set change and the | ||
| /// type is re-indexed. Idempotent. Default no-op. | ||
| fn mark_vector_index_backfilled( | ||
| &self, | ||
| tenant: &str, | ||
| entity_type: &str, | ||
| vector_set: &str, |
There was a problem hiding this comment.
vector_set
The doc says "vector_set is the sorted, comma-joined declared vector-path NAMES". The actual format produced by declared_vector_set_signature is semicolon-joined full descriptors (name:property:model_property:dims:metric), not comma-joined names. A future store implementer following this doc would produce an incompatible format, breaking the changed-set detection in the backfill.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/temper-runtime/src/persistence/mod.rs
Line: 249-262
Comment:
**Doc comment describes the wrong format for `vector_set`**
The doc says "`vector_set` is the sorted, comma-joined declared vector-path **NAMES**". The actual format produced by `declared_vector_set_signature` is semicolon-joined full descriptors (`name:property:model_property:dims:metric`), not comma-joined names. A future store implementer following this doc would produce an incompatible format, breaking the changed-set detection in the backfill.
How can I resolve this? If you propose a fix, please make it concise.
Comment on lines
+347
to
+397
| let mut value: Vec<serde_json::Value> = Vec::with_capacity(k); | ||
| for scored in ranked { | ||
| if value.len() >= k { | ||
| break; | ||
| } | ||
| let materialized = materialize_entity_set_entities( | ||
| state, | ||
| tenant, | ||
| &entity_type, | ||
| &set_name, | ||
| std::slice::from_ref(&scored.entity_id), | ||
| true, | ||
| None, | ||
| ) | ||
| .await; | ||
| let Some(mut body) = materialized.entities.into_iter().next() else { | ||
| continue; | ||
| }; | ||
| // Defense in depth: never serve a soft-deleted entity, even if its vector row | ||
| // has not yet been purged by the write-path reconcile / backfill. | ||
| if is_deleted(&body) { | ||
| continue; | ||
| } | ||
| if let Some(pairs) = &equality_filter | ||
| && !body_matches_equality(&body, pairs) | ||
| { | ||
| continue; | ||
| } | ||
| if authorize_read( | ||
| state, | ||
| tenant, | ||
| security_ctx, | ||
| READ_ACTION, | ||
| &entity_type, | ||
| &scored.entity_id, | ||
| &body, | ||
| ) | ||
| .is_err() | ||
| { | ||
| continue; | ||
| } | ||
| if let Some(obj) = body.as_object_mut() { | ||
| obj.insert( | ||
| "@temper.score".to_string(), | ||
| serde_json::Number::from_f64(scored.score as f64) | ||
| .map(serde_json::Value::Number) | ||
| .unwrap_or(serde_json::Value::Null), | ||
| ); | ||
| } | ||
| value.push(body); | ||
| } |
There was a problem hiding this comment.
Each scored candidate triggers a separate materialize_entity_set_entities call with a single-element slice. With a tight equality filter or narrow Cedar policy, the function can make up to candidate_budget individual store reads before collecting k results. The PR description notes "inline filter in v1" as a known decision; tracking a batched-materialization follow-up would make this concrete.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/temper-server/src/odata/nearest.rs
Line: 347-397
Comment:
**Per-candidate entity materialization produces N individual store reads**
Each scored candidate triggers a separate `materialize_entity_set_entities` call with a single-element slice. With a tight equality filter or narrow Cedar policy, the function can make up to `candidate_budget` individual store reads before collecting `k` results. The PR description notes "inline filter in v1" as a known decision; tracking a batched-materialization follow-up would make this concrete.
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kernel-v1 of RFC-0003: vector similarity as a first-class declared access path. Slices: spec decl + cascade verification; sim index + Temper.Nearest + DST; postgres co-commit; turso write-behind + watermark; backfill driver + boot wiring + HTTP e2e. Decisions: bound function (agent-first), inline filter in v1, turso watermark semantics, exact scan under the existing budget, deterministic ordering. DRAFT until the implementing agent's full verification report lands (it resumes after a usage-limit reset).
🤖 Generated with Claude Code
https://claude.ai/code/session_01VebbwT1GQeWibjsZd9YPQy
Greptile Summary
This PR implements kernel-v1 of RFC-0003/ADR-0155: vector similarity as a declared access path. It introduces a
[[vector]]spec declaration, a newentity_vector_indextable (Postgres migration + Turso DDL), co-commit / write-behind vector maintenance in the entity actor and store backends, a boot-time backfill driver with watermark semantics, and theTemper.NearestOData bound function for exact-scan kNN queries under Cedar gates and a candidate budget.temper-spec,temper-jit): AddsVectorDecl/DeclaredVectorparsed from[[vector]]TOML blocks; the TransitionTable carries avectorsfield used by the actor and backfill.entity_actor/actor.rs, all store backends): Vector rows are co-committed with the journal append on Postgres/sim and written-behind with retries on Turso; delete transitions emit emptyvector_rowswithreconcile_vectors = trueto purge stale index entries.odata/nearest.rs,vector_index.rs):Temper.Nearestresolves a partition's candidate rows, ranks them in-kernel via f64-accumulated cosine/dot/L2 scoring (deterministic, overflow-safe), then lazily materializes + equality-filters + Cedar-gates each result untilkare collected.Confidence Score: 3/5
The core vector index machinery is well-constructed and consistent across backends. One concrete defect in the backfill driver emits a misleading success log even when the watermark write failed, masking persistent failures from operators.
The ranking math is thoroughly tested and overflow-safe. The write path is consistent across backends. The backfill watermark logic has a concrete defect where the success log fires unconditionally even after a watermark write error. The per-candidate one-at-a-time materialization in handle_nearest is a stated v1 tradeoff. The doc mismatch on the vector_set format in the trait comment would mislead a future store implementer.
projection_backfill/vector_index.rs (watermark log logic) and persistence/mod.rs (trait doc for mark_vector_index_backfilled)
Important Files Changed
Temper.NearestOData bound function implementing exact-scan kNN. Cedar gates and candidate budget are correctly enforced; per-candidate materialization is a stated v1 trade-off.mark_vector_index_backfilledincorrectly describes the vector_set format.Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Agent participant EntityActor participant Store as Store (PG/Sim/Turso) participant VectorIndex as entity_vector_index participant NearestHandler as Temper.Nearest Note over EntityActor,Store: Write path Agent->>EntityActor: POST /tdata/Items/Embed EntityActor->>EntityActor: derive key_rows + vector_rows from post-state EntityActor->>Store: "append_with_index_rows(events, key_rows, vector_rows, reconcile_vectors=true)" Store->>VectorIndex: DELETE entity rows, INSERT current rows Store-->>EntityActor: Ok(new_seq) Note over NearestHandler: Read path Agent->>NearestHandler: "GET /tdata/Items/Temper.Nearest(decl='taste',to='ref-id',k=10)" NearestHandler->>NearestHandler: Cedar list gate NearestHandler->>Store: vector_candidates(tenant, type, decl, model_tag, budget+1) Store->>VectorIndex: SELECT entity_id, vector ORDER BY entity_id LIMIT budget+1 VectorIndex-->>Store: candidates Store-->>NearestHandler: Vec EntityVectorCandidate NearestHandler->>NearestHandler: rank_nearest() f64 cosine/dot/L2 loop for each scored entity until k collected NearestHandler->>Store: materialize_entity(entity_id) NearestHandler->>NearestHandler: filter + Cedar read gate end NearestHandler-->>Agent: "OData response with @temper.score" Note over Store: Boot backfill Store->>VectorIndex: vectored_entity_ids_for_type loop for each un-indexed entity Store->>Store: load_entity_current_fields Store->>VectorIndex: backfill_entity_vectors end Store->>VectorIndex: mark_vector_index_backfilled%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Agent participant EntityActor participant Store as Store (PG/Sim/Turso) participant VectorIndex as entity_vector_index participant NearestHandler as Temper.Nearest Note over EntityActor,Store: Write path Agent->>EntityActor: POST /tdata/Items/Embed EntityActor->>EntityActor: derive key_rows + vector_rows from post-state EntityActor->>Store: "append_with_index_rows(events, key_rows, vector_rows, reconcile_vectors=true)" Store->>VectorIndex: DELETE entity rows, INSERT current rows Store-->>EntityActor: Ok(new_seq) Note over NearestHandler: Read path Agent->>NearestHandler: "GET /tdata/Items/Temper.Nearest(decl='taste',to='ref-id',k=10)" NearestHandler->>NearestHandler: Cedar list gate NearestHandler->>Store: vector_candidates(tenant, type, decl, model_tag, budget+1) Store->>VectorIndex: SELECT entity_id, vector ORDER BY entity_id LIMIT budget+1 VectorIndex-->>Store: candidates Store-->>NearestHandler: Vec EntityVectorCandidate NearestHandler->>NearestHandler: rank_nearest() f64 cosine/dot/L2 loop for each scored entity until k collected NearestHandler->>Store: materialize_entity(entity_id) NearestHandler->>NearestHandler: filter + Cedar read gate end NearestHandler-->>Agent: "OData response with @temper.score" Note over Store: Boot backfill Store->>VectorIndex: vectored_entity_ids_for_type loop for each un-indexed entity Store->>Store: load_entity_current_fields Store->>VectorIndex: backfill_entity_vectors end Store->>VectorIndex: mark_vector_index_backfilledPrompt To Fix All With AI
Reviews (1): Last reviewed commit: "chore(vector): record review-fix growth ..." | Re-trigger Greptile