feat: semantic search — pgvector vector(1024) + HNSW, Ollama default#3
Merged
Conversation
…provider Index standard is 1024 by design: arctic-embed2 native, Qwen3-Embedding MRL ceiling, OpenAI truncatable via the dimensions param. Model swaps are re-embedding jobs (provenance columns exist for that), never schema migrations. 1536 is deliberately rejected — it is OpenAI-shaped, no local model emits it. - alembic 004: embedding JSONB -> vector(1024) NOT NULL + HNSW cosine index (guarded: refuses on non-empty table) - search_memory mode=semantic: query embedding + '<=>' cosine ranking, per-hit distance in results; fts mode unchanged - OllamaProvider implemented (/api/embed) and made the DEFAULT provider (local-first); default model snowflake-arctic-embed2; built-in dims map + SCOPED_MCP_EMBEDDING_DIMENSIONS override for unknown models - OpenAIProvider truncates to 1024 (matryoshka) instead of native 1536 - write-path guard: non-1024 embeddings refused — the index can never silently mix dimensions - live-verified on pgvector/pg17: migrations 001->004, semantic ranking (distance ~0 for exact-match stub vector, ordered correctly), plus the full prior integration suite. 131 unit + 5 live tests. Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the 1024 index standard decided today (arctic-embed2 native / Qwen3 MRL ceiling / OpenAI-truncatable; model churn = re-embed job, never a schema migration).
embeddingJSONB →vector(1024)+ HNSW cosine index (empty-table guarded)search_memorymodesemantic(now the tool-schema default): cosine ranking, per-hitdistance/api/embed) and promoted to default — local-first; default modelsnowflake-arctic-embed2; dims map + explicit override for unknown modelsdimensionsparam (matryoshka) — no more 1536 anywhereLive-verified against
pgvector/pg17: all 4 migrations, semantic ranking correctness (deterministic stub vectors: exact match at distance ≈0, ordered), full prior integration suite. 131 unit + 5 live integration tests.Deploy note: after merge,
ollama pull snowflake-arctic-embed2on the Mini (tags list is currently empty) and runalembic upgrade headbefore first write.🤖 Generated with Claude Code