Skip to content

feat(rag): index the Obsidian memory vault into Qdrant (semantic memory search)#77

Merged
AlienWalker1995 merged 1 commit into
mainfrom
feat/memory-vault-adoption
Jul 9, 2026
Merged

feat(rag): index the Obsidian memory vault into Qdrant (semantic memory search)#77
AlienWalker1995 merged 1 commit into
mainfrom
feat/memory-vault-adoption

Conversation

@AlienWalker1995

Copy link
Copy Markdown
Owner

Phase 2B of the Obsidian memory system (follows #76): notes the agent stores through the memory-vault MCP become semantically searchable via qdrant_search — one nomic-768 space shared by ingest and query.

What

  • v2/plugins/rag/plugin.yaml — mounts the vault read-only at /watch/memory-vault, nested inside the existing watch root, so the recursive watcher/scanner picks it up with zero ingester config. Same ${MEMORY_VAULT_PATH} var the memory-vault MCP mounts (one vault, two interfaces).
  • rag-ingestion/ingest.py — hidden-path rule: any path component starting with . is skipped (.obsidian/ workspace JSON, .trash/ soft-deleted notes, .gitkeep stubs), on both the periodic scan and watchdog-event paths.
  • v2/plugins/rag/plugin.yamlllamacpp-embed gains --batch-size 2048 --ubatch-size 2048. Root-cause fix for a live 500 caught during validation: embeddings require each input to fit in one physical batch, and a 400-word markdown chunk routinely exceeds the upstream n_ubatch=512 default (input (556 tokens) is too large to process. increase the physical batch size (current batch size: 512) — CONVENTIONS.md, live).

Validation (live on ordo-v2)

  • Throwaway-container gate: ruff clean, 189 passed / 2 skipped (baseline 181/2; +8 new tests: vault ro-mount render, contract-unchanged, CPU shape, embed batch flags, hidden-path unit/scan/refusal).
  • Re-rendered in a --gpus all container from --source out/ordo.yaml: compose diff = only the vault mount line + the 4 embed batch-flag lines; llamacpp chat block byte-identical (pin GPU-97fe65ee unchanged). Applied with up -d --no-deps to rag-ingestion and llamacpp-embed only.
  • Ingestion proof: documents points 1 → 6 (5 vault notes + smoketest); .obsidian/.trash/.gitkeep.md excluded; the pre-fix 500 on CONVENTIONS.md cleared on the first scan after the batch fix (Ingested memory-vault/CONVENTIONS.md).
  • Semantic proof through the qdrant-rag MCP (the agent's channel): query "how should dashboards be validated" → top hit 0.7377 = memory-vault/memories/user/dashboard-validation-by-content.md, the note Hermes itself wrote via the memory-vault MCP in Phase 2A.

🤖 Generated with Claude Code

…ry search)

Phase 2B of the memory system: vault notes written through the memory-vault
MCP become semantically searchable via qdrant_search (same nomic space).

- plugins/rag: mount the vault READ-ONLY at /watch/memory-vault, nested
  inside the existing watch root so the recursive watcher/scanner ingests it
  with zero ingester config (same ${MEMORY_VAULT_PATH} the MCP mounts).
- rag-ingestion: hidden-path rule — skip any path component starting with a
  dot, keeping vault internals (.obsidian/ workspace JSON, .trash/ soft
  deletes, .gitkeep stubs) out of the index. Applies to both the periodic
  scan and watchdog events.
- plugins/rag: raise llamacpp-embed --batch-size/--ubatch-size to 2048.
  Root-cause fix for a live 500 ("input (556 tokens) is too large ...
  physical batch size (current batch size: 512)"): embeddings require each
  input to fit in ONE physical batch, and a 400-word markdown chunk
  routinely exceeds the upstream n_ubatch=512 default.
- tests: 8 new (render: vault ro-mount, contract unchanged, CPU shape,
  embed batch flags; ingester: hidden-path unit + scan + refusal paths).

Validated live on ordo-v2: vault ingested (points 1 -> 6, .obsidian/.trash
excluded), semantic query "how should dashboards be validated" returns the
agent-written vault note at 0.74 via the qdrant-rag MCP. llamacpp chat
block byte-identical (GPU-97fe65ee pin unchanged); only rag-ingestion +
llamacpp-embed recreated.

Co-Authored-By: Claude Fable 5 <[email protected]>
@AlienWalker1995
AlienWalker1995 merged commit 1098cf0 into main Jul 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant