feat: Centralised configuration utility and urgent bug fixes

[nicoloesch]

Summary

Adapts omop-emb to the oa-configurator configuration layer, replacing environment-variable-based setup with a typed TOML-backed config and a omop-config configure omop_emb subcommand.

• FIXES Include oa-configurator for centralised configuration #23

Notes

Due to the urgency, this PR also absorbed four bug fixes uncovered while doing the migration, plus two more found auditing those fixes:

• FIXES FAISSCache._load_index calls read_index() on every invocation - no in-memory caching of the loaded index object #24
• FIXES Distance Metric Inconsistency - FAISS vs Database Backends #27
• FIXES OOM Error on export + wrongly normalised FAISS indices #30

Includes also a registry staleness gap and a FAISS cache round-tripping gap, neither tracked by an existing issue (see "Additional fixes" below)

Changes

Configuration (#23)

• OmopEmbConfig subclasses PackageConfigBase, exposing all package settings as typed Pydantic fields backed by [tools.omop_emb] in ~/.config/omop/config.toml.
• Entry point registered under omop.config so omop-config configure omop_emb prompts interactively or via named flags (--host, --database-name, --backend, --provider-type, …).
• OmopEmbConfig.get_config() / Resolver.from_active_config() / OmopEmbConfig.configure_logging(verbosity=…) replace the old standalone env-var-driven equivalents
• EmbeddingReaderInterface no longer reaches into global config itself for faiss_cache_dir. Instead, it's a plain, fully-explicit constructor argument now (None means no FAISS, unambiguously). Resolution against the active config moves to the one place that owns it: the CLI (cli_embeddings.py's search command) and omop-graph's KnowledgeGraph. This also means the test suite no longer has an incidental, unrelated dependency on a config file existing on disk.
• CI (release.yml) now runs omop-config configure omop_emb non-interactively before pytest, provisioning a real test_emb_db resource
  • pgvector integration tests run for real in CI instead of silently skipping.
• docker-compose.yaml updated from --resource-set/--set flags to the current named flags.

FAISS index caching + efSearch (#24)

• FAISSCache._load_index() re-read the index from disk on every search call, and HNSWIndexConfig.ef_search was never applied at query time.
  • Fixed with an in-memory index cache, keyed on (metric_type, index_config) so alternating between multiple metric/index combinations on one instance doesn't thrash a single slot

Distance-metric inconsistency, FAISS vs DB backends (#27)

• FAISS's IndexFlatIP (COSINE) returned raw inner product ∈ [-1, 1] but was fed directly into a conversion formula expecting cosine distance ∈ [0, 2], compressing and sometimes inverting similarity scores relative to pgvector/sqlite-vec. - IndexFlatL2 similarly returned squared Euclidean distance where the shared formula expected true (non-squared) distance.
• Both are now converted to the correct convention before scoring (1 - IP for cosine, sqrt() for L2), verified against the DB backends with parity tests.

OOM on export + silent COSINE corruption (#30)

• FAISSCache.export() accumulated every batch into a Python list before a single vstack, peaking at ~2x dataset size in memory and getting OOM-killed before any disk write on large exports.
• Separately, the only on-disk artifact was the .faiss file itself, which for COSINE holds L2-normalized vectors. Reimporting from it silently discarded original magnitudes.

Fix

• Replaced with a streamed HDF5 bundle (storage/embedding_bundle.py) as the single source of truth
  • raw, never-normalized embeddings + metadata, written via chunked dataset-slice assignment
  • no list accumulation, no vstack: peak memory is one batch regardless of total row count).
• FAISS is now a derived, rebuildable accelerator built from the bundle, never the round-trip artifact
  • a narrow reconstruct_to_bundle() escape hatch covers migrating standalone legacy .faiss files with a clear data-loss warning.
• CLI gained generic export/import bundle operations and build-faiss-cache.
• FAISS pre-filtering (concept_filter) moved from a frozen metadata.npz snapshot to a live query against the backend (get_concept_ids_matching_filter), so filtering can never drift from what the database currently says.

Additional fixes found during this work

• Registry staleness gap: embedding upserts never bumped model_registry.updated_at, so FAISSCache.is_fresh() could report a cache fresh indefinitely even after new embeddings were added.

  • Fixed with refresh_model_updated_at_timestamp(), called after every upsert path through EmbeddingWriterInterface.

• import_bundle() force-overwrite staleness gap: re-importing into an already-registered model with force=True replaced its content outright but never bumped updated_at.

  • Fixed by setting updated_at during this operation

• Registry timestamp precision: refresh_model_updated_at_timestamp switched from DB-side func.now() to a Python-side timestamp, since SQLite's CURRENT_TIMESTAMP only has whole-second resolution

• Unified how both embedding backends' define and access their table schema, replacing independent hardcoded/string-based column handling with one shared, typed source of truth.

  • Centralized column definitions into a single spec shared by both backends
  • Rebuilt sqlite-vec on real SQLAlchemy Core objects instead of raw SQL strings
  • Tied record/table types to the shared spec so they can't silently drift
  • Strengthened typing across both backends' table-descriptor handling

[gkennos]

_build_filter_positions reads metadata.npz from disk on every call

effectively the same cache-miss problem this PR just fixed for the FAISS index

At 1.8M concepts the .npz file is >100 MB - re-reading per query will dominate latency for filtered searches

Consider adding _metadata_cache: Optional[Tuple[np.ndarray, ...]] attribute alongside _index_cache (holding concept_ids, domain_ids, vocabulary_ids, is_standard, is_valid arrays in memory once loaded), invalidated on the same freshness check that governs _index_cache

freshness check should verify the .npz file mtime, not just .json sidecar.

[nicoloesch]

This no longer applies. _build_filter_positions and the metadata.npz-based filtering path are gone. Concept filters are now resolved live against the backend (get_concept_ids_matching_filter), and the per-index .faiss/.json freshness check already covers staleness correctly (tested in test_is_fresh_false_after_later_upsert). metadata.npz is kept only for one-time migration of pre-existing legacy caches.

[gkennos]

Now that _load_index caches the index in memory, get_embeddings_by_concept_ids can bypass the DB using faiss.IndexIDMap.reconstruct_batch(ids)

current implementation always hits self._backend regardless of FAISS config

for bulk lookups this is a significant unnecessary DB round trip.

Suggested resolution:

add FAISSCache.get_embeddings_by_concept_ids(concept_ids) using index.reconstruct_batch(np.array(concept_ids, dtype=np.int64)), then call from EmbeddingReaderInterface.get_embeddings_by_concept_ids when self._faiss_cache is available / fresh

Prerequisite for the centroid/joint-embedding capability

[nicoloesch]

Eager-loading at construction vs. lazy-loading on first search() doesn't reduce total work. faiss.read_index runs exactly once either way, this only moves when. That only pays off for a long-lived process that constructs the interface once and serves many requests afterward, and ONLY for the first request as subsequent request are solved through the cache. If the index would change between requests, this pattern could also actively "waste" work: faiss_index_config is supplied per-call to get_nearest_concepts, independent of construction time, so eagerly loading one config on the chance it's the one used later can load an index that's never queried.

[gkennos]

FAISSCache._index_cache is initialised to None

first search() call after server startup pays full disk-read cost even though the cache object was just created

this update fixes repeated-call overhead but not cold-start latency, which is user-facing on the first request

Consider adding a FAISSCache.warm(metric_type, index_config) method that calls _load_index eagerly, and calling from EmbeddingReaderInterface.__init__ when self._faiss_cache.is_fresh(model_record, ...) returns true

--> moves load cost to server startup where it is expected and invisible to end users

[nicoloesch]

Looked into this in depth, and there isn't a version of it that's worth doing due to the following reasons:

1. COSINE-metric indices store L2-normalized vectors, not the raw values the DB returns, so this would be a silent behavior change for COSINE models.
2. the index wrapper this code actually uses (IndexIDMap) doesn't support reconstruct/reconstruct_batch at all in the installed faiss-cpu
   • only IndexIDMap2 does, meaning a breaking on-disk cache format change and a rebuild of every existing .faiss file
   • a single missing key fails the whole batch call, so we'd need to reimplement the DB path's per-ID-missing error contract ourselves.
3. reconstruct_batch also has no concept of filtering, so anywhere we actually combine filters with ID lookups we're hitting the DB regardless.
4. The one real caller of get_embeddings_by_concept_ids (stream_embedding_batches, used by both export and by FAISSCache.build_from_backend itself) calls the backend method directly, never through the interface layer this comment targets
   • implementing it there wouldn't speed up the one place it's actually used
   • Doing it where it would help means pulling FAISS-awareness into the backend classes themselves, which currently have zero knowledge of FAISS by design.

Don't think it is worth it...

[gkennos]

get_config() calls load_stack_config(), which raises FileNotFoundError when ~/.config/omop/config.toml does not exist

tool is completely non-functional on a fresh machine before omop-config configure has been run

even when the user supplies every required parameter on the command line

[nicoloesch]

resolve_backend() and resolve_omop_cdm_engine() both need config internally and have no CLI override for backend selection or CDM engine connection, so the same FileNotFoundError would just resurface a few lines later regardless of which flags are supplied. Making config genuinely optional isn't something this command structure supports today without also exposing backend/DB overrides on the CLI, which is a bigger change than this fix.

What's actually missing: oa-configurator already validates required resources with an actionable error once a config file exists (points the user at omop-config configure); the only gap is the bare FileNotFoundError when the file is completely absent, which has no such guidance. Wrapping that specific case to point at omop-config configure too, rather than trying to make any of this configuration-free.

[gkennos]

only these ones to block deployment (either easy or significant)

• silent wrong results every time a user omits --model with a non-default configured model. Easy to hit, silent, wrong output.
• SQLite :memory: fallback - this is just a warning
• concept_filter.limit ignored in sqlite-vec: confirmed behavioural divergence between backends. Callers depending on the limit get wrong counts silently.
• OmopEmbConfig.get_config() unconditional: tool is completely broken on fresh install if omop-config configure hasn't been run - blocks onboarding