Summary
The anatomic location hybrid search returns badly wrong results for straightforward multi-word queries:
- "cardiac chamber" → returns eye chambers instead of heart chambers
- "cardiac muscle" → returns shoulder muscles instead of myocardium
The semantic (vector) component appears to be matching on generic terms ("chamber", "muscle") while ignoring the qualifying anatomic context ("cardiac").
Investigation areas
- Embedding quality — are the stored embeddings capturing anatomic context, or just individual word similarity? Check what vectors are stored for cardiac structures vs. eye/shoulder structures.
- FTS vs. semantic weighting — the hybrid search blends FTS (BM25) and vector scores. If FTS correctly ranks cardiac results higher but the semantic score overwhelms it, the weights need adjustment.
- Query preprocessing — is the full query string being embedded, or is it being tokenized/split in a way that loses the multi-word context?
- Index coverage — verify the anatomic location database actually contains the expected cardiac entries (heart chambers, myocardium, etc.) with appropriate synonyms.
- Eval suite —
evals/anatomic_search.py has 10 test cases. Check whether these cases cover multi-word anatomic queries and whether scores are acceptable. Add failing cases for the examples above.
Reproduction
from anatomic_locations import AnatomicLocationIndex
async with AnatomicLocationIndex() as idx:
results = await idx.search("cardiac chamber")
# Expected: heart chambers (left ventricle, right atrium, etc.)
# Actual: eye chambers (anterior chamber, posterior chamber, etc.)
Summary
The anatomic location hybrid search returns badly wrong results for straightforward multi-word queries:
The semantic (vector) component appears to be matching on generic terms ("chamber", "muscle") while ignoring the qualifying anatomic context ("cardiac").
Investigation areas
evals/anatomic_search.pyhas 10 test cases. Check whether these cases cover multi-word anatomic queries and whether scores are acceptable. Add failing cases for the examples above.Reproduction