Quality eval harness for chat + RAG answers (JOB-3)

[Pragadeesh122]

Summary

Makes answer quality measurable so we can ship RAG/agent changes with confidence. Builds on the existing evals/ harness, which was written against a stale schema and had never successfully run — this PR makes it actually execute end-to-end, fixes a silently-broken retrieval metric, wires it into CI, and records a baseline.

What changed

Harness fixes (it now runs)

• Eval runner targeted plural tables (projects/documents) but the live schema is singular (project/document), and inserted a string user_id into a non-nullable UUID FK. Now creates an ephemeral user + project correctly and tears both down.
• Retrieval relevance metrics (recall@k, MRR, NDCG) were always 0 — they compared expected dataset filenames against the vector store's source (a {document_id}.ext storage name). Now maps each retrieved chunk's document_id back to its original filename.

Repeatable + one command

• make eval / make eval-retrieval / make eval-unit.

CI (advisory at first)

• Always-on PR gate: the offline eval unit tests now run in pr.yml (added evals/** + tests/** to the api paths filter). Added tests/test_eval_judge.py (judge parsing + fallback); metrics were already covered.
• Advisory eval.yml: manual + weekly, spins up the stack via compose, runs the full eval against hosted Pinecone + an LLM key, uploads reports as an artifact and writes the aggregate table to the job summary. continue-on-error, gated on secrets (no-ops without them).

Docs + baseline

• Rewrote evals/README.md (how to add cases, read results, baselines, CI).
• Committed evals/baselines/smoke.md with current scores.
• Fixed .gitignore: evals/reports/ was mis-merged into a nested path and never actually ignored.

Baseline (smoke: 4 docs, 10 queries; gen gpt-5.4, judge gpt-4o-mini)

Retrieval	Mean		Answer quality (1–5)	Mean
recall@5	1.000		faithfulness	5.00
ndcg@5	0.605		completeness	5.00
mrr	0.475		hallucination	4.60
substring_recall	1.000		format_adherence	4.30

Expected docs always retrieved (recall@5=1.0); ordering has headroom (MRR/NDCG). Answers fully grounded and complete.

Verification

• make eval-unit → 34 passed (offline).
• Full uv run python evals/run_eval.py --dataset smoke executed against the local stack + hosted Pinecone + OpenAI; ephemeral resources cleaned up.

Notes / follow-ups

• Action needed (CEO): add repo secrets PINECONE_API_KEY and OPENAI_API_KEY to enable the advisory eval.yml runs; until then it no-ops.
• The judge's negatively-framed hallucination dimension occasionally inverts its scale (scored 1 while reasoning "no hallucinations present"). Documented as a caveat; candidate follow-up is to reword it to groundedness.
• Security/data-isolation: the eval only ever touches its own unique Pinecone namespace and a throwaway DB user/project, all deleted on completion.

🤖 Generated with Claude Code