JOB-4: LLM cost & latency observability + spend guardrails

[Pragadeesh122]

Summary

JOB-4 — make LLM spend and latency visible and bounded on the existing Prometheus/Grafana/OTel stack. Most per-request token/cost tracking and latency dashboards already existed; this PR closes the remaining acceptance gaps: RAG retrieval latency percentiles, a per-session spend guardrail, and a documented baseline.

Changes

Spend guardrail (new) — api/session_budget.py

• Redis-backed cumulative prompt+completion token counter per chat session (shares the 24h session TTL).
• Enforced at the top of both general (api/chat.py) and project (api/project_chat.py) chat turns. Once MAX_SESSION_TOKENS (default 2,000,000, env-tunable, 0 disables) is exceeded, the turn is refused with an error SSE + budget_exceeded done event.
• Fails open on any Redis error — an observability guardrail must never take down the chat path.
• Complements the existing per-turn context cap (MAX_PROMPT_TOKENS → summarization), which bounds a single request; this bounds a whole session.

Latency visibility

• New agenticrag_retrieval_duration_seconds histogram (cache_status = hit/miss), emitted from pipeline/retriever.py — gives p50/p95 for RAG retrieval.
• New agenticrag_session_budget_blocked_total{chat_type,limit} counter for guardrail trips.

Grafana

• UX & Latency: RAG Retrieval Latency (p50/p95) + Chat/Stream Completion Latency (p50/p95) by Mode.
• Economics: Session Spend Guardrail Blocks.

Docs — docs/architecture/observability.md

• New Spend Guardrails section (both layers, defaults, env vars) and a Baseline section with copy-paste PromQL for cost/1K-tokens and p50/p95 latency, plus a bounded worst-case cost model.

Acceptance criteria

• Per-request token + cost by model and chat mode — Economics dashboard (pre-existing) documented.
• p50/p95 latency for chat/stream and RAG retrieval — added retrieval histogram + p50/p95 panels.
• A basic guardrail with sane defaults, documented — per-session token ceiling + existing context caps, all documented.
• Baseline noted — methodology, read queries, and bounded cost model in docs.

Security / isolation note

The guardrail reads/writes only a hashed-key Redis counter and never logs raw session content; it fails open so it can't break chat. No auth/permission/RAG-isolation surfaces are touched.

Testing

• tests/test_session_budget.py (10 tests): budget check/record/enforce, fail-open, disabled mode, and metric emitters.
• Full related suite green: test_session_budget test_llm_stream_metrics test_retrieval_cache test_chat_orchestration test_observability_context → 33 passed.

🤖 Generated with Claude Code