Skip to content

Add route outcome diagnostics#50

Open
guelfoweb wants to merge 1 commit into
mainfrom
route-outcome-observability
Open

Add route outcome diagnostics#50
guelfoweb wants to merge 1 commit into
mainfrom
route-outcome-observability

Conversation

@guelfoweb

Copy link
Copy Markdown
Owner

Summary:

  • Adds env-gated non-content route outcome diagnostics under ORBIT_KV_DIAG.
  • Emits kv_diag_route_outcome events correlated with request_id, model_call_id, pass_index, phase, and tools_mode.
  • Classifies route outcomes as route_direct_final_stop, route_no_decision_length_retry, route_parsed_tool, route_parsed_chat, route_invalid_output, or route_other_retry.
  • Logs only metadata: finish reason, decision type, output length/token count, and retry reason.
  • Does not log raw prompt, raw route output, raw tool output, or user-visible content.
  • Documents the metric in docs/ROUTE_OUTCOME_OBSERVABILITY.md.

Validation:

  • PYTHONPATH=src python3 -m unittest tests.test_kv_diag -q: PASS
  • python3 -m unittest discover -s tests -q: PASS
  • python3 -m compileall -q src tests scripts: PASS
  • git diff --check: PASS

Scope:

  • Diagnostics only
  • No prompt changes
  • No routing policy changes
  • No tool-selection changes
  • No backend/native/MTP changes
  • No final policy changes
  • No KV/cache behavior changes
  • No visible behavior changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant