feat(infra): LiteLLM gateway for task-based model routing + observability (#478, PR1)#490
Open
adi-devv wants to merge 2 commits into
Open
feat(infra): LiteLLM gateway for task-based model routing + observability (#478, PR1)#490adi-devv wants to merge 2 commits into
adi-devv wants to merge 2 commits into
Conversation
…lity (beenuar#478) Adds a LiteLLM proxy as the single entry point for live LLM calls. AiSOC requests a logical task alias (aisoc-triage / recon / investigation / copilot / summary / report / nl); the alias -> real-model mapping lives entirely in infra/litellm/config.yaml, so operators assign local or hosted models per task and swap them without any AiSOC code change. Per-task latency, tokens, cost, errors, retries and fallbacks are exported on /metrics and scraped by the bundled Prometheus (new aisoc-litellm job; the third-party image is allowlisted in the target audit so CI stays green). Opt-in and non-breaking: unset OPENAI_BASE_URL keeps calls going direct to the provider, and the deterministic offline path is unaffected. PR1 of 2 for beenuar#478 (gateway + observability). A follow-up wires the ~10 in-code callsites to request these aliases via model_pins and removes the hardcoded gpt-4o-mini default, which closes beenuar#478. Contents: - infra/litellm/config.yaml: 7 task aliases, prometheus callback, env-based master key + provider keys, unauthenticated /metrics for scraping, commented Ollama/vLLM/Anthropic alternatives - docker-compose.yml: litellm service + LLM_GATEWAY_URL on agents/api - infra/docker/prometheus.yml + scripts/audit_prometheus_targets.py: scrape job + third-party allowlist - .env.example, apps/docs/docs/operations/llm-gateway.md (+ sidebar), CHANGELOG - services/agents/tests/test_litellm_config.py: 6 config contract tests Live-verified: the proxy boots against the config, all 7 aliases load, alias routing reaches the provider, and /metrics serves per-alias counters. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fix the I001 import-order the CI ruff check gate flagged; no behavioural change. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #478 — PR1 of 2 (gateway + observability). Additive and non-breaking.
What this adds
A LiteLLM proxy as the single entry point for live LLM calls. AiSOC requests a logical task alias; the alias → real-model mapping lives entirely in
infra/litellm/config.yaml, so operators assign different local or hosted models per task — and swap them — without any AiSOC code change.Aliases mirror AiSOC's distinct workloads:
aisoc-triage,aisoc-recon,aisoc-investigation,aisoc-copilot,aisoc-summary,aisoc-report,aisoc-nl.Observability: per-task latency, tokens, cost, errors, retries and fallbacks are exported on
/metricsand scraped by the bundled Prometheus (newaisoc-litellmjob). The third-party image is allowlisted inscripts/audit_prometheus_targets.pyso the CI target audit stays green.Contents
infra/litellm/config.yaml— 7 aliases, prometheus callback, unauthenticated/metricsfor scraping, env-based keys, commented Ollama/vLLM/Anthropic alternativesdocker-compose.yml—litellmservice +LLM_GATEWAY_URLonagents/apiinfra/docker/prometheus.yml+scripts/audit_prometheus_targets.py— scrape job + third-party allowlist.env.example,apps/docs/docs/operations/llm-gateway.md(+ sidebar),CHANGELOG.mdservices/agents/tests/test_litellm_config.py— 6 config contract testsVerification
audit_prometheus_targets.py --checkexits 0; compose + config YAML parse./health/liveliness200,/v1/modelslists all 7 aliases,aisoc-triageroutes to the provider,/metricsserves per-alias counters. (Surfaced and fixed a real gap:/metricsneedsrequire_auth_for_metrics_endpoint: falseor unauthenticated Prometheus scrapes 401.)Follow-up
PR2 (
feat/litellm-alias-routing) wires the in-code callsites to request these aliases and removes the hardcodedgpt-4o-minidefault, closing #478.🤖 Generated with Claude Code