feat(infra): LiteLLM gateway for task-based model routing + observability (#478, PR1)

.env.example

@@ -85,6 +85,24 @@ QDRANT_URL=http://localhost:6333
 OPENAI_API_KEY=sk-your-openai-api-key-here
 OPENAI_MODEL=gpt-4-turbo-preview
 
+# ─── LLM gateway (LiteLLM) — issue #478 ───────────────────────────────────────
+# LiteLLM is the single entry point for live LLM calls. AiSOC asks for a logical
+# task alias (aisoc-triage, aisoc-investigation, …); the alias → real-model map
+# lives in infra/litellm/config.yaml, so you can swap local/hosted models there
+# without changing AiSOC. The service is defined in docker-compose.yml.
+#
+# Key that AiSOC uses to authenticate to the gateway (also LiteLLM's admin key):
+LITELLM_MASTER_KEY=sk-aisoc-local
+# In-network URL of the gateway (already set per-service in docker-compose.yml):
+LLM_GATEWAY_URL=http://litellm:4000/v1
+#
+# To ROUTE AiSOC through the gateway, uncomment the two lines below — they make
+# every service send its calls to LiteLLM instead of straight to the provider.
+# Put your real provider key in OPENAI_API_KEY above (LiteLLM reads it), then:
+#   OPENAI_BASE_URL=http://litellm:4000/v1
+#   OPENAI_API_KEY=${LITELLM_MASTER_KEY}
+# Leave them unset to keep calling the provider directly (unchanged behaviour).
+
 # ─── Threat Intelligence ──────────────────────────────────────────────────────
 # Open-source / community feeds (work out of the box, no key required for some)
 TAXII_FEEDS=https://cti-taxii.mitre.org/taxii/,enterprise-attack,,

CHANGELOG.md

@@ -9,6 +9,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **LLM gateway (LiteLLM) — task-based model routing + observability
+  ([#478](https://github.com/beenuar/AiSOC/issues/478), PR1).** New `litellm`
+  service in `docker-compose.yml` as the single entry point for live LLM calls.
+  AiSOC requests a **logical task alias** (`aisoc-triage`, `aisoc-recon`,
+  `aisoc-investigation`, `aisoc-copilot`, `aisoc-summary`, `aisoc-report`,
+  `aisoc-nl`); the alias → real-model mapping lives entirely in
+  `infra/litellm/config.yaml`, so operators assign different local or hosted
+  models per task — and swap them — without any AiSOC code change (commented
+  Ollama/vLLM/Anthropic examples ship in the config). Per-task latency, tokens,
+  cost, errors, retries, and fallbacks are exported on `/metrics` and scraped by
+  the bundled Prometheus (new `aisoc-litellm` job; the third-party image is
+  allowlisted in `scripts/audit_prometheus_targets.py`). Opt-in and
+  non-breaking: unset `OPENAI_BASE_URL` keeps calls going direct to the provider,
+  and the deterministic offline path is unaffected. Docs:
+  `apps/docs/docs/operations/llm-gateway.md`; config test
+  `services/agents/tests/test_litellm_config.py`. (A follow-up PR wires the ~10
+  in-code callsites to request these aliases via `model_pins` and removes the
+  hardcoded `gpt-4o-mini` default, closing #478.)
 - **v8 P4 — Compounding Memory (verdicts that measurably improve).** New
   `services/fusion/app/memory/`: a nightly-distillable institutional memory that
   makes verdicts more accurate the longer an instance runs. **Distillation**

apps/docs/docs/operations/llm-gateway.md

@@ -0,0 +1,96 @@
+---
+title: LLM gateway (LiteLLM)
+description: Route every live LLM call through a single LiteLLM gateway — assign local or hosted models per task by alias, and get centralized latency/token/cost/error metrics — without changing AiSOC code.
+---
+
+# LLM gateway (LiteLLM)
+
+AiSOC runs several distinct LLM workloads — triage, recon, investigation, the
+contextual copilot, summaries, reports, and natural-language generation. The
+**LiteLLM gateway** is the single entry point for every *live* LLM call these
+workloads make. AiSOC asks for a **logical task alias**; the gateway decides
+which real provider and model that alias resolves to.
+
+```
+AiSOC task ──▶ alias (e.g. "aisoc-triage") ──▶ LiteLLM ──▶ real model
+```
+
+This gives operators two things without any AiSOC code change:
+
+1. **Per-task model assignment.** Point `aisoc-triage` at a cheap local model
+   and `aisoc-investigation` at a strong hosted one — or swap either at any time
+   — by editing one config file.
+2. **Centralized observability.** LiteLLM exports per-task latency, tokens,
+   cost, errors, retries, and fallbacks on `/metrics`, scraped by the bundled
+   Prometheus (job `aisoc-litellm`). This complements the
+   [Investigation Ledger](../concepts/llmops.md), which records *what the agent
+   decided*; the gateway records *what each model call cost and how it behaved*.
+
+The gateway sits in front of the LLM tier of the
+[multi-model router](../concepts/model-router.md). When no live model is
+reachable, AiSOC still degrades to its **deterministic offline path** — the
+gateway is never on the critical path for a baseline triage.
+
+## Task aliases
+
+The shipped aliases mirror AiSOC's workloads. They live in
+`infra/litellm/config.yaml`:
+
+| Alias                 | Workload                                   | Shipped default   |
+| --------------------- | ------------------------------------------ | ----------------- |
+| `aisoc-triage`        | Auto-triage of fused alerts (high volume)  | `gpt-4o-mini`     |
+| `aisoc-recon`         | Recon / enrichment reasoning               | `gpt-4o-mini`     |
+| `aisoc-investigation` | Deep multi-step investigation              | `gpt-4o`          |
+| `aisoc-copilot`       | Contextual analyst copilot                 | `gpt-4o-mini`     |
+| `aisoc-summary`       | Alert / incident summaries                 | `gpt-4o-mini`     |
+| `aisoc-report`        | Analyst-facing report write-ups            | `gpt-4o`          |
+| `aisoc-nl`            | NL→query / NL→detection translation        | `gpt-4o-mini`     |
+
+The "shipped default" is only the *example* mapping in the config — the whole
+point is that you change it. The alias names stay constant.
+
+## Enable the gateway
+
+The `litellm` service is defined in `docker-compose.yml` and starts with the
+stack. To route AiSOC through it, set in your `.env`:
+
+```bash
+LITELLM_MASTER_KEY=<a-strong-key>          # AiSOC authenticates to the gateway with this
+OPENAI_API_KEY=<your-real-provider-key>    # LiteLLM uses this to reach the upstream model
+OPENAI_BASE_URL=http://litellm:4000/v1     # send AiSOC's calls to the gateway
+# and set AiSOC's client key to the gateway key:
+# OPENAI_API_KEY=${LITELLM_MASTER_KEY}     # (in the AiSOC services' environment)
+```
+
+Leave `OPENAI_BASE_URL` unset to keep calling the provider directly — the
+gateway then runs idle (still serving `/health` and `/metrics`) and behaviour is
+unchanged.
+
+## Re-point a task to a local model
+
+Duplicate the alias in `infra/litellm/config.yaml` with a local backend. The
+alias name **must stay the same** so AiSOC is unaware of the swap:
+
+```yaml
+- model_name: aisoc-triage
+  litellm_params:
+    model: ollama/llama3.1
+    api_base: http://ollama:11434
+```
+
+Commented Ollama, vLLM, and Anthropic examples ship in the config. For a fully
+offline deployment, see [air-gapped operation](./air-gapped.md), which fronts a
+local Ollama.
+
+## Observe
+
+- **Metrics:** `curl http://localhost:4000/metrics` (or the Grafana/Prometheus
+  stack under the `monitoring` profile) shows `litellm_*` counters broken down
+  by task alias and model.
+- **Health:** `curl http://localhost:4000/health/liveliness`.
+
+## Notes
+
+- Host port `4000` is bound to `127.0.0.1` only, like the rest of the stack.
+- No provider key is ever written to `infra/litellm/config.yaml` — aliases
+  resolve credentials from the process environment (`os.environ/...`).

apps/docs/sidebars.ts

@@ -125,6 +125,7 @@ const sidebars: SidebarsConfig = {
         "operations/secrets",
         "operations/airgap",
         "operations/air-gapped",
+        "operations/llm-gateway",
         "operations/codespaces",
         "operations/theming",
         "operations/upgrades",

docker-compose.yml

@@ -241,6 +241,10 @@ services:
       SECRET_KEY: dev_secret_key_change_in_production
       ENVIRONMENT: development
       LOG_LEVEL: info
+      # Issue #478 — LiteLLM gateway. Route this service's live LLM endpoints
+      # (translation, hunts, KB, phishing, NL-detection) through the gateway by
+      # setting OPENAI_BASE_URL=http://litellm:4000/v1 + OPENAI_API_KEY=$LITELLM_MASTER_KEY.
+      LLM_GATEWAY_URL: http://litellm:4000/v1
     # Phase 2.6 — use /readyz, not /health, so the orchestrator
     # only routes traffic once the lifespan has finished wiring
     # Postgres, Redis, Kafka, OpenSearch, Qdrant, and Neo4j.
@@ -381,11 +385,54 @@ services:
       OPENAI_API_KEY: ${OPENAI_API_KEY:-}
       ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
       OPENAI_MODEL: ${OPENAI_MODEL:-gpt-4o-mini}
+      # Issue #478 — the LiteLLM gateway (below) is the single entry point for
+      # live LLM calls when enabled. To route this service through it, set in
+      # .env: OPENAI_BASE_URL=http://litellm:4000/v1 and
+      # OPENAI_API_KEY=$LITELLM_MASTER_KEY. Left unset, calls go direct to the
+      # provider (unchanged behaviour).
+      LLM_GATEWAY_URL: http://litellm:4000/v1
       ATTCK_DATA_PATH: /app/data/enterprise-attack.json
     networks:
       - aisoc
     restart: unless-stopped
 
+  # ─── LLM gateway (LiteLLM) — issue #478 ─────────────────────────────────────
+  # Single entry point for every live LLM call. AiSOC requests a logical task
+  # alias (aisoc-triage, aisoc-investigation, …); the mapping alias → real model
+  # lives entirely in infra/litellm/config.yaml, so operators swap local/hosted
+  # models without touching AiSOC. Per-task latency/tokens/cost/errors are
+  # exported on /metrics (scraped as job `aisoc-litellm`). Lightweight and inert
+  # unless a service is pointed at it via OPENAI_BASE_URL — see the note on the
+  # `agents` service above.
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    container_name: aisoc-litellm
+    command: ["--config", "/etc/litellm/config.yaml", "--port", "4000"]
+    volumes:
+      - ./infra/litellm/config.yaml:/etc/litellm/config.yaml:ro
+    environment:
+      # Clients authenticate to the gateway with this key (AiSOC sends it as its
+      # OPENAI_API_KEY when OPENAI_BASE_URL points here). Override in .env.
+      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:-sk-aisoc-local}
+      # Upstream provider keys the aliases resolve against (os.environ/... in the
+      # config). Empty is fine — the gateway still boots, serves /health and
+      # /metrics, and AiSOC falls back to its deterministic path.
+      OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+    ports:
+      - "127.0.0.1:4000:4000"
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:4000/health/liveliness',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"
+      interval: 15s
+      timeout: 5s
+      retries: 10
+      start_period: 30s
+    networks:
+      - aisoc
+    restart: unless-stopped
+
   osquery-tls:
     build:
       context: ./services/osquery-tls

infra/docker/prometheus.yml

@@ -93,6 +93,17 @@ scrape_configs:
       - targets: ['threatintel:8005']
     metrics_path: /metrics
 
+  # LiteLLM gateway (issue #478) — third-party image that exports
+  # per-task LLM latency, tokens, cost, errors, retries and fallbacks
+  # on /metrics via its prometheus callback (see
+  # infra/litellm/config.yaml). Not a service in our `services/` tree,
+  # so it's allowlisted in scripts/audit_prometheus_targets.py under
+  # THIRD_PARTY_HOSTS. Container listens on 4000 inside the network.
+  - job_name: 'aisoc-litellm'
+    static_configs:
+      - targets: ['litellm:4000']
+    metrics_path: /metrics
+
   # Phase 2.4 — self-scrape Alertmanager so we have visibility
   # into the notification pipeline (delivery failures show up as
   # `alertmanager_notifications_failed_total > 0`).

infra/litellm/config.yaml

@@ -0,0 +1,113 @@
+# ─── AiSOC LiteLLM gateway config (issue #478) ──────────────────────────────────
+#
+# LiteLLM is the single entry point for every *live* LLM call AiSOC makes. AiSOC
+# asks for a logical **task alias** (the `model_name` values below); this file —
+# and only this file — decides which real provider/model each alias resolves to.
+# Operators re-point a task to a different local or hosted model here, with no
+# AiSOC code change and no redeploy of the app services.
+#
+#   AiSOC task ──▶ alias (e.g. "aisoc-triage") ──▶ LiteLLM ──▶ real model
+#
+# The aliases below mirror AiSOC's distinct LLM workloads. Keep this set in sync
+# with the role pins in services/agents/app/llm/model_pins.py — the agents test
+# `test_litellm_config.py` asserts they match.
+#
+# Every alias resolves its credential from the process env (os.environ/...), so
+# no secret is ever written to this file. Point AiSOC at the gateway by setting
+# (see .env.example):
+#     OPENAI_BASE_URL=http://litellm:4000/v1
+#     OPENAI_API_KEY=$LITELLM_MASTER_KEY
+#
+# Air-gapped / fully-local deployments: swap the commented Ollama/vLLM blocks in
+# for the `openai/*` lines — the alias names stay identical, so AiSOC is unaware
+# of the switch. When no live model is reachable AiSOC still degrades to its
+# deterministic offline path (the `deterministic` floor in model_pins.py).
+
+model_list:
+  # ── Auto-triage: high volume, cheap+fast. ─────────────────────────────────────
+  - model_name: aisoc-triage
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Recon / enrichment reasoning. ─────────────────────────────────────────────
+  - model_name: aisoc-recon
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Deep investigation: the multi-step reasoning path — worth a stronger model.
+  - model_name: aisoc-investigation
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Contextual copilot: interactive analyst Q&A. ──────────────────────────────
+  - model_name: aisoc-copilot
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Summaries: alert/incident condensation. ───────────────────────────────────
+  - model_name: aisoc-summary
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Reports: analyst-facing write-ups — quality over cost. ────────────────────
+  - model_name: aisoc-report
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Natural-language generation: NL→query / NL→detection translation. ─────────
+  - model_name: aisoc-nl
+    litellm_params:
+      model: openai/gpt-4o-mini
+      api_key: os.environ/OPENAI_API_KEY
+
+  # ── Local / air-gapped example (Ollama) ───────────────────────────────────────
+  # Duplicate any alias above with a local backend to keep the call on-prem. The
+  # alias name MUST stay the same so AiSOC is unaware of the swap:
+  #
+  # - model_name: aisoc-triage
+  #   litellm_params:
+  #     model: ollama/llama3.1
+  #     api_base: http://ollama:11434
+  #
+  # ── Self-hosted vLLM (OpenAI-compatible) example ──────────────────────────────
+  # - model_name: aisoc-investigation
+  #   litellm_params:
+  #     model: openai/Qwen2.5-32B-Instruct
+  #     api_base: http://vllm:8000/v1
+  #     api_key: os.environ/VLLM_API_KEY
+  #
+  # ── Anthropic example ─────────────────────────────────────────────────────────
+  # - model_name: aisoc-report
+  #   litellm_params:
+  #     model: anthropic/claude-3-5-sonnet-latest
+  #     api_key: os.environ/ANTHROPIC_API_KEY
+
+# ── Gateway behaviour ─────────────────────────────────────────────────────────
+litellm_settings:
+  # Silently drop provider-specific params a given backend doesn't support, so
+  # the same AiSOC request body works across OpenAI / Ollama / vLLM / Anthropic.
+  drop_params: true
+  # Per-request latency, tokens, cost, errors, retries and fallbacks are exported
+  # on GET /metrics for Prometheus (scraped as job `aisoc-litellm`). This is the
+  # centralized observability layer that complements the Investigation Ledger.
+  callbacks: ["prometheus"]
+  # Prometheus scrapes /metrics with no Authorization header; LiteLLM otherwise
+  # requires auth on that endpoint (and the auth path needs the DB layer). Allow
+  # unauthenticated scrapes so the `aisoc-litellm` job actually collects. The
+  # port is bound to 127.0.0.1 in compose, so /metrics isn't publicly exposed.
+  require_auth_for_metrics_endpoint: false
+  # Never wait forever on a stalled provider — fall through to AiSOC's own
+  # deterministic path instead of hanging the alert drawer.
+  request_timeout: 60
+
+general_settings:
+  # Clients (the AiSOC services) authenticate to the gateway with this key. Set
+  # it in the environment; AiSOC sends it as its OPENAI_API_KEY when
+  # OPENAI_BASE_URL points here. Never commit a real key.
+  master_key: os.environ/LITELLM_MASTER_KEY

scripts/audit_prometheus_targets.py

@@ -75,6 +75,11 @@
 # stack.
 THIRD_PARTY_HOSTS: set[str] = {
     "alertmanager",
+    # Issue #478 — LiteLLM gateway. A third-party image (berriai/litellm)
+    # that exports per-task LLM latency/tokens/cost/errors on /metrics via
+    # its prometheus callback. No source lives under services/, so the
+    # "does the service tree expose /metrics" check does not apply.
+    "litellm",
 }
 
 # Services we are intentionally NOT scraping even though they CAN