STREAMIND — Real-Time Meeting Intelligence Pipeline

Real-time AI pipeline that ingests a live audio stream, transcribes it incrementally, and emits structured LLM summaries (a short summary plus exactly three keywords) over rolling 300-second windows with minimal latency. The pipeline is built as composable Juturna nodes; audio is delivered over WebRTC through a Janus gateway.

Pipeline

audio (Opus RTP via Janus)
  → chunk (short overlapping windows)
  → incremental ASR (faster-whisper large-v3-turbo)
  → hallucination filter
  → novel-chunk extraction (dedup overlap)
  → 300 s window aggregation
  → summarize (vLLM, Qwen3.5-4B-AWQ, in-process)
  → store locally + POST

All inference is CUDA-native via vLLM, loaded in-process at warmup — no daemon, no external LLM service.

Output

Each window produces one JSON object with exactly these keys:

{
  "from": 0.0, 
  "to": 300.0, 
  "summary": "...", 
  "keywords": ["...", "...", "..."], 
  "proc_time": 1.03 
}

Sample outputs: submission/sample_outputs/.

Requirements

• NVIDIA GPU, CUDA 12.4, ≥ 24 GB VRAM
• Docker + Docker Compose with the NVIDIA Container Toolkit

Build

docker compose build

The build fetches and bakes the model weights (Qwen3.5-4B-AWQ summarizer + faster-whisper ASR) into the image. This is the only step that needs network access. The runtime container is fully offline (HF_HUB_OFFLINE=1).

Run

docker compose up

The pipeline listens for an Opus RTP stream on 0.0.0.0:8888 (forwarded by the Janus container). Window results are written to ./results/ and, when a target is configured, POSTed to it:

DESTINATION_ENDPOINT=https://your.endpoint/submit docker compose up

Feed a remote audio source

Audio reaches the pipeline only through the Janus gateway — never directly into the container. Janus runs the VideoRoom plugin: a WebRTC client joins room 1234 as a publisher and publishes an Opus audio track, then Janus rtp_forwards that audio as plain RTP to the pipeline's audio_rtp node (audio_pt 97 → udp/8888). Any WebRTC source — a browser, an upstream SFU, or the bundled reference publisher — feeds the pipeline this way; nothing on the pipeline side changes for a different source.

For a remote source, expose the gateway so the source can reach it:

• 8088/tcp — Janus HTTP signaling API (path /janus)
• 10000-10050/udp — WebRTC media (ICE) ports

Then have the remote source join VideoRoom room 1234 as a publisher and issue the rtp_forward. tools/send_audio.py is the reference client for that handshake and runs from any host — point --janus-url at the gateway and leave --pipeline-host as Janus's own (co-located) route to the pipeline container:

uv sync --extra client

# audio = any ffmpeg-readable input (local file path or a stream URL)
uv run python tools/send_audio.py <audio> \
  --janus-url http://<gateway-host>:8088/janus \
  --room 1234

Window results land in ./results/ and POST to DESTINATION_ENDPOINT (when set) exactly as for a co-located source.

Configuration

The complete pipeline graph is a single file — pipelines/config-submission.json: eight nodes, a fixed 300-second summary window, the locked Qwen3.5-4B-AWQ summarizer, and the result sink. The container launches it directly (no runtime assembly).

Approach

Challenge restatement

Real-time meeting intelligence from a live audio stream. Per-chunk score C_i = B_i + K_i + L_i (B: LLM judge, 5 Likert criteria incl. conciseness, max 25; K: keyword relevance, max 6; L: latency reward, 10·e^(−0.5·proc_time) gated by B_i ≥ 10). Audio-level score averages chunk scores and adds a flat +4 Janus bonus.

Pipeline architecture

Eight sequential Juturna nodes, wired in pipelines/config-submission.json:

1. audio_rtp — receives mono Opus/48k RTP on 0.0.0.0:8888. Janus gateway forwards audio into this port. Config: channels: 1, encoding_clock_chan: "opus/48000/1".
2. audio_chunker — splits the live stream into 5 s overlapping chunks (1 s overlap) so the ASR sees audio incrementally, not as one blob.
3. transcriber_whisper — faster-whisper large-v3-turbo, device: auto (GPU when available), compute_type: float16. OpenAI's distilled large-v3 (~809M params) — near large-v3 WER at half the size. Loads ~2 GB on GPU alongside the summarizer; the 4B summarizer plus ASR plus KV cache fits comfortably in 24 GB. Linked against CTranslate2 with PyTorch-bundled cuBLAS via an LD_LIBRARY_PATH export in the container entry point so the stack stays self-contained. language: "en" set explicitly (multilingual model, no .en suffix).
4. hallucination_filter — post-processes ASR output to drop known hallucinatory sentences before aggregation.
5. novel_extractor — deduplicates the overlapping ASR output, keeps only the new content each chunk contributes.
6. window_aggregator — concatenates novel transcript segments into a rolling 300-second context window. Emits one message per completed window.
7. summarizer_vllm — native CUDA via vLLM in-process. Loads the model from ./models/<name>/ at warmup, generates {summary, keywords: [3]} JSON per window. No daemon, no middleware. Refuses to start if the model directory is missing.
8. result_transmitter — maps internal keys to the challenge contract (from, to, summary, keywords, proc_time), writes the result locally, and POSTs to destination_endpoint when configured.

Key design decisions

• Janus WebRTC bonus. Audio enters through a Janus gateway in both local dev and the evaluation environment — no environment-specific code paths. This earns the flat +4 audio-level bonus at zero architectural cost.
• ASR model — faster-whisper large-v3-turbo. OpenAI's distilled large-v3 (~809M params, ~2 GB float16). Near-identical WER to large-v3 at half the compute. Runs on GPU (float16) co-resident with vLLM — the 4B summarizer + KV cache + ASR fit in 24 GB with margin. GPU placement removes the ~2 s/chunk CPU bottleneck the int8/CPU variant had, lifting WER (better transcripts → higher B_i) without costing latency.
• Summarizer backend — vLLM in-process. Chosen over Ollama (server), MLX (Apple-only), and raw transformers (slower):
  • No daemon. Inference is a Python library call. The submission Docker has one process to launch; nothing to start, nothing to health-check, nothing to crash independently of the pipeline.
  • Paged attention + batching. vLLM's continuous batching and paged KV cache keep proc_time low even at larger model sizes, which is exactly the lever for L_i = 10·e^(−0.5·proc_time).
  • Single-GPU fit. A 8–9 B fp16 model fits inside the 32 GB RTX Pro 4500 (Blackwell) — and even the tighter 24 GB dev RTX 4090 — with headroom for KV cache; vLLM's gpu_memory_utilization knob controls the reservation explicitly.
  • SGLang considered, not adopted. A May-2026 engine survey found SGLang (sgl.Engine, RadixAttention) ~29% higher throughput / lower TTFT than vLLM as a near-drop-in replacement, and TensorRT-LLM faster still at the cost of per-model compilation. vLLM is kept for its widest model support and zero compilation; SGLang remains the primary latency lever to revisit if L_i becomes the binding constraint.
• Model weights ship with the submission. tools/fetch_models.sh populates ./models/<name>/ during development; the same script is invoked at docker build time so the resulting image carries weights. The pipeline aborts at warmup if the directory is missing — failures are loud, never silent.
• Single summarizer profile. One Juturna node, one locked summarizer baked into the image. No MLX/Ollama variants to maintain or drift between.
• Judge runs offline, sequentially. Quality is scored after the pipeline exits and the summarizer is unloaded: the judge model is loaded with vLLM, scores every window in results/.../, and writes its verdicts alongside. This avoids VRAM co-residency (an 8 B summarizer + 7 B judge cannot share 24 GB at fp16) and reflects reality: the held-out judge is the only one that matters at submission time.
• Prompt engineering for conciseness + specificity. Prompts demand 1–2 sentences, facts present in the transcript only, and specific noun phrases for keywords. This doubles up: higher B_i conciseness score and lower proc_time (smaller max_tokens suffices), which boosts L_i.
• Keyword validation. ensure_three_keywords strips banned generic terms (general, discussion, meeting, …) that would be scored as irrelevant, and backfills missing slots by extracting proper-noun / content words from the transcript. Turns the previous −6 worst case on LLM failure into a +0 floor: even a broken summarizer no longer tanks K_i.
• Robust JSON extraction. The summarizer strips ChatML special tokens (<|im_end|>, <|endoftext|>) and extracts the outermost balanced JSON object from the raw output — tolerates trailing stop tokens or commentary the model may emit after the JSON.
• Hallucination filter. Inline post-processing guards against invented names, numbers, or facts — a known failure mode for small LLMs on short transcripts. Protects B_i factual-consistency.
• Quality gates latency. The scoring rule zeroes L_i when B_i < 10. The pipeline is tuned so each window clears that floor before optimizing latency further. Bigger models / longer prompts are preferred to faster but lower-quality combinations.
• Extractive fallback (never empty summary). The summarizer node has two guarded paths into a deterministic extractive summary (first sentence(s) of the transcript, capped at 300 chars) under _summarizer_common/extractive.py: (a) transcripts shorter than min_transcript_chars=80 skip the LLM entirely; (b) LLM failures (parse error, empty summary, exception) fall through to the same helper. Output is always non-empty, faithful (verbatim), and effectively zero proc_time — preserves the L bonus on edge windows.
• End-of-stream flush watchdog. The window aggregator runs an inactivity watchdog (default 30 s). When no chunks arrive for flush_timeout seconds, the partial window is flushed with its actual chunk_end (not window_start + duration). Captures the tail of finite audio that would otherwise be dropped.

Measured performance

Submission summarizer Qwen3.5-4B-AWQ (AWQ-int4), 300 s windows. Scored offline by the cross-family 3-judge consensus panel (mistral-small-24b-awq, phi-4-awq, gemma3-27b-it-int4-awq); each value is the per-window mean over the three judges, averaged over windows and over repeated runs per fixture. Janus +4 applied at the audio level.

Fixture (300 s windows)	B (≤25)	K (≤6)	L (≤10)	C (avg)	Janus	Final
rev16 ep27 (What We Own)	24.5	6.0	6.57	37.02	+4	41.02
rev16 ep10 (Own Lane)	23.9	5.3	7.06	36.32	+4	40.32
ietf Media-Over-QUIC	23.8	5.8	6.73	36.24	+4	40.24
ietf Computing-Aware-Tr	23.6	5.9	6.79	36.18	+4	40.18
rev16 ep11 (Podcast Tips)	23.2	5.1	6.79	35.07	+4	39.07
mean ± sd (5 fixtures)	23.8	5.6	6.79	36.2	+4	40.2 ± 0.6

L ≈ 6.8 corresponds to ~0.77 s proc_time per window; every window clears the B_i ≥ 10 gate so the latency reward always applies. Judge agreement is high (per-window C stdev across the panel ≈ 0.3–0.5), so the consensus is corroborated rather than driven by a single model.

Submission packaging

• Dockerfile targets nvidia/cuda:12.4.1-runtime-ubuntu22.04, installs Python 3.12 + uv, runs uv sync --no-install-project (vLLM and huggingface-hub are base dependencies; no extras enter the image). The build then invokes ./tools/fetch_models.sh cyankiwi/Qwen3.5-4B-AWQ-BF16-INT4 qwen3.5-4b-awq to bake the summarizer weights, and huggingface_hub.snapshot_download(...) to bake faster-whisper-large-v3-turbo. The resulting image carries every model byte; zero network dependency at runtime.
• docker-compose.yml runs the Janus service and the pipeline service. Janus forwards RTP to the pipeline container on UDP/8888.
• Entry point docker/entrypoint-pipeline.sh launches the single pre-assembled graph pipelines/config-submission.json directly (no runtime assembly — the submission is locked to one summarizer). The launch uses juturna's --auto flag so the container starts without interactive input.
• destination_endpoint is left empty in the committed pipelines/config-submission.json — the challenge spec publishes no fixed receiving URL, and the graded artifact is the local progressive-summary JSON. Every window is always written to ./results/<model>/<window_s>/window_N.json (exposed via the host volume mount). To enable POSTing, the evaluator sets the DESTINATION_ENDPOINT environment variable (DESTINATION_ENDPOINT=... docker compose up); result_transmitter reads it as a fallback when the config field is empty and POSTs every window (httpx, configurable timeout) in addition to the local write — no file edit required.
• .dockerignore keeps the image lean (~20 GB content vs ~150 GB with default context) by excluding models/ (baked separately), datasets/, results/, and .venv/.