Reduce termchart diagram-generation latency + tokens (experiment)

[ivanmkc]

Goal

Reduce the end-to-end cost of generating a termchart diagram from a coding agent, on two axes:

• Latency — wall-clock from "agent decides to draw" to "diagram visible".
• Tokens — input+output(+cache) tokens to produce a correct diagram, incl. retries.

Measured across three runners (Claude Code, AGY, OpenCode) on VMs, baseline vs a combined-fixes build.

Plan

Full plan tracked in-repo: docs/plans/2026-06-07-latency-token-experimentation-plan.md (branch perf/reduce-latency).

Where the cost comes from (grounded)

• Tokens: plugin/skills/diagram-recipes/examples/ is ~298 KB; the two *-matrix.component.json (~89 KB) are pretty-printed.
• Latency: A* router cliff at 25 nodes (render.ts), --fit 10x re-render, begin = 3 POSTs, push vs patch (18–190x tokens/change), 2–3 server dagre passes, cold npm i -g.

Decisions

• One shared model held constant across all runners (fair comparison).
• First pass = pilot: {baseline, c1} only, then gate to the full T*/L* ablation.

P0 status (landed)

Harness in scripts/experiments/ + corpus_run.py --metrics-out; --dry-run produces a full report; 20 unit tests pass.

Next

• Validate runner headless invocation (AGY is the flagged risk)
• Stand up shared-model proxy + viewer on a VM
• Run pilot (3 runners x {baseline, c1} x 5 tasks x 5 reps)
• Implement C1 fixes (T1 minify matrix examples, T5 fix stale AGENTS.md exit code, L1 single-shot begin, ...)

[ivanmkc]

Pilot run #1 — local podman, Vertex Gemini 2.5 Flash

Apparatus is fully working: podman → headless runner → LiteLLM proxy (shared model) → Vertex Gemini → termchart → rendered diagram, with tokens/latency captured per cell. 24 cells (2 runners × 2 conditions × 3 tasks × 2 reps), 100% success.

Model fallback: Claude on Vertex isn't enabled in the project (404 in all regions), so the shared model is Gemini 2.5 Flash for now. Flipping to Claude is a one-line EXPERIMENT_MODEL change once Anthropic Model Garden is enabled.

Δ vs baseline (c1 = baseline + T1 minify-examples + T5 AGENTS exit-code)

metric	c1 vs baseline	significant?
median tokens	−1.0%	no (CIs overlap)
median latency	+62.8%	no (small-N noise, heavy tails)

This is expected and honest: T1 only saves tokens when an agent actually reads a large example, and T5 is a one-character correctness fix — neither materially affects these tasks. The pilot's value is proving the apparatus and establishing baselines.

Headline finding — runner choice dominates

condition	runner	median input tokens	median latency
baseline	claude-code	29,510	15.2s
baseline	opencode	8,522	3.5s
c1	claude-code	38,986	14.8s
c1	opencode	8,742	4.4s

OpenCode spends ~3–4× fewer input tokens than Claude Code for the same diagram tasks (smaller system prompt + far less repo exploration).

Runner status

• Claude Code — working (claude -p --permission-mode bypassPermissions, runs as non-root node)
• OpenCode — working (opencode run --model openai/shared-model, openai provider → proxy)
• AGY (Antigravity) — deferred: agy 1.0.6 exists and supports -p, but has no custom base-URL flag, so it can't share the proxy/model or be token-measured uniformly; also needs ANTIGRAVITY_API_KEY.

Fixes shipped as PRs (into master, not merged)

• fix(docs): correct AGENTS.md push/status exit code (4, not 3) #147 — T5: correct AGENTS.md push/status exit code (4, not 3)
• perf(skills): minify diagram example JSONs (~45% smaller) #148 — T1: minify diagram example JSONs (~45% smaller; 305KB→168KB)
• perf: latency/token experiment — plan + P0 harness #143 — c1 integration branch (draft)

Next

To get a real token delta, c1 needs higher-impact fixes on the always-loaded path: T2 (slim AGENTS.md / SKILL.md) and T3 (tighter routing so the agent reads exactly one detail file + one example). Those should move Claude Code's input tokens, which dominate.