python-vs-rust-ai-agents-runtime-matters

A head-to-head runtime & NFR comparison of the same multi-agent supervisor built twice:

• Python — LangGraph + FastAPI
• Rust — Axum + graph-flow + rig

Everything else is shared and identical: the DAG, the config contract, PostgreSQL, MinIO, the (mock or real) LLM, and the DeepWiki MCP server. Only the agent runtime differs, so latency, throughput, memory, CPU, and cold-start deltas are attributable to the runtime — not to different prompts, models, or infrastructure.

What it measures

Non-functional requirements, in priority order: latency > throughput > memory > CPU > cold-start.

The workload is a supervisor DAG run by both runtimes. The supervisor logically dispatches two workers, but both runtimes execute the graph as a single sequential chain (worker A then worker B — no fan-out concurrency), so the per-request work is identical and the runtime is the only variable:

flowchart LR
    pre["pre-process<br/>(CPU)"] --> sup["supervisor<br/>(LLM route)"]
    sup --> a["worker A: research<br/>(DeepWiki MCP)"]
    a --> b["worker B: analysis<br/>(LLM)"]
    b --> agg["aggregate<br/>(LLM)"]
    agg --> post["post-process<br/>(CPU)"]
    post --> persist["persist<br/>(Postgres + MinIO)"]

Loading

Per request: 3 LLM calls (supervisor, worker B, aggregate) + 1 DeepWiki MCP tool call (worker A).

Two run modes (set in config/runtime.yaml):

• isolated (primary) — deterministic mock LLM, deterministic research stub, no network/cost. This is what the NFR deltas are measured in.
• realistic — real Anthropic/Azure provider + real DeepWiki MCP; validates end-to-end.

Stated biases (by design)

The comparison deliberately favors idiomatic code over a forced apples-to-apples port:

• Deterministic pre/post-processing is pure-language (no numpy) — the naive-developer default, which favors Rust on the CPU-bound similarity matrix.
• In-node concurrency is idiomatic: Rust uses rayon/threads; Python stays single-threaded (GIL).

Both are documented so readers can weigh them. A numpy variant would narrow the CPU gap.

Repository layout

apps/python/        Python agent (LangGraph + FastAPI)      → service python-agent:8001
apps/rust/          Rust agent (Axum + graph-flow + rig)    → service rust-agent:8002
config/             Shared config + JSON schemas (agent.yaml, runtime.yaml)
infra/              docker compose stack (postgres, minio, mock-llm, prometheus, cadvisor, grafana)
infra/mock-llm/     Deterministic stub LLM (hashes user-message text; async, non-blocking)
bench/seed/         Deterministic corpus loader (MinIO + Postgres)
bench/equivalence/  Behavioral-equivalence rig (per-field diff, isolated mode)
bench/k6/           k6 latency + throughput scenarios
bench/coldstart/    Cold-start measurement
bench/report/       Report generator + docker-stats resource sampler
docs/RESULTS.md     Generated benchmark results (not committed)
docs/assets/        Generated charts (dag/latency/sweep PNGs — not committed)

Prerequisites

• Native Docker Engine (docker-ce) on the Linux host (Ubuntu) + docker compose v2.
• A few GB of free RAM.
• No local Python/Rust/k6 toolchain required — everything runs in containers.

The CPU and memory numbers come from each agent's own /metrics (process_cpu_seconds_total / process_resident_memory_bytes, read from /proc/self — prometheus_client's ProcessCollector on Python, the metrics-process crate on Rust). This is cgroup-independent and identical across runtimes, so it works on any engine (native docker-ce, Docker Desktop, podman) with no cAdvisor, cgroup-namespace, or storage-driver setup. cAdvisor is still in the stack for container-level infra metrics, but the report and the Grafana CPU/RSS panels no longer depend on it (see Known environment notes for the cAdvisor-specific gotchas if you want its per-container series).

Recreate the benchmark locally

One command

cp .env.example .env
make benchmark

make benchmark runs the whole pipeline end-to-end:

1. up — start shared infra (waits for real health)
2. seed — load the deterministic corpus into MinIO + Postgres
3. build-python, build-rust — build both agent images
4. bring up python-agent + rust-agent as compose services
5. coldstart — fresh-container → first /healthz 200, per runtime
6. bench — equivalence gate, then k6 latency + throughput for both runtimes, then a docker stats resource sample
7. report — generate docs/RESULTS.md

When it finishes:

• docs/RESULTS.md — side-by-side latency / per-step p95 / RSS+CPU / cold-start tables
• Grafana at http://localhost:3000 (admin/admin), dashboard "Python vs Rust Agent Runtime" — 4 live panels

Step by step

make up                              # shared infra
make seed                            # deterministic corpus
make build-python && make build-rust # both agent images
docker compose --env-file .env -f infra/compose.yml up -d python-agent rust-agent

make equiv                           # equivalence gate (must pass before publishing)
make bench-latency RUNTIME=python    # constant-arrival-rate latency (RATE=, DURATION= optional)
make bench-latency RUNTIME=rust
make bench-throughput RUNTIME=python # ramping-arrival-rate throughput
make bench-throughput RUNTIME=rust
make coldstart                       # cold start both runtimes
make resources                       # per-container CPU%/RSS via docker stats
make report                          # write docs/RESULTS.md

The equivalence gate is the publication contract: a comparison is only meaningful if both runtimes produce the same result for the same input (isolated mode) — identical ranked_findings chunk-id/source sequence, scores within tolerance, and equal summaries. If make equiv fails, it names the divergent field; fix the responsible runtime before trusting any numbers.

Tunable parameters

Knobs you can tweak before a run. Env vars are read by the make targets / bench scripts; the two YAML files change what the agents compute (edit those → re-run make equiv).

Load & sweep (env vars):

Var	Default	Effect
SWEEP_RATES	2 5 10 25 50 100 150	Space-separated offered-RPS ladder for make sweep. Default tops at 150 (clean, pre-saturation band); extend (e.g. "… 200 300 500") to probe the saturation cliff.
SWEEP_WARMUP	5s	Discarded warm-up per sweep step.
SWEEP_MEASURE_S	20	Measured steady-state window per sweep step (seconds).
RATE	20	make bench-latency constant arrival rate (req/s).
DURATION	30s	make bench-latency duration.
MAX_VUS	2000	k6 max virtual users (both latency.js + throughput.js). Must exceed rate × p95_seconds or the open-model executor drops iterations and "achieved RPS" understates the agent.
PREALLOC_VUS	200	k6 VUs pre-allocated before the run (avoids mid-test allocation stalls).

CPU isolation (env vars → docker cpuset):

Var	Default	Pins
AGENT_CPUSET	0-3	Both agents (only one is loaded at a time — same budget = fair).
K6_CPUSET	4-5	k6 load generator (kept off the agent's cores).
DEPS_CPUSET	6-7	Postgres, MinIO, mock-LLM, Prometheus, Grafana, cAdvisor.

Assumes an 8-core host (nproc). On fewer cores, set these to non-overlapping ranges that fit, or unset them (edit infra/compose.yml) to disable pinning. Widen DEPS_CPUSET if the shared deps become the bottleneck at high RPS.

Workload & deployment (YAML — changes computation → re-run make equiv):

Location	Key	Effect
.env	MOCK_LLM_LATENCY_MS (200)	Per-call sleep of the mock LLM (each /invoke makes 3 calls).
config/agent.yaml	workload.num_docs / chunk_chars / vector_dim	Corpus size + CPU cost of the similarity matrix (bump these to make the workload more CPU-bound).
config/agent.yaml	thresholds.dedup_similarity / top_k	Ranking/dedup behavior.
config/runtime.yaml	mode (isolated|realistic) / provider	Deterministic mock vs real provider. Sweep interpretation only holds in isolated.

Service ports

Service	URL	Notes
python-agent	http://localhost:8001	/invoke, /healthz, /metrics
rust-agent	http://localhost:8002	/invoke, /healthz, /metrics
mock-llm	http://localhost:8090	deterministic stub LLM
Prometheus	http://localhost:9090	scrapes both agents + cAdvisor
Grafana	http://localhost:3000	admin/admin
cAdvisor	http://localhost:8080	container metrics
MinIO	http://localhost:9010 / 9011	S3 API / console (offset ports)
Postgres	localhost:5432	agent/agent, db agentbench

Fire a single request

make invoke-python     # POST /invoke at the running python-agent
make invoke-rust       # POST /invoke at the running rust-agent

Tests

make test-python       # unit tests (no infra needed)
make test-rust         # unit tests (host cargo, no infra needed)
make lint-python       # byte-compile check
make lint-rust         # clippy -D warnings

Known environment notes

• Container tooling is native docker-ce on the Ubuntu host. Raw compose calls pass --env-file .env.
• The agent apps run as compose services (not make dev-*) so they're scraped/measured like the benchmark. make dev-python / make dev-rust run an agent on the host against localhost infra ports for interactive development.
• CPU/RSS are self-reported by the agents (process_cpu_seconds_total / process_resident_memory_bytes from /proc/self). RESULTS.md reports the process_cpu_seconds_total counter delta over the run (baseline captured post-coldstart by make cpu-baseline) and peak process_resident_memory_bytes; the Grafana "process CPU/RSS by runtime" panels use the same series. No cAdvisor / cgroup / storage-driver setup is needed for these — they work on any engine.
• cAdvisor is optional (container-level infra view only). If you do want its per-container series to resolve on native docker-ce, two things are required — this history is why the project left macOS/podman (podman-machine only exported the root cgroup) and why plain Docker Desktop doesn't help either (containers live in a separate VM):
  1. Host cgroup namespace — cgroup v2 defaults containers to cgroupns=private, so a private-ns cAdvisor sees only its own cgroup subtree. infra/compose.yml sets cgroup: host.
  2. Classic overlay2 graphdriver — Docker 29 defaults to the containerd snapshotter (docker info → Driver: overlayfs), whose layout cAdvisor v0.49.1 can't read (it drops each container → no name= label). Revert via /etc/docker/daemon.json { "features": { "containerd-snapshotter": false } }, sudo systemctl restart docker, rebuild.
• k6 runs as --user 0 so it can write summary JSON; a k6 threshold breach (exit 99) is recorded as data and does not fail the pipeline (the equivalence rig is the gate).
• Default credentials and anonymous Grafana are for local use only.

Realistic mode

Set mode: realistic in config/runtime.yaml, provide real provider keys via env, rebuild the agents, and re-run. The equivalence rig is isolated-mode only (real LLM output is non-deterministic).