bsp: make the compile server explicable — heap at boot, and telemetry that charts what it records

[oyvindberg]

The compile server is a long-lived JVM you never launched and cannot see. This makes it say what it is doing, and makes the dashboard show what it already records.

Grew well past its original one-line change — worth reading as three pieces.

1. The heap, at boot

Ensuring BSP server (mode=shared, heap=max 12g of 48g RAM, socket=…)

The cap was invisible until something went wrong, at which point it surfaced in an OOM message. It is worth stating up front because the default surprises: min(16g, max(4g, RAM/4)) clamps to its 4g floor on anything at or below 16GB, so a 16GB laptop and a 14GB CI runner are sized identically and neither gets "a quarter of RAM".

RAM	quarter	actual
14 GB (macos-15-intel runner)	3.5 g	4 g — floor
16 GB (windows runner, a laptop)	4.0 g	4 g — floor
48 GB	12 g	12 g

max, because -Xmx is a ceiling and ZGC with ZUncommitDelay=30 hands memory back; a bare size next to the machine's RAM reads like a reservation.

Verified against both paths: default, max-memory 6g, and max-memory-clear.

2. Telemetry the dashboard was throwing away

bleep server-metrics and the CI summariser read the same file and had drifted. Of twenty event types the server writes, the dashboard handled fourteen and dropped six via case _ => () — and the six were the ones people arrive with questions about: machine, admission_defer, compile_allocation, analysis_cache, workspace_state.

Five new charts, six new cards. Two worth calling out:

• Scheduling draws CPU-in-use against cores with running and queued on top, because saturated and starved look identical from outside — a slow build — and have opposite fixes.
• Deferred compiles are split by reason. stagger is the scheduler spreading first starts apart and says nothing about memory; heap_pressure is a compile actually waiting for heap. Only the second argues for a bigger -Xmx. On a real 18-core dlab run: 115 defers, every one a stagger, zero heap pressure — precisely the case that used to read as memory-starved under the old label.

Legacy heap_pressure_stall events are read too, and shown as unlabelled (pre-rename) rather than assigned a cause the data does not state. Not theoretical: the released M9 spawned a server mid-testing that wrote 42 of them.

server-metrics --file <path> opens any metrics.jsonl, so one downloaded from CI is read by the same dashboard as a local one — same file, same viewer. Verified on a real artifact of 353 events.

3. Forked test JVMs, which was recorded nowhere

The server recorded what it compiled; what it ran was invisible. Five events joined on pid: fork_start, fork_reused, fork_end, suite_scheduled, suite_finished.

bleep-core cannot call bleep-bsp, and a singleton it could reach is the global mutable state this codebase refuses — so JvmPoolListener inverts it: the pool announces, whoever builds it decides who listens, default no-op. fork_end carries killed_by separately from the exit summary because destroyForcibly sends SIGKILL and a fork bleep killed is otherwise indistinguishable from one the OS killed.

Charted as one lane per JVM — the pale bar is the fork's life, the segments on it are the suites that ran there. That answers "which suite was on the JVM that got killed", otherwise a hand-join across three event types.

What it immediately showed on dlab (73 suites, 500 tests):

66 forks started, 7 reused          ~10% reuse; 59 forks ran exactly one suite
fork lifetime median 31.3s, max 1728.9s
26 suites timed out at ~28.8 min, across four different pids

The reuse rate may be intentional — the pool destroys rather than caches under memory contention — but no metric expressed it before.

Docs

docs/guides/compile-servers.mdx gains the floor table, a telemetry section (where the files are, what the event types mean, stagger vs pressure), a CI recipe with tested jq one-liners, and an explicit note that metrics.jsonl records nothing about individual tests — bleep test --junit-report is where per-test timing lives.

Verification

Everything here was checked by running it and reading the output, not by a green compile. That caught: fork_end never firing (shutdown bypasses destroy), lifetime_ms: -1 (start instant read after the process died), outcome: "Success$", a default parameter added mid-list that rebound every positional argument, and — twice — a dispatch edit that silently matched nothing, leaving charts guarded off on real data.

Final state exercised end to end on a 42-project dlab build: all nine charts and eight cards render.

🤖 Generated with Claude Code