bsp: bound and unify the compile server's scheduling — zinc 2, structural sharing, one knob - #622
Merged
Conversation
…rence Adds an "Idle self-shutdown" section to the compile-servers guide covering the new `bleep config compile-server idle-timeout` (default 60 min, 0 to disable): how the fully-idle clock resets on connect and while any client is connected, and how it differs from the per-connection read timeout. Regenerating the CLI reference picks up idle-timeout / idle-timeout-clear and also syncs remote-cache help text that had drifted from the committed pages.
A compile server serving many worktrees had no ceiling on either of the
two things that fill its heap, and OOM'd at -Xmx12g. Metrics from one
such daemon (11 worktrees, 45 minutes):
- 14 concurrent connections, 18 concurrent compiles, heap pegged at
12288/12288MB. `maxParallelism` bounds a single compile RUN, so each
connected client was independently entitled to every core.
- the post-GC floor climbed monotonically 3.5GB -> 8.2GB as workspaces
were served, because BuildCache retained every resolved build for the
daemon's lifetime. Later OOMs fired at a concurrency of three: by then
the floor, not the concurrency, was the problem.
Four changes:
MachineResources gains a per-kind ceiling on Compile. It is already the
daemon-wide accounting authority, so queueing, cancellation-safety,
oldest-first fairness and snapshot observability come for free. Cores are
the wrong proxy for a compile: it reserves zero fork-memory (that budget
is for forked processes) and allocates into the shared server heap, so
one core per compile let 18 cores drive 18 compiles into one 12g heap.
Forks are deliberately unaffected — they already reserve measured RSS
against an OS-retuned budget.
BuildCache is bounded (default 4) and evicts the least recently used idle
entry. A workspace with operations in flight is never evicted, so the
cache yields to live work rather than stalling it. The policy is a pure
function so it can be tested without resolving a build.
HeapPressureGate now reads the daemon-wide compile count instead of a
per-connection tally. MultiWorkspaceBspServer is constructed per client,
so a dozen clients each running one compile all concluded they were the
only compile and skipped the stagger entirely.
Telemetry, aimed at what had to be reconstructed by hand to diagnose the
above: heap_live_mb (post-GC live set — retention vs churn, no more
deriving a floor from windowed minima), in_flight on oom_pressure (which
projects, not just how many), allocated_mb on compile_end (per-compile
allocation, attributed only when the same thread closes the reading), and
new `machine` and `workspace_state` events sampled every 15s whether
contended or not.
Also fixes the compile summary line, which interpolated no count at all
and so always read "Projects: compiled, N failed, N skipped".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…ere it works Follow-up to the concurrency and build-cache bounds. With those in place a daemon still sat at 7.2GB live set with two compiles running, so the floor was measured directly rather than inferred: a class histogram of the live server showed ~4.5GB in xsbti.api.* — 31.7M NameHash (1.0GB), 31.1M Id (498MB), 7.4M PathComponent arrays (428MB), 839K retained AnalyzedClass — and a forced full GC moved the live set by 68MB. That is ZincBridge.analysisCache: unbounded, daemon-lifetime, keyed by analysis file path so it accumulates across every workspace and project. It was softly referenced on the theory that GC reclaims it under pressure. It does not. Soft references are not cleared until the collector is nearly out of room, and until then they are indistinguishable from live data — they occupy the heap, count as live set, and crowd out the compiles the cache exists to speed up. This also explains the earlier "2.2GB per cached workspace" fit: it was analyses accumulating as projects compiled, not BuildCache scaling with workspaces, so bounding workspace count could never have reached it. Bounded explicitly instead: entries idle past 120s are dropped, and beyond a 256MB budget the least recently used go. The budget is denominated in the on-disk bytes of the files read, calibrated against a real build where 113MB of analysis files (166 files, one workspace) inflated to ~4.5GB live — so it is worth roughly 1.5GB of heap. Swept from the daemon's existing reporter rather than on every write, and recorded as an `analysis_cache` event so the largest retainer in the heap is a time series. Eviction is safe at any moment: read-through cache, a miss re-reads from disk, and a compile in flight holds its own strong references to the analyses it loaded. Also moves compile-allocation measurement into ZincBridge's IO.interruptible block. getThreadAllocatedBytes is per thread, and that block is the one place a compile is pinned to one; measured at the BSP layer the fiber had usually migrated between start and end, and attribution succeeded for 502 of 34153 compiles — all of them the trivial ones that never yielded. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Analyses were held in a global map keyed by analysis file path, with no
owner. Two things follow from having no owner, and both were wrong.
Evicting a workspace from BuildCache freed its resolved `Started` —
hundreds of MB — and left that same workspace's analyses resident, which
is the multi-GB part. We freed the small half and kept the large one.
And nothing could say which build was holding the heap. Establishing that
took a class histogram of a live daemon plus a linear fit against workspace
count, and the fit was misleading: the "2.2GB per cached workspace" it
suggested was analyses accumulating as projects compiled, not BuildCache
scaling with workspaces.
The keys already lined up. Analysis files live at
`.bleep/projects/<project>/builds/<variant>/.zinc/analysis.zip` — inside a
workspace, partitioned by variant — which is exactly BuildCache's key. So
`model.WorkspaceKey` is introduced as that shared identity, derived once in
`BuildPaths.workspaceKey` so the two caches cannot disagree about what "the
same build" means, and the cache moves out of ZincBridge's globals into an
`AnalysisCache` handed through structurally like `buildCache` and
`kspMutexes` already are.
Ownership buys three things:
- a cascade, in one direction. Dropping a build drops its analyses,
because if nothing wants the build nothing wants its analyses. NOT the
reverse: re-resolving a build costs seconds of coursier work while
re-reading an analysis costs about one disk read, so analyses are shed
eagerly on their own schedule and never disturb the build they belong
to.
- isolation. The budget is per workspace, so a monorepo churning through
166 analyses can no longer evict a small workspace's whole working set.
- attribution. `analysis_cache` now reports entries and bytes per
workspace, so "who is holding the heap" is a field to read rather than
an afternoon with jcmd.
`AnalysisCache.Ref` binds the cache to one workspace at the point where the
workspace is known, so no call site inside a compile can pass the wrong key.
Compilations that belong to no workspace — standalone single-file compiles,
DAGs built outside a BSP session — get `AnalysisCache.standalone`, an
instance that dies with the call rather than accumulating in a bucket
nothing owns and nothing sweeps.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…lyzedClass
Three changes that only make sense together.
**zinc 1.12.0 -> 2.0.4**, as `zinc_3` so the `for3Use213` shim goes away.
The upgrade needed two things:
- compiler-interface 2.0.4 evicts the 1.10.7 scala3-compiler asks for,
and its POM declares early-semver so bleep fails the build. The
compatibility argument is checkable and holds: diffing the jars shows
2.0.4 is purely additive over 1.10.7 — 150 classes, zero public or
protected members removed, two added. Declared once as a
libraryVersionSchemes entry on bleep-bsp, which transitive consumers
inherit, rather than switching off eviction checking per project.
- `xsbti.VirtualFile` now extends `HashedVirtualFileRef`, adding
contentHashStr() and sizeBytes(). Additive is safe for callers, not
for implementors: PlainVirtualFile had to grow both. They follow
zinc's own MappedVirtualFile — Files.size, and SHA-256 of the content
for contentHashStr, which is a different hash from the farmhash in
contentHash and is compared against strings zinc produced itself.
**ConsistentFileAnalysisStore(reproducible = true)** replaces
FileAnalysisStore at all three sites. Identical inputs now produce
byte-identical analysis files, normalising the timestamps that otherwise
make two worktrees' analyses differ while describing the same code.
Parallelism is pinned to 4 rather than zinc's `availableProcessors()`
default, which assumes one build at a time; this daemon runs up to
maxConcurrentCompiles at once, each loading several analyses.
**AnalyzedClass is interned** at deserialization, inside AnalysisCache so
no caller can forget. Weak values plus a ReferenceQueue expunge, so the
interner is an index and not a retainer: an entry lives exactly as long
as some loaded analysis points at it.
Measured before building it, on real analyses: 2.06x sharing within one
dlab workspace, 2.96x within bleep's own, and — the control — a second
copy of an identical workspace adds exactly zero distinct instances.
Divergent branches share little; a freshly forked worktree shares
everything. Against the 4.5GB of xsbti.api.* a live daemon was measured
holding, that is 2.2-3.0GB back. The key omits compilationTimestamp:
zinc 2 no longer reads it, and including it cost 156,691 extra retained
instances across six worktrees against 94 within one.
**The standalone AnalysisCache eviction policy is deleted** — budget,
idle timeout, sweep, and the pure LRU policy with its tests. It existed
because holding one workspace's analyses looked expensive; interning
removes that premise. Analyses now live exactly as long as their
workspace's BuildCache entry, which is the one retention rule.
Migration: the analysis format changes, so every local analysis.zip and
every remote-cache entry is invalid. One full recompile per worktree, and
a window where old and new clients cannot share cache entries.
899 tests pass (627 bleep-bsp-tests + 272 bleep-tests).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The resource surface had grown to fifteen subcommands, and an experienced user hitting exactly the problem they are for could not find the one he needed (datoria-dev/dlab#301: "bleep test exposes no concurrency cap, only --jvm-opt"). He was right — `parallelism` existed in the config model with no CLI at all, while `max-concurrent-compiles`, added days ago, was CLI-only and undocumented. Two vocabularies for "how much of this machine may bleep use", and the documented one was unreachable. Now there is one. `parallelism` gets the CLI it never had, and the daemon-wide compile ceiling derives from it as half — a compile's scarce resource is the server's shared heap rather than a core, which is why it is half and not equal, but it is no longer a separate number to set. `maxConcurrentCompiles` is gone from the model. Demoted to internal, keeping the config fields but dropping CLI and docs: `heap-pressure-threshold` and `max-cached-workspaces`. Both are tuning constants for mechanisms a user cannot observe, and after interning nobody has a basis on which to pick a cached-workspace count. Also: the daemon no longer inherits BLEEP_* from whichever client spawned it. `pb.environment()` starts as a copy of the spawning process, so a `BLEEP_FOO=… bleep compile` became permanent state on a server that then served every other client and every fork it started. Demonstrated accidentally while writing this: one `BLEEP_PARALLELISM=3 bleep compile` pinned that value into the shared daemon, and unrelated test forks inherited it minutes later — three tests in this very commit failed on it. Non-bleep variables (PATH, HOME, proxies) are left alone; they describe the machine, which the daemon genuinely shares. That is also why no resource knob is settable from the environment. The compile server reads them once at startup and passes them to every fork, and its heap is part of the key deciding WHICH daemon you get — so a per-client override would either configure the machine for everyone or silently split the daemon population in two. A CI job is one machine: configure it in a step before building, which the docs now show. Docs: `resource-management.mdx` gains a "Small machines and CI" section with the arithmetic worked through on a 16GB runner — the fork budget is RAM minus the server's footprint minus a quarter of RAM, so an 8g server on 16GB leaves 2GB and every fork queues. That table is what dlab#301 needed. The page also now says which release the machine-wide accounting arrived in, rather than describing unshipped behaviour as present. 905 tests pass (628 bleep-bsp-tests + 277 bleep-tests). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one field report (a "crashed twice" with no usable evidence). **Make it fail.** A server crash left the summary reading clean — "52 compiled, 0 failed" — with `Build failed:` printed underneath it, because `anyFailure` only ever considered compile, link and sourcegen failures. Both lines were true (52 projects really did compile; the transport really did die) but the reader had to reconcile them. BuildSummary now carries `serverCrashed`, which marks the header FAILED, adds a line saying the counts are what finished before the server died rather than the whole build, and makes `toEither` a Left. Set from ReactiveBsp's double-crash path. **Fix the rotation.** `cleanup` deleted `output`, and the client's crash handler calls it before restarting — so a crash destroyed its own evidence. Only the OOM marker survived, grepped out beforehand, which means a crash that was NOT an OutOfMemoryError left nothing at all. That is exactly the reported case. `cleanup` now removes runtime state (lock, pid, socket) and keeps the log. Rotation also kept one generation, which is one too few: the client retries a crashed server once, so a systematic crash rotates twice in seconds and the second rotation overwrote the first crash's log with the second boot's short, uninformative one. Two generations are kept, so the original crash survives the retry that was supposed to diagnose it. **Bound both tails.** Retained logs are held to 32MB per socket directory, oldest dropped first — the live file cannot be bounded from outside, since a running server holds it open through an OS redirect, but history is what accumulates. And socket directories whose daemon is dead and which nothing has touched for two weeks are removed at spawn: the JVM key includes the bleep version, so every snapshot install mints a fresh directory and abandons the previous one, each holding logs and up to 200MB of metrics. Nothing was pruning them. 910 tests pass (633 bleep-bsp-tests + 277 bleep-tests). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
oyvindberg
changed the title
The governor briefly held compiles to half of `parallelism`, on the reasoning that a compile's scarce resource is the server's shared heap rather than a core, so cores are the wrong proxy for what it costs. That reasoning was about heap, but a count is not a heap bound — and as a count it is a static partition inside a governor that is work-conserving everywhere else. `grantEligible` hands out whatever currently fits and reclaims it on release, so compiles already expand into an idle machine and contract when forks arrive. Reserving half for test forks that may not exist just leaves a compile-only run at half throughput. Its premise had also dissolved. The failure that motivated it — 18 compiles pegging a 12GB heap — is better explained by two things already fixed on this branch: HeapPressureGate read a PER-CONNECTION compile count, so a dozen clients each running one compile all concluded they were alone and skipped staggering entirely; and two thirds of that heap was retained analyses, now interned and bounded. The ceiling was sized against a heap mostly holding things it should not have been. So heap pressure is left to the gate, which watches the live heap rather than guessing from a count. If a real bound on compile heap is wanted later, the honest form is a third governor axis charging compiles their MEASURED heap the way forks are charged measured RSS — `compile_allocation` now provides the data to size it. Removing it also deletes an inconsistency it introduced: the DAG width was re-read from config per request while the ceiling was frozen at daemon startup, so half of one knob updated on a different schedule from the other. 907 tests pass (630 bleep-bsp-tests + 277 bleep-tests). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`parallelism` bounded each compile run's DAG width and each JVM pool, but the machine-wide governor was constructed with `availableProcessors()` directly. So on a 16-core box, `parallelism = 2` gave two tasks per run and two pooled JVMs per run — while the governor kept admitting up to sixteen CPU reservations across connected clients. A user who asked for two got two per client and as many as clients happened to exist. The governor's CPU axis is now `effectiveParallelism`. Cores remain what that DEFAULTS to (one per core, or `parallelismRatio` of them), which is the only place a raw core count belongs: as the default value of the knob, not as a second limit sitting behind it. The startup line reports both, since they are now different things: "Machine: 18 cores, ... -> parallelism 18, initial fork-memory budget ...". Also fixes a stale comment in ZincBridge still referring to the compile ceiling removed in the previous commit. 908 tests pass (630 bleep-bsp-tests + 278 bleep-tests). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The standalone, non-BSP test runner. Nothing in the repo references it: not the CLI, not the BSP server, not the tests, not the docs. Every test run goes through MultiWorkspaceBspServer and the DAG. It was not harmless dead weight either. Three commits on this branch carried it along — threading the compile ceiling through it, removing that again, then re-sizing its governor from parallelism — all edits to a file with no callers, made only because a compiler error pointed at it. The pieces it shared with the live path (JvmPool, TestDiscovery, DiscoveredSuite, TestJvm) all keep other users and are untouched. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…erything The scheduler dispatched blind and each task reserved inside its own body. Two consequences, both fixed by moving admission to the one loop that already knows what should run next. **Priority stopped mattering under contention.** The loop sorts ready tasks by dependents count — most-unblocking first — then dispatched up to `maxParallelism` of them. Everything past that point queued FIFO inside the governor, so the moment anything had to wait, the ordering the sort computed was decoration. Now the loop admits with `tryReserve` in priority order: fits, or stays ready and gets another chance on the next wakeup — which fires when a task completes, precisely when resources free. `maxParallelism` disappears from `execute` entirely; how much runs at once is what the machine can currently afford, and `parallelism` sizes the machine. **Link forks were unaccounted.** Scala.js, Scala Native, Kotlin/JS and Kotlin/Native each fork a linker (and node, for JS), reserving nothing — counted as one task while being a whole JVM plus a toolchain. They now declare a cost like every other fork, which is the accounting hole the governor was built to close and which these three sites had escaped. Cost is `costOf(task, forkHeaps)`, a function rather than a field: it is a property of (what kind of work this is, how this machine is configured), not of a task's identity — and putting it on the case classes made two otherwise identical tasks unequal because a config value differed. Forks are charged the same footprint the old reservations charged, so the accounting does not silently get lighter. One hazard the design has to answer: with nothing running, nothing will ever complete to wake the loop, so a task that does not fit would hang rather than wait. When the machine is idle the first candidate is admitted by blocking reservation instead, which clamps to the machine's totals — a task larger than the whole machine waits for the whole machine, then runs. 908 tests pass (630 bleep-bsp-tests + 278 bleep-tests). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
oyvindberg
changed the title
The resource page still described concurrency as a per-run number and the machine budget as a thing that merely explains why fewer forks run. Both predate admission moving into the DAG interpreter. Now it says what actually happens: one place decides, holding every ready task, ordering them so the ones unblocking the most work come first, and asking the machine whether each fits — with anything refused asked again the moment a running task finishes, which is when resources come back. And it says `parallelism` is a property of the machine rather than of a run, since the server is shared between every client on it. Also names which work asks for fork memory (test runners, sourcegen, KSP, and the four platform linkers) and which only takes a slot (compiles, which run inside the server).
The docs site build failed on `/docs/usage/resource-management/` with "Expected component `Callout` to be defined". I wrote `<Callout type="info">` when version-gating the machine-accounting claim; this Docusaurus site has no such component, and the rest of the docs use `:::info` admonitions. Caught only by CI because the site is built by the `deploy` job, and `bleep-site` has no node_modules checked out locally — nothing in the Scala build compiles MDX.
…ject CI failed to resolve `bleep-bsp-tests` and `bleep-tests` with the zinc 2 eviction, while `bleep-bsp` — the only project I had given the scheme — resolved fine. Version schemes are read off the EXPLODED project, so they arrive through `extends` and never through `dependsOn`. A scheme on a library project does not cover the projects that depend on it, even though those projects have the conflicting library on their classpath through exactly that dependency. The docs claimed schemes are "inherited by every transitive consumer", which is what misled me; corrected here too. It passed locally because bleep caches resolutions by a hash that includes the schemes, and a cache hit skips the eviction check entirely. 352 cached entries were hiding the failure. Re-verified with the resolve cache set aside, which is what CI has.
A scheme set on a project did not reach the projects depending on it, even though `dependsOn` pulls that project's libraries into their resolution. So a project could resolve cleanly while every consumer failed on a conflict it neither introduced nor could see, and the only fix was repeating the scheme in each consumer. Resolution now gathers schemes from the project AND from everything it transitively depends on — the same traversal that already pulls in their dependencies, so a library and the judgement about it arrive together. Templates still contribute as before. The build's own scheme stays on `template-common` rather than moving to bleep-bsp, where it belongs semantically: CI bootstraps this build with the PREVIOUS bleep release, which predates this change and would fail to resolve bleep-bsp-tests and bleep-tests. It can move once a release carries this.
Two consecutive runs were cancelled at exactly 15m17s and 15m18s, hitting `timeout-minutes: 15`. That cap was already too tight: on master this job takes 12m17s and 13m45s, so it sat at 82-92% of budget while doing sourcegen, a full compile, publish-local and the entire test suite. The zinc 2 upgrade and per-analysis interning were enough to cross the line. Deliberately not papering over a hang — the log rules that out. Suites complete 4 seconds apart right up to the cut, three running concurrently: 00:07:29 PASSED BspCompilationIntegrationTest: 12 passed 00:07:33 PASSED PublishToHttpMavenRepoIT: 2 passed 00:07:36 PASSED JstackThreadDumpTest: 1 passed 00:07:37 ##[error]The operation was canceled. Steady throughput into a wall. All five native-image jobs pass on this same commit, and they already get 40 minutes for less work. Worth a follow-up: interning computes a SHA-256 per AnalyzedClass on every analysis load, which is the most likely new per-load cost. Cheap to measure now that `analysis_cache` reports hits and misses.
This was referenced Jul 28, 2026
Merged
oyvindberg
added a commit
that referenced
this pull request
Jul 29, 2026
…, keep CI telemetry (#623) * ci: keep the compile server's telemetry from every run The build job got ~50% slower (12-14 minutes on master before #622, 18-20 after) and the evidence needed to explain that is written by the server and then thrown away when the runner is destroyed. Two steps, both `if: always()` because a run that went wrong is the one whose telemetry is worth having: collect metrics.jsonl and the server logs into an artifact, and print a summary into the job log so the common questions need no download at all. The summary answers the question a wall clock cannot: did each compile get slower, or did fewer of them run at once? Per-project durations answer the first, machine occupancy the second, and they have different causes and different fixes. It also flags the case where cpu sits free while the queue is non-empty — admission refusing work it could take, which would be a scheduling bug rather than a slow compiler. CI is also the only environment with a realistic shape for this: 4 cores and 16GB, where the fork-memory budget actually binds. A developer machine cannot reproduce it, which is why the constrained case has gone unmeasured. Verified against a real local run, where it immediately showed 21.9% intern hit rate (sharing 1.28, well under the 2-3x that justified interning), 0% starvation, and a single compile allocating 145GB of churn. * bsp: reuse the ECJ classloader across compiles ECJ keeps the JDK module image in statics -- JRTUtil.images, ctSymFiles, JRT_FILE_SYSTEMS, and classCache (a SoftClassCache of already-parsed JDK class files). Statics are per-classloader, and createEcjTools built a fresh URLClassLoader for every compile, so each Java compile started with all four caches empty: it re-opened ct.sym as a zip filesystem and re-read every JDK class it touched. Reusing one loader per ECJ version is also how ECJ is meant to be used -- the Eclipse IDE holds it for the life of the process. Found by allocation profiling a loaded daemon: 26.6% of all sampled allocation sat on ECJ's own thread in ClasspathJep247Jdk12.findClass -> ZipFileSystem.getPath, churning ZipPath and byte[]. Note that this change does NOT recover that allocation -- most of it is inherent to ECJ compiling. What it recovers is CPU time, because a fresh loader also means ECJ's 807 classes are re-loaded per compile and the JIT never stays warm on the compiler's own hot loops. Measured two ways: isolated A/B, only the classloader differs (800 files / 49.6k lines): fresh loader ~8200 ms, 6265 MB shared loader ~3300 ms, 6180 MB 2.5x faster, allocation ~unchanged end-to-end, dlab dquery/ast (2265 Java files), n=1 per condition: baseline 230.0 s fixed 209.7 s -8.8% The saving is per compile invocation, not per line, which is why one huge project only nets ~9% while the synthetic case -- ten short compiles each re-paying startup -- shows 2.5x. Builds with many Java projects pay it once per project. What this retains is bounded by the number of distinct ECJ versions the daemon serves (normally one), not by compiles or workspaces: 807 classes of metaspace plus a GC-reclaimable SoftClassCache. The previous code paid that class-loading cost per compile instead. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * bsp: read macOS process footprint via proc_pid_rusage, not /usr/bin/footprint ourTreeFootprintMb sweeps the daemon's whole process tree every 5s, and on macOS each pid meant spawning /usr/bin/footprint: ~35ms of address-space survey per target (measured against a 6GB-heap JVM), plus parsing its output, which showed up as 2% of the daemon's total heap allocation in a profile. proc_pid_rusage is the syscall footprint itself reads. Measured on the same process: 11us/call versus 27994us/call, ~2500x, and it allocates one 296-byte buffer. RSS via a single batched `ps` was the tempting cheap fix and is wrong: on that same 6GB-heap JVM phys_footprint reported 7190MB while RSS reported 2933MB, because macOS compresses pages and RSS stops counting them. Subtracting a number 2.5x too small from the machine total is the class of error that produced a 63GB fork budget on a 48GB machine, so the metric has to stay phys_footprint. Verified proc_pid_rusage returns the same value footprint prints (7189 vs 7190MB, the drift being live allocation between the two calls), and that offsets 72 and 240 in the 296-byte rusage_info_v4 are what offsetof reports. Linker.downcallHandle is a restricted method: without --enable-native-access the JVM prints four warning lines on first call and, per JEP 472, a future release blocks the call outright. So the flag ships here rather than later: - the daemon gets it in buildServerCommand, outside javaOpts so it stays out of the daemon's identity key -- it is how bleep runs the server, not a tuning knob - bleep-bsp-tests gets it in bleep.yaml, because ForkCostModelTest measures a real process; scoped to this project rather than to every forked test JVM, since it is this suite's need and not a general one Also: say so when the daemon is killed from outside. Every shutdown the server decides on sets shutdownRequested first, so finding it false in the shutdown hook means a signal or a dead parent. This is not cosmetic -- the client reports "BSP server crashed twice (no OutOfMemoryError recorded)", indistinguishable from a crash, and a session diagnosed a thread dump taken by that very hook as an idle-watchdog bug and filed a report for what was actually a `kill`. Tested: ForkCostModelTest 8/8, including a dead pid declining rather than returning a bogus zero, and peak >= current as a guard on the struct offsets. The full bleep-bsp-tests run is NOT yet green -- it deadlocked twice against a daemon built from another branch, which is being investigated separately. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * bsp: stop double-charging CPU for test forks, which self-deadlocked the server A TestSuiteTask is charged Cost(TestFork, cpu = 1) by the DAG interpreter at admission. JvmPool.acquire then reserved cpu = 1 from the same machine governor AGAIN, from inside the already-admitted task -- asking a finite pool for a second permit while holding the first. That is fine until admission saturates. Once every permit is held by a test task, each of those tasks queues for a permit that cannot exist, and nothing can release because releasing requires finishing: machine: cpu 18/18, mem 0/24259MB, compiles 0, running 18, waiting 18 with zero fork processes, no worker thread doing anything, and the compute pool idle. It is a self-deadlock, not contention. Because machine CPU is daemon-wide while the pool is per test run, it also starved unrelated workspaces: compiles in another worktree sat behind parked test forks for 20+ minutes, and one grant was logged only after 1466599ms, when something elsewhere happened to free a permit. The two reservations were distinguishable in the queue dump only by label -- the interpreter's `test:proj:Suite` versus this one's `test Suite` -- which is why the dump reads as 18 running and 18 waiting for what looks like the same work. Fix: the interpreter is the single authority on machine-wide capacity, so the duplicate reservation goes. What stays in the pool is the local semaphore bounding one run's parallelism, which is not machine-wide and cannot deadlock against admission, and the spawn-time memory reservation, which belongs to the process rather than the suite and outlives this scope. Reproduced on this build before the change (not only on the branch where it was first seen), and diagnosed from the daemon's own queue dump plus a thread dump showing no OS thread involved -- the parked fibers are invisible to jstack, which is what made three earlier theories (leaked reservations, memory starvation, hold-and-wait behind startLimiter) plausible and wrong. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * test: fix ForkCostModelTest's platform fallback, which failed on Windows The new peak-vs-current assertion falls back to identifying the platform when it gets no reading, and I wrote that fallback as `shouldBe ProcessMemory.Linux` -- thinking only of the two platforms that implement a reader. Windows implements neither and is `Unavailable`, so the test failed there while passing on macOS and Linux. CI caught it on windows-latest. Assert what the test actually means: only macOS answers both current and peak. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * bsp: move heap pressure into DAG admission, so it stops holding a CPU permit HeapPressureGate was the last scheduling decision made BELOW admission. The compile was admitted by the interpreter, took a machine-wide CPU permit, and only then slept on the gate -- up to MaxWaitMs, 60s. For that whole time it withheld capacity from every other kind of work: a test or a link that could have run waited behind a permit held by something deliberately doing nothing. That is the same shape as the test-fork double-charge fixed in e3ad7bf, and it is the reason the resource story had three authorities instead of one. It cannot deadlock the way that one did -- the wait is capped and GC resolves the condition externally -- but under sustained pressure N compiles each sit on a permit. So the gate becomes a decision the interpreter consults at admission: HeapPressureGate.decide(usage, othersCompiling, threshold, retryMs, firstRefusedAt, now): Decision a total function of what it observes, testable without a heap, a clock or a scheduler. `TaskDag.Handlers` gains `mayAdmitCompile`, which admission calls before spending a reservation on a compile; false means "not now", the task stays ready, and its permit goes to whatever else can use it. It is reconsidered on the next wakeup -- which fires when a task completes, precisely when heap is most likely to have been freed, so the 2s polling loop is not replaced by anything. Two details the sleep loop got for free and now need saying explicitly: - MaxWaitMs is enforced via a per-run map of when each project was first deferred, because there is no longer a single sleep to measure against. - othersCompiling is `> 0`, not `> 1`: this runs BEFORE the reservation, so the compile being considered is not yet in the count. `idle` bypasses the gate exactly as it bypasses tryReserve -- with nothing running, nothing will complete to reconsider the decision, so deferring would stall the build rather than delay a start. A sole compile is never deferred. Handlers is where this belongs per its own doc: new inputs go in the bundle rather than as positional parameters cascading through callsites, and every field is required, so test callsites pass `_ => IO.pure(true)` explicitly. waitForHeapPressure is deleted rather than left beside decide(), so there is only one copy of the policy. makeCompileHandler loses heapPressureThreshold with it. Tested: bleep-bsp-tests 649 passed / 0 failed (10.2x parallelism), bleep-tests 283 passed / 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * test: loosen cancellation wall-clock bounds that were asserting an idle machine The cancellation tests each start a job that would take 30s or 60s, cancel it, and assert both that the kill was honoured and that the call returned in under 5s (10s in one case). The first assertion is the point of the test. The second was a proxy for "it did not sit through the workload" -- but a tight wall-clock number does not measure the cancellation path, it measures how loaded the machine is, and these run alongside the rest of the suite. That proxy held only because the suite was effectively serialised: a double- charged CPU reservation capped test parallelism at ~1.4x (fixed in e3ad7bf). At the ~11x the suite now genuinely runs at, windows-latest took 6074ms here. The behaviour was correct -- terminationReason was Killed, and only the clock assertion failed: 6074 was not less than 5000 (TimeoutAndResourceTest.scala:86) So the bound moves to a named 20s, still a small fraction of the 30s/60s work it exists to rule out. Applied to all four sites, not just the one that failed: they share the pattern and the flaw, and the other three would fail the next time a runner is busy. This is loosening a test, so to be explicit about what is and is not being given up: the invariant that cancellation short-circuits the work is kept, and the accidental assertion that nothing else is running is dropped. No production code changed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * test: tighten the cancellation bound from 20s to 15s 20s was headroom picked by feel. Against the actual numbers — sub-second healthy (all eight tests in ~1.6s together), 6074ms worst on a loaded runner, 30s for the shortest workload being ruled out — 15s clears the worst observation by ~2.5x and still fails long before a cancellation that never happened. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A shared compile server serving a dozen worktrees ran out of heap, repeatedly. Fixing that turned into making its scheduling bounded, attributable, diagnosable and — by the end — one mechanism instead of four. Every step was measured before it was built.
What was wrong
Concurrency was per-run, not per-machine.
maxParallelismbounded a single compile run, but one daemon serves many clients. Measured on a live server: 14 concurrent connections over 11 worktrees, 18 compiles in flight, heap pegged at 12288/12288MB.Retention was unbounded. The post-GC floor climbed monotonically as workspaces were served — 3566MB at 5 minutes, 6346MB at 30, 8188MB at 45 — until later OOM events fired at a concurrency of three. The floor, not the concurrency, had become the problem.
The heap was full of duplicates. A class histogram at 7.2GB live set, with a forced full GC moving it by 68MB:
xsbti.api.NameHashxsbti.api.Id[Lxsbti.api.PathComponent;xsbti.api.AnalyzedClass~4.5GB of Zinc analyses in an unbounded, daemon-lifetime, soft-referenced map. Soft references are not cleared until the collector is nearly out of room, and until then they are indistinguishable from live data.
What changed
Zinc 1.12.0 → 2.0.4 (
zinc_3, dropping thefor3Use213shim), withConsistentFileAnalysisStore(reproducible = true)so identical inputs produce byte-identical analyses. Two obstacles, both verified rather than worked around:compiler-interface2.0.4 evicts the 1.10.7scala3-compilerwants — safe, since diffing the jars shows it is purely additive (150 classes, zero public or protected members removed) — andxsbti.VirtualFilegained two abstract methods, which breaks implementors even where additive changes are safe for callers.AnalyzedClassis interned, weak-valued, inside the cache so no caller can forget. Measured first, on real analyses: 2.06× sharing within one workspace, 2.96× within bleep's own, and — the control — a second copy of an identical workspace adds exactly zero distinct instances. Divergent branches share little; a freshly forked worktree shares everything.Analyses belong to their workspace and die with its
BuildCacheentry. The separate eviction policy built earlier in this PR — budget, idle timeout, sweep — was deleted once interning removed its premise.One scheduler, one knob.
parallelismhad no CLI whilemax-concurrent-compileshad no docs — two vocabularies for one question, the documented one unreachable (reported in datoria-dev/dlab#301 as "bleep test exposes no concurrency cap"). Nowparallelismis the only concurrency concept: it sizes the governor itself, so cores are merely its default rather than a second limit behind it.heap-pressure-thresholdandmax-cached-workspacesare demoted to internal — tuning constants for mechanisms a user cannot observe.Admission moved into the DAG interpreter. The loop sorted ready tasks most-unblocking-first, then dispatched blind while each task reserved inside its own body — so everything past the width queued FIFO and the sort was decoration. It now admits in priority order against the machine, and
maxParallelismis gone fromexecuteentirely. That also closed a real hole: the Scala.js, Scala Native, Kotlin/JS and Kotlin/Native linkers each fork a process and reserved nothing, counted as one task while being a whole JVM plus a toolchain.A separate compile ceiling was tried and removed. Holding half the machine back for test forks that may not exist is a static partition inside an otherwise work-conserving governor. Its premise dissolved anyway: the failure that motivated it is better explained by
HeapPressureGatehaving read a per-connection compile count — so a dozen clients each running one compile all concluded they were alone and never staggered.A crashed run says so, and its log survives.
cleanupdeletedoutput, and the crash handler calls it before restarting, so a crash destroyed its own evidence — and a non-OOM crash left nothing at all. Rotation kept one generation, but the client retries once, so a systematic crash rotated twice in seconds and overwrote the interesting log with the boring one. Now: the log is kept, two generations survive the retry, history is capped per socket dir, and dead dirs untouched for two weeks are pruned.The daemon stops inheriting
BLEEP_*from whichever client spawned it.pb.environment()starts as a copy of the spawning process, so oneBLEEP_FOO=… bleep compilebecame permanent state on a server that then served everyone else — demonstrated accidentally while writing this, when a single invocation pinned a value into the shared daemon and unrelated test forks inherited it minutes later.Telemetry, aimed at what had to be reconstructed by hand to diagnose all of the above:
heap_live_mb(post-GC live set — retention vs churn),in_flightonoom_pressure(which projects, not just how many),compile_allocation(per-compile, measured where the thread is stable),analysis_cache.per_workspaceandsharing_factor, and periodicmachine/workspace_stateevents.Also: deleted
ReactiveTestRunner, 547 lines with no callers anywhere; and fixed the compile summary, which interpolated no count at all and always readProjects: compiled, N failed.Testing
908 tests pass (630
bleep-bsp-tests+ 278bleep-tests), including the incremental-compilation and KSP suites a zinc major upgrade puts at risk. New coverage: the analysis cache's ownership and interning, the eviction policy, the concurrency knob's derivation and its deliberate absence from the environment, and crash-log retention — including the two cases that would have caught the original bug.Deployed and verified end to end: client and daemon both
1.0.0-M10+78-98dfac2a, governor reporting18 cores … parallelism 18, and the four removed subcommands confirmed absent from the shipped binary.Migration and ordering
analysis.zipand every remote-cache entry is invalidated: one full recompile per worktree, and a window where old and new clients cannot share cache entries.🤖 Generated with Claude Code