Add a read-only operational web UI to the proxy by pambrose · Pull Request #224 · pambrose/prometheus-proxy

pambrose · 2026-07-21T09:06:06Z

Implements Feature 5, the last of the five in docs/FEATURE_PROPOSALS_JULY_2026.md. Answers the question this system raises most often — why isn't this target scraping? — which today means log-diving on two machines or curling a plain-text debug servlet.

A read-only master-detail dashboard: connected agents on the left; for the selected agent its identity, registered paths, backlog, eviction countdown, and its own recent scrapes on the right. Live over a WebSocket.

⚠️ Off by default, and unauthenticated

--ui / UI_ENABLED / proxy.ui.enabled, listening on its own port (8094). It has no auth and no TLS, the same posture as the admin and metrics endpoints, and it puts agent names, hostnames, target URLs and recent activity in one place. Documented in four places as internal-only. Worth confirming at review that it's prominent enough.

Two deliberate departures from the proposal

Not the admin port — despite the proposal's own section title. That port is a Jetty servlet container created inside common-utils whose only extension point is a path → Servlet map; Ktor routing and WebSockets cannot attach. A separate port is also the better posture: Kubernetes probes target /ping and /healthcheck on the admin port, so a shared port couldn't be firewalled without dropping health checks.

Not "no framework, no external assets" — htmx is both. It arrives as a WebJar rather than vendored source, so no third-party JavaScript is committed here and versions stay in normal dependency tooling. Still no CDN: the assets ship in the fat JAR, which matters for a product whose purpose is bridging restricted networks. A CDN-loaded UI would render blank in exactly that environment, failing at page-open during an incident rather than at deploy.

Both are recorded in the as-built notes rather than left to be discovered.

Supporting infrastructure

The proxy had no observability hooks — every collection was a plain map with no listeners. This adds:

ProxyEvent bus (MutableSharedFlow). tryEmit with DROP_OLDEST never suspends, which is what makes it safe inside synchronized(pathMap) and on gRPC transport threads — it can never stall a registration.
ProxySnapshot — an immutable, layout-neutral projection collected once and fanned out to all sessions.
ScrapeRecord — a structured queue alongside /debug's text one, which carries no agent attribution.
Three previously-private AgentContext fields, plus a wall-clock connectTime (every other timing field is a Monotonic TimeMark that can't render as a time of day).

Bugs the tests caught

ConcurrentHashMap rejects null values — storing "no selection" as null threw on every WebSocket connect. Only the end-to-end test could see it.
Ktor's staticResources didn't resolve the WebJar layout — replaced with an explicit two-entry allowlist, which also removes path-traversal surface.
The push loop ran at scrape rate, not the refresh interval — CONFLATED collapses bursts but imposes no floor, so a busy proxy collected snapshots ~40× more often than intended, taking the scrape path's own monitor. Found by the cleanup pass.
Adversarial review found the AgentConnected KDoc was factually false (it fires at transport-ready, before auth and identity) and that identity assignment had no event at all.

Both end-to-end specs were verified to fail when the feature is broken. ContainersWebUiTest exists specifically to catch a fat-JAR packaging regression that drops the WebJar resources — invisible to any test that only checks HTML.

Known limits, all documented

A departed agent's paths vanish from the UI. The proxy deletes them on disconnect, and the layout is agent-centric — so a path that stopped working is absent rather than shown broken. This is the sharpest limit relative to the feature's purpose. Unlike the two below it needs no protocol change: recentScrapes already retains the data. A path-centric layout would close it, and is planned as a follow-up.
Path source is invisible — whether a path came from static config or dynamic discovery is an agent-side fact the registration RPC doesn't carry.

Failover state was the other half of this, and is now closed. RegisterAgentRequest gained proxy_endpoints and current_endpoint_index, so an agent reports its configured failover list and which entry this connection uses. The detail pane renders via proxy-b:50051 (2 of 2) plus a failed over tag when the index is past the first entry — which is what distinguishes an agent that failed over to this proxy from one that started here. Registration is the right moment to report it: a failover is a reconnect, so the index is accurate whenever the proxy learns it. Both fields are additive and optional, so an agent predating them leaves them unset and the line is omitted.

🤖 Generated with Claude Code

Groundwork for the operational web UI: server-rendered HTML via the Ktor HTML DSL, htmx for interaction, WebSockets for live updates. Nothing consumes them yet. All five artifacts publish at 3.5.1, the version already pinned, so no Ktor upgrade is needed. ktor-htmx and its siblings are marked experimental upstream. Verified the fat JAR still assembles correctly, since kotlinx-html arrives as a new transitive (645 classes) and this project has previously lost META-INF service entries to ShadowJar's duplicate handling: the gRPC NameResolverProvider and LoadBalancerProvider files still carry their full entries, so the DNS resolver regression did not recur. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Declares proxy.ui { enabled, port, path, refreshIntervalSecs, recentScrapesQueueSize } with --ui / --ui_port / --ui_path CLI flags and UI_ENABLED / UI_PORT / UI_PATH env vars, resolved CLI > env > config like every other option. Nothing reads them yet. Off by default, matching the admin and metrics posture: the UI renders agent names, hostnames, target URLs and recent activity in one place, on a port with neither auth nor TLS. The port is deliberately its own rather than the admin port's. Kubernetes liveness and readiness probes target /ping and /healthcheck on 8092, so sharing would mean a UI cannot be firewalled without taking the probe endpoints with it. A test pins that the two ports differ. Unlike agent.proxy.endpoints, every key here is a scalar, so tscfg emits a hasPathOrNull guard for each and for the block itself -- there is no unguarded getList, and a config predating the UI loads unchanged. A ConfigValsTest case pins that so the guard cannot silently regress. The EnvVars drift guard did its job and is updated: 47 entries to 50, with the three new names added to both the completeness list and the contains-all list. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Three source changes, none of which alter existing behavior. remoteAddr and launchId become readable on AgentContext. Both were private and leaked only through toString(). remoteAddr is the only field that establishes which machine an agent is actually on -- agentName is self-reported -- and launchId distinguishes two runs of the same agent name. AgentContext gains a wall-clock connectTime. Every other timing field is a Monotonic TimeMark, which measures elapsed time correctly but cannot be rendered as a time of day. The marks are left alone: they are immune to clock adjustments and remain the right basis for eviction. ScrapeRequestResponse now carries the agentId that served it, and a structured EvictingQueue<ScrapeRecord> is populated alongside the existing text queue. The /debug servlet's queue holds pre-formatted strings with no agent attribution and no field boundaries, so a per-agent scrape view could only be built from it by parsing display text back apart. The two queues are populated from one site and neither depends on the other; the text format stays a stable operator surface, and the structured one is sized independently because the UI shows more history than a debug dump needs. agentId is carried on the response rather than read at the logging site because executeScrapeRequests loses the AgentContext at awaitAll(). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Nothing in the proxy was observable before this: every collection was a plain ConcurrentHashMap or a synchronized HashMap with no listeners, callbacks, or flows, so the only way to watch state was to poll it. The bus emits at the five points where topology actually changes -- agent connect and disconnect, path register and unregister, scrape completion. Emitting is a tryEmit on a SharedFlow configured DROP_OLDEST, which never suspends and never blocks. That is what makes it safe at these call sites: they run inside synchronized(pathMap) and on gRPC transport threads, where a slow or absent subscriber must never be able to stall a registration or a disconnect. Dropping rather than buffering is deliberate. Consumers re-read a full snapshot when woken, so a missed event costs a slightly later refresh, never a wrong render -- the bus is a wake-up signal, not a ledger. Scope is deliberately narrow: only transitions with an identifiable moment. Values that drift -- backlog depth, map sizes, eviction countdowns -- have no such moment and are left to be sampled on a timer by whoever needs them, which will be the UI service rather than this bus. AgentContextManager's eventBus parameter is defaulted so the ~15 existing test call sites keep working; a private bus with no subscribers is a no-op. Proxy passes its shared instance explicitly. Tests use onSubscription rather than sleeps or yields, so subscriber registration is ordered deterministically against the first emit. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

ProxySnapshot materializes agents, paths, recent scrapes and health counters into immutable views. Nothing consumes it yet. Materializing rather than referencing live state is the point: AgentContextManager.agentContextEntries is the ConcurrentHashMap's live entry set, not a snapshot, so two passes over it can disagree. Copying once means every WebSocket session renders the same consistent picture. Deliberately avoids two accessors that look useful and are not. ProxyPathManager.toPlainText() holds the path monitor across a full sort plus a toString() per entry. totalAgentScrapeRequestBacklogSize is O(agents x backlog depth) because ConcurrentLinkedQueue.size() traverses -- it is slowest exactly when backlogs are deep, which is when a health view is most likely to be watched. Cost is two monitor acquisitions per collect regardless of session count, which is why the caller must collect once and fan out rather than collect per session. Documented as unsafe to run on a Ktor CIO thread: Kotlin's synchronized parks the carrier thread, so collecting on the event loop would couple the operator UI to scrape latency. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Three review lenses each came back BROKEN. Six findings survived judging; eleven were refuted, including the two scariest -- that tryEmit resumes subscribers inline inside synchronized(pathMap), and that a lock-ordering inversion with recentScrapes becomes reachable. Neither holds. AgentConnected claimed "an agent completed registration and is now serving". It does not. It is emitted from the gRPC transport filter, which runs before per-call auth and before registerAgent, so identity is still "Unassigned" and the peer may yet be rejected -- anything completing an HTTP/2 handshake reaches it, including a health probe or a bad-token client. The KDoc now says so. Identity assignment had no event at all, so an agent registering zero paths would have shown as "Unassigned" indefinitely: nothing would ever wake a consumer to re-read it. AgentRegistered is emitted from registerAgent where the identity actually arrives. AgentDisconnected moved from AgentContextManager to Proxy.removeAgentContext, so its happens-before edge covers the path sweep as well as the context removal. Emitted at the old site, a consumer could wake between the two calls and snapshot a path map that still listed the departed agent. removeFromPathManager emitted nothing when a consolidated path survived losing one agent. That is the one case with no later event to self-correct from, so the stale render would have stuck rather than resolving on the next wake-up. Also: ScrapeRecord.statusCode is this agent's leg of a scrape, not necessarily the merged status Prometheus saw on a consolidated path. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

ktor-htmx ships only Kotlin attribute constants (HxSwap, HxEvents, and friends) and no JavaScript, so the htmx client library has to come from somewhere. htmx 2.x also moved WebSocket support out of core into a separate extension, so two files are needed rather than one. WebJars make both normal Gradle dependencies instead of committed source: the assets arrive inside a JAR under META-INF/resources/webjars/ and Ktor serves them from the classpath. No third-party JavaScript is committed to this repository, the versions are visible to the same dependency tooling as everything else, and the assets ship inside the fat JAR -- which matters for a product whose whole purpose is running where network access is restricted. A CDN would render a blank page in exactly that environment, and it would fail at page load rather than at deploy. htmx-ext-ws declares a compatible range on core that resolves to the same 2.0.10, so the two cannot drift apart. Verified both land in the fat JAR (htmx.min.js at 51KB, ws.js at 15KB) and that the gRPC NameResolverProvider service file survived, since that merge has regressed before. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

ProxyUiService runs its own Ktor CIO server on its own port, off by default. Not the admin port -- that is a Jetty servlet container created inside common-utils whose only extension point is a path-to-Servlet map, so Ktor routing, the HTML DSL and WebSockets cannot attach to it. Not the scrape port either: that one is Prometheus-facing and should stay predictable. A separate port also lets the UI be firewalled without taking /ping and /healthcheck with it, which Kubernetes probes target. Everything is server-rendered. The WebSocket carries HTML fragments rather than JSON, so there is no client-side templating and no duplicated view logic. Fragments carry stable ids and hx-swap-oob, letting one frame update the agent list, the detail pane and the status bar independently. Selection lives in the URL via hx-push-url, so it survives reload and is bookmarkable. The only hand-written JavaScript reads that selection back out and tells the server, since the push loop needs to know whether this session wants a detail pane -- the one thing htmx does not model. One shared push loop, not one per session: it wakes on an event or a timer tick, collects a single snapshot, and fans the same fragments to every session. The wake channel is CONFLATED so a fleet reconnecting collapses into one collect. The timer is what keeps drifting values live -- backlogs and eviction countdowns have no moment to emit from and would otherwise sit frozen between topology changes. Two things the end-to-end test caught that unit tests structurally could not. ConcurrentHashMap rejects null values, so storing "no selection" as null threw on insert and killed every session at connect; it is a sentinel now. And Ktor's staticResources did not resolve the webjar layout, whose path embeds a version -- replaced with an explicit two-entry allowlist, which also removes any path-traversal surface. The harness test connects an agent AFTER the socket is already open, so the fragment it asserts on can only have come from the push path. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

ContainersWebUiTest exists for one reason the harness test cannot cover: the fat JAR. htmx ships as a WebJar served from the classpath, so a packaging regression that drops those resources would still render the page while leaving every interaction dead -- invisible to any test that only checks HTML. Verified it fails when the asset path is broken. Documentation records the two places this implementation deliberately departs from the proposal, rather than quietly diverging. The proposal's own section title says "on the Proxy Admin Port". That is not achievable: the admin port is a Jetty servlet container created inside common-utils whose only extension point is a path-to-servlet map, so Ktor routing and WebSockets cannot attach to it. The UI runs on its own port, which also turns out to be the better posture -- Kubernetes probes target /ping and /healthcheck on the admin port, so a shared port could not be firewalled without taking the probes with it. The proposal also specifies "no framework, no external assets". As built there is still no build toolchain and no CDN, but htmx is a framework and a third-party asset. It arrives as a WebJar rather than vendored source, so nothing third-party is committed here and versions stay visible to normal dependency tooling. New website page covering what the UI shows, why it has its own port, why it works airgapped, and its limits -- including that it is unauthenticated and must be treated as internal. Cross-referenced from Troubleshooting, which is where someone lands with the question the UI answers. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Three things operators will ask the UI for are invisible to it for one shared reason: they are facts the agent knows and the registration RPC does not carry. Path source (static vs discovered, Feature 1), failover endpoint and rotation state (Feature 2), and Feature 3 per-agent identities. The failover case has a real operational consequence worth stating before someone hits it during an incident rather than after. In an HA pair each proxy's UI shows only its own agents, so a failover makes an agent disappear from one dashboard and reappear on the other with nothing on either screen explaining why -- at exactly the moment someone is watching. There is a partial thread to pull, and the UI already renders it: launchId is generated once per agent process, so it survives a failover, while agentId is assigned by each proxy independently and does not. An operator can correlate the same agent process across two proxies by eye. Documented as the workaround it is, not as a feature. Closing any of these properly means extending RegisterPathRequest / RegisterAgentRequest -- worth doing once for all three rather than three times, which is why it is logged as one open question rather than three. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The push loop ran at scrape rate rather than the refresh interval. Every ScrapeCompleted woke it, and Channel.CONFLATED collapses bursts but imposes no floor -- so at roughly twenty scrapes a second the loop collected a snapshot twenty times a second, taking the same path-map monitor every scrape request takes. That defeated the design: ProxySnapshot.collect goes to lengths to run off the CIO event loop so the UI cannot couple to scrape latency, and the wake path coupled them anyway. Scrape history is a drifting value, which by the event bus's own contract is the timer's job, so it no longer wakes the loop. The same KDoc also overclaimed: it said it avoided an O(agents x backlog) cost, then paid exactly that per agent via scrapeRequestBacklogSize. It now says so, and explains why the refresh-interval bound makes it acceptable. renderStatus wrapped itself in a span carrying the status id, and pushFragment wrapped it again with the same id, so every frame contained nested duplicate ids. All three regions are now self-wrapping behind an oob flag, which also removed the duplicated attribute setup between renderPage and pushFragment. agentId came off ScrapeRequestResponse. It needed seven construction sites from two different expressions and defaulted to empty, so any future branch that forgot it would silently emit an unattributed record. map and awaitAll preserve order, so a zip recovers provenance at the single site that needs it. logActivityForResponse is renamed recordScrapeOutcome, since it is now where a scrape becomes observable rather than merely logged. The UI server skipped ProxyHttpConfig.configureKtorServer, so htmx.min.js went out uncompressed and failures returned bare 500s; assets were also re-read from the JAR per request on a port with no auth. Now configured like the proxy's own HTTP service, with assets read once and marked immutable. Also: ktor-htmx, ktor-htmx-html and ktor-server-htmx were shipping in the fat JAR with nothing importing them -- added on the assumption of the typed DSL, then every attribute was hand-written. Dropped. SessionKey plus a sentinel plus three null conversions became one Session object. Five unread fields removed from the snapshot DTOs, along with a test asserting a state the path manager cannot produce. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

When an agent disconnects, ProxyPathManager deletes its paths rather than leaving them empty, so a zero-agent path cannot exist. Combined with an agent-centric layout, a path that has stopped working is absent from the UI entirely -- Prometheus gets a 404 and the dashboard shows nothing named after it. That is the exact question this feature was built to answer, so it is worth recording plainly rather than leaving for an operator to discover. Distinguished from the other two gaps on purpose: path source and failover visibility need a proto change, because the agent knows facts the proxy never receives. This one does not -- recentScrapes already retains records naming the path. What is missing is a view onto state the proxy already keeps, which a path-centric layout would provide. The website page frames it as troubleshooting guidance rather than a disclaimer, since that is how someone will meet it: if Prometheus reports a target failing and the path is not on the dashboard, look for a missing agent instead. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

codecov · 2026-07-21T09:11:12Z

Codecov Report

❌ Patch coverage is 92.53333% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.06%. Comparing base (c68d605) to head (6cf5293).

Files with missing lines	Patch %	Lines
...in/kotlin/io/prometheus/proxy/ui/ProxyUiService.kt	83.69%	9 Missing and 6 partials ⚠️
.../main/kotlin/io/prometheus/proxy/ui/ProxyUiHtml.kt	92.85%	0 Missing and 8 partials ⚠️
...rc/main/kotlin/io/prometheus/proxy/ProxyOptions.kt	85.71%	0 Missing and 2 partials ⚠️
...ain/kotlin/io/prometheus/proxy/ui/ProxySnapshot.kt	97.36%	0 Missing and 2 partials ⚠️
...rc/main/kotlin/io/prometheus/proxy/ScrapeRecord.kt	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master     #224    +/-   ##
========================================
  Coverage   92.05%   92.06%            
========================================
  Files          45       50     +5     
  Lines        2947     3316   +369     
  Branches      541      594    +53     
========================================
+ Hits         2713     3053   +340     
- Misses        107      116     +9     
- Partials      127      147    +20

Flag	Coverage Δ
unittests	`92.06% <92.53%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/main/kotlin/io/prometheus/Proxy.kt	`85.25% <100.00%> (+1.22%)`	⬆️
...ain/kotlin/io/prometheus/agent/AgentGrpcService.kt	`90.28% <100.00%> (+0.15%)`	⬆️
src/main/kotlin/io/prometheus/common/EnvVars.kt	`89.39% <100.00%> (+0.50%)`	⬆️
...rc/main/kotlin/io/prometheus/proxy/AgentContext.kt	`100.00% <100.00%> (ø)`
.../kotlin/io/prometheus/proxy/AgentContextManager.kt	`96.96% <100.00%> (+0.09%)`	⬆️
...c/main/kotlin/io/prometheus/proxy/ProxyEventBus.kt	`100.00% <100.00%> (ø)`
...main/kotlin/io/prometheus/proxy/ProxyHttpRoutes.kt	`87.50% <100.00%> (+0.83%)`	⬆️
...ain/kotlin/io/prometheus/proxy/ProxyPathManager.kt	`90.00% <100.00%> (+0.43%)`	⬆️
...ain/kotlin/io/prometheus/proxy/ProxyServiceImpl.kt	`93.18% <100.00%> (+0.03%)`	⬆️
...rc/main/kotlin/io/prometheus/proxy/ScrapeRecord.kt	`87.50% <87.50%> (ø)`
... and 4 more

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c68d605...6cf5293. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Each proxy's UI shows only its own agents, so a failover looked like an agent vanishing from one dashboard and a stranger appearing on the other -- at exactly the moment someone is watching. RegisterAgentRequest gains proxy_endpoints and current_endpoint_index. The agent reports both at registration rather than on a heartbeat because registration is precisely when they change: a failover IS a reconnect, so the index is accurate whenever the proxy learns it. Both fields are additive and optional, so an agent predating them simply leaves them unset and the UI omits the line. The detail pane renders "via proxy-b:50051 (2 of 2)" plus a "failed over" tag whenever the index is past the first entry, which is what distinguishes an agent that failed over to this proxy from one that started here. An out-of-range index degrades to no position rather than throwing -- this is remote input and an older or misbehaving agent could send anything. The harness test drives the real path: an agent whose primary endpoint is a dead port connects to its secondary, and this proxy's UI says so. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01TSB1RVyoc2UXsKjJAUJbZZ

pambrose and others added 12 commits July 21, 2026 00:01

pambrose merged commit 15526fc into master Jul 21, 2026
6 checks passed

pambrose deleted the feature/proxy-web-ui branch July 21, 2026 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a read-only operational web UI to the proxy#224

Add a read-only operational web UI to the proxy#224
pambrose merged 13 commits into
masterfrom
feature/proxy-web-ui

pambrose commented Jul 21, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pambrose commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Off by default, and unauthenticated

Two deliberate departures from the proposal

Supporting infrastructure

Bugs the tests caught

Known limits, all documented

Uh oh!

codecov Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pambrose commented Jul 21, 2026 •

edited

Loading

codecov Bot commented Jul 21, 2026 •

edited

Loading