feat(api): serve the provider egress probe schedule from the server#7
Open
Ryanmello07 wants to merge 2 commits into
Open
feat(api): serve the provider egress probe schedule from the server#7Ryanmello07 wants to merge 2 commits into
Ryanmello07 wants to merge 2 commits into
Conversation
The operator's prober decided what to probe from an in-memory ttl cache, so a restart re-probed the whole provider population and nothing durable recorded what was actually due. The freshness data already exists server-side in provider_egress_location.observed_at; this exposes it so the server owns the schedule. model.GetProviderEgressLocationDue sources candidates from the live provider population (network_client_location_reliability, connected + valid) and LEFT JOINs the egress row, rather than selecting from provider_egress_location. The dominant case is a provider that has never been probed and so has no egress row at all; selecting from that table would return exactly the providers that least need probing and none of the ones that most do. Only providers holding a Public provide key are returned. Probing tunnels through the provider itself, i.e. opens a contract from outside the provider's own network, which a provider without a Public key refuses -- offering one to the prober would burn a probe slot on a guaranteed failure. Same EXISTS filter UpdateClientLocations and UpdateClientScores already apply. The staleness cutoff is computed in Go and bound as a query argument. observed_at is a naive `timestamp` holding utc, so comparing it against sql now() would cast through the session timezone and silently skip a window. GET /network/provider-egress-due is operator-to-server, authenticated by the same X-UR-Operator-Secret shared secret as the ingest endpoint (hmac.Equal against the memoized, fail-closed operatorIngestSecret) rather than a network jwt. Its cutoff is half ProviderEgressLocationMaxAge: at the full max age every location would lapse to the mmdb fallback at the exact moment it became due and stay lapsed until the prober reached it. Co-Authored-By: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01QtgqtCmKJRXdsQ5ktiqwkg
…t succeed The only thing that moved a provider off the head of the due queue was a successful probe writing provider_egress_location. A provider that connects and holds a Public provide key but always fails to probe -- firewalled egress, dead upstream, anything other than the missing Public key the query already screens for -- never gets a row, so its observed_at stays NULL, so it sorts ahead of every stale-but-refreshable provider on every poll, forever. Five hundred such providers and a prober asking for limit=500 gets the same five hundred dead providers every time; no healthy provider's location is ever refreshed. It fails silently: the endpoint keeps returning a full, plausible-looking batch. The in-memory ttl cache this replaced was incidentally immune, because it marked a provider probed whether or not the probe worked; moving the schedule server-side dropped that protection. Record attempts, not just successes. provider_egress_probe_attempt is a small table keyed by client_id -- the attempt cannot live on provider_egress_location, because the case it exists to handle is precisely a provider with no row there. GetProviderEgressLocationDue now requires both no fresh success and no recent attempt, with a much shorter backoff for attempts (6h) than for success freshness (half ProviderEgressLocationMaxAge, 3.5d): a failing provider should be retried periodically, just not on every poll. Both cutoffs are computed in Go and bound as arguments -- attempt_at, like observed_at, is a naive `timestamp` holding utc, and comparing it to sql now() casts through the session timezone. The prober reports an attempt to POST /network/provider-egress-attempt, on the same operator-authenticated surface as the other two endpoints (X-UR-Operator-Secret, hmac.Equal, the memoized fail-closed operatorIngestSecret). The server timestamps the attempt rather than trusting the prober's clock. Attempt rows are swept by the existing expiry task. This pulls probe_attempt_at/probe_failure forward from the P2 data model in docs/superpowers/specs/2026-07-25-enforced-provider-geo-probing-design.md, because P1's schedule cannot function without them. Only those two columns, not the rest of the verdict model. Also adds a client_id secondary sort key, so batch composition under a limit is deterministic rather than plan-dependent -- the whole never-probed population otherwise ties on NULL observed_at. Tests: a provider never probed successfully but attempted seconds ago must not be due, must not take the single batch slot from a stale-but-refreshable provider, and must become due again once the backoff elapses; the same over http via the attempt endpoint. Plus the two coverage gaps review flagged: deleting either the connected or the valid predicate now fails a test, and the handler's staleness cutoff is exercised by a probed (not merely never-probed) provider. Co-Authored-By: Claude <[email protected]> Claude-Session: https://claude.ai/code/session_01QtgqtCmKJRXdsQ5ktiqwkg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
feat/probe-location-force-minimum(P1 Task 1, #6), not onbeta/self-contained-env. That PR is still open; basing here keeps this PR to the single commit it actually contains instead of replaying Task 1's. Retarget tobeta/self-contained-envonce #6 merges (GitHub does this automatically on merge).P1 Task 2 of the automated confined probing plan.
Why
The operator's prober decides what to probe from an in-memory TTL cache, so a restart re-probes the whole provider population and there is no durable record of what is actually due. The freshness data already lives server-side in
provider_egress_location.observed_at. This exposes it, so selection is durable and the server owns the schedule.What
model.GetProviderEgressLocationDue(ctx, minObservedAt, limit) []server.Id— client ids of Public providers whose newest probe is older thanminObservedAtor absent, oldest first.GET /network/provider-egress-due?limit=N→{"client_ids":["..."]}.Three things about the shape of the query matter:
provider_egress_location. Candidates are sourced fromnetwork_client_location_reliability(connected = true AND valid = true) with the egress row LEFT JOINed on. The dominant case is a provider that has never been probed and therefore has no egress row at all; selecting fromprovider_egress_locationwould return exactly the providers that least need probing and none of the ones that most do.EXISTS (SELECT 1 FROM provide_key ... provide_mode = $1)filterUpdateClientLocations/UpdateClientScoresalready apply.observed_atis a naivetimestampholding utc; comparing it against SQLnow()casts through the session timezone and silently skips a window. This codebase has shipped that bug before.Ordering is
observed_at ASC NULLS FIRSTso the longest-unprobed (including never-probed) go first.Auth
Operator-to-server, not a client route: the same
X-UR-Operator-Secretheader,hmac.Equalcomparison and memoizedoperatorIngestSecret()as the existing ingest handler, which returns""when the vault resource is missing so the endpoint fails closed.limitdefaults to 100 and is clamped to 500. Alimitthat is not a positive integer is a 400 rather than being clamped up:limit=0would come back as an empty list, indistinguishable from "nothing is due".The handler's staleness cutoff is
model.ProviderEgressLocationMaxAge / 2. At the full max age, every location would lapse to the mmdb fallback at the exact moment it became due and stay lapsed until the prober worked around to it.Tests
TDD: model test first (failed to compile,
undefined: GetProviderEgressLocationDue), then implementation, then handler tests, then handler.TestGetProviderEgressLocationDuestands up four connected + valid providers differing only in probe freshness and provide mode — fresh (1h), stale (72h), never probed, and one holding only a Network key — and asserts membership, ordering, andlimit.Handler tests cover no secret → 401, wrong secret → 401, altered secret with the vault configured → 401, correct secret → 200 with a real never-probed provider present in
client_ids,limit=1honoured, and badlimit→ 400.Mutation-verified (each mutation reverted afterward):
OR TRUE)must not contain the provider probed an hour ago=→<=)must not contain the provider without a Public provide keyNULLS FIRST→NULLS LASTnever-probed provider at 1 must sort before the stale one at 0status = 401, want 400/ 200)That last row is the point of the accept-case test: an auth suite that only asserts rejections passes against a handler that is broken shut.
27 tests pass across
model,api/handlersandcontroller(all pre-existingTestProviderEgressLocation*andTestSubmitProviderEgressLocation*included).go build ./...andgo vet ./...clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01QtgqtCmKJRXdsQ5ktiqwkg