Fix/connection introspection diagnostics#850
Open
girishjeswani wants to merge 4 commits into
Open
Conversation
Introduces connection_error.ts: classifies a (flattened) schema-fetch error string into auth / transport / not_found / other, and builds a single developer-facing diagnostic naming the connection, table, HTTP status, and likely cause. Only auth/transport are flagged as infrastructure failures (the ones we'll short-circuit); not_found and "other" are left to produce normal Malloy compile diagnostics. Maps auth -> ConnectionAuthError (HTTP 422) and transport -> ConnectionError (HTTP 502), reusing the existing error types. Pure functions with unit tests covering 401/403/404/5xx/408, raw network/timeout errors, keyword fallbacks, and the not-a-model-error messaging. Groundwork for fixing the "expired token -> import reference failure -> dozens of 'X is not defined'" cascade. Signed-off-by: Girish Jeswani <[email protected]>
When a connection can't introspect a table or SQL schema for an infrastructure reason (expired token, unreachable data plane), the pool now records the classified failure on the job (handleSchemaForTables and handleSchemaForSql, both the errors-map and thrown-error paths) and, in completeJob/errorJob, fails the whole package load with a single ConnectionAuthError (422) / ConnectionError (502) instead of resolving the worker's cascade of derivative "X is not defined" errors. Package.loadViaWorker and reloadAllModels now let those typed connection errors bubble through with their 4xx/5xx mapping rather than rewrapping them as a misleading 503. Genuine model errors are untouched: a 404 / "table does not exist" and any unclassified error are NOT recorded, so they still compile to normal Malloy diagnostics. Signed-off-by: Girish Jeswani <[email protected]>
Adds a FlakyConnection (throws like PublisherConnection on an expired
token) and three real-worker tests over a model that references it:
- auth (401) -> loadPackage rejects with a single ConnectionAuthError
naming the connection + table + status; the "X is not
defined" / "import reference failure" cascade does not
leak into the message.
- transport (503) -> rejects with a single ConnectionError.
- not-found (404) -> NOT short-circuited; resolves with the normal in-band
per-model compilationError (genuine model errors are
preserved).
Signed-off-by: Girish Jeswani <[email protected]>
… path) The first cut classified failures only in handleSchemaForTables / handleSchemaForSql. But @malloydata/db-publisher's PublisherConnection.create() validates the token eagerly (getConnection + test) at connection-resolution time, so an expired token (401) / unreachable plane (5xx) throws out of lookupConnection — the worker's connection-metadata RPC — BEFORE any schema fetch. That path was unclassified, so a real expired token still produced the full "import reference failure" + "X is not defined" cascade (HTTP 424), verified live against a live publisher-type data plane. Fix: classify infra failures in handleConnectionMetadata's catch too, and make SchemaFetchFailure.target optional — at resolution no specific table is in hand, so the diagnostic reads "while establishing the connection". Result, verified live: one ConnectionAuthError (422) / ConnectionError (502), no cascade. Tests: add real-worker regression tests that fail at RESOLUTION (a lookupConnection that throws, mirroring create()), confirmed to fail without this change and pass with it; add a unit test for the connection-level phrasing; and correct FlakyConnection's doc comment, which wrongly claimed it failed the way a real expired token does. Signed-off-by: Girish Jeswani <[email protected]>
girishjeswani
force-pushed
the
fix/connection-introspection-diagnostics
branch
from
June 26, 2026 20:00
fc06ed2 to
6057843
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a connection fails to introspect a schema for an infrastructure reason —
an expired/invalid token (HTTP 401/403) or an unreachable/timing-out data plane
(5xx / network / timeout) — Publisher surfaced it as a cascade of Malloy
compile errors (
import reference failure+ dozens of'X' is not defined).That sends developers (or their agents) to debug their
.malloymodel when the real cause is theconnection. This PR classifies such failures during package load and fails with a
single, accurate connection diagnostic instead — while leaving genuine model
errors (and real "table not found") untouched.
Before
A package whose source references a connection with an expired token fails to load with:
HTTP 424
Error(s) compiling model:
line 4: import reference failure
line 5: 'customer_id' is not defined
line 6: 'order_total' is not defined
line 7: Reference to undefined value revenue
... (one per field referencing the unresolved source)
Nothing here names the connection, the status, or the fact that a token expired.
Root cause
Two-stage information loss:
@malloydata/db-publisher'sPublisherConnectioncollapses anAxiosErrorto a bareError("Request failed with status code 401")— no status, connection, table,or auth-vs-not-found signal.
conn.table(...)unresolved, so the source and every field referencing it emittheir own
"X is not defined".Two findings that shaped the fix (both verified empirically):
internally and still produces the full cascade. The failure must be
short-circuited on the main thread, before the worker's result is consumed.
PublisherConnection.create()validates the token eagerly (getConnection()+test()), so the 401 throws out oflookupConnection— the worker'sconnection-metadataRPC — before anyfetchSchemaForTables.The fix
connection_error.ts): maps a flattened error string toauth/transport/not_found/other(HTTP-status-first, withnetwork/timeout keyword fallback) and builds a developer-facing diagnostic.
auth/transportfailure across all three introspection RPC handlers —
handleSchemaForTables,handleSchemaForSql, andhandleConnectionMetadata(the resolution path) —covering both the per-table
errorsmap and thrown-error paths. The load thenfails with one typed error instead of resolving the cascade.
auth → ConnectionAuthError(422),transport → ConnectionError(502).Package.loadViaWorker/reloadAllModelslet these bubble through with their 4xx/5xx mapping instead of rewrapping as 503.
not_found(404 / "table does not exist")and anything unclassified are not short-circuited — they still compile to
normal Malloy diagnostics.
After
HTTP 422
Connection "prod_warehouse" returned 401 Unauthorized while establishing the
connection. The access token is likely expired or invalid; refresh the connection
and reload. This is a connection problem, not an error in your Malloy model
(underlying error: Request failed with status code 401).
Testing
keyword fallbacks, connection-level vs table-level phrasing).
connection-resolution path, for
auth(→ oneConnectionAuthError, no cascade)and
transport(→ oneConnectionError);not_found(404) asserted to remaina normal in-band per-model compile error. The resolution-path tests were
confirmed non-vacuous (they fail without the fix).
invalid token: branch-before returned the 424 cascade; branch-after returns the
single 422 diagnostic above.
Follow-ups (next steps)
This PR makes the failure legible; it does not reduce how often it happens.
The most common trigger — a token that aged out — is best removed by the next two
changes, which this PR is a prerequisite for (they previously presented as a
model cascade and so were hard to even identify as connection problems):
connection — token and all — under
publisher_data/and reuses it acrossrestarts, so injecting a fresh token and reloading can keep using the expired
one until
publisher_data/is cleared. This makes the "refresh the connectionand reload" remediation in the new diagnostic only partially effective today.
Next PR: reconcile/evict the persisted connection on reload so a config token
change actually takes effect.
(a window reload/re-login doesn't refresh the connection's token either). The
connection/credential layer that owns the token should refresh it proactively
so the expired-token case largely stops occurring. Separate layer/repo from
this server-side fix.
Out of scope (this PR)
@malloydata/db-publishercarrying{ status, kind }— lives in a separate repo; this PR classifies server-sidefrom the (status-bearing) error string.
"not defined"errors under the root cause forunclassified connection failures (defense-in-depth).
Compatibility / risk
No behavior change for other connection types (BigQuery, DuckDB, Postgres, …) on
success, and genuine model/compile errors are unchanged — only infrastructure
auth/transport failures are short-circuited.