Skip to content

test(materialization): GC + scheduler state-machine transition coverage#902

Open
girishjeswani wants to merge 3 commits into
malloydata:mainfrom
girishjeswani:test/materialization-gc-coverage
Open

test(materialization): GC + scheduler state-machine transition coverage#902
girishjeswani wants to merge 3 commits into
malloydata:mainfrom
girishjeswani:test/materialization-gc-coverage

Conversation

@girishjeswani

Copy link
Copy Markdown
Collaborator

Summary

Closes #895 — the GC + state-machine-transition coverage split out of #888. All additive tests plus one documenting comment; no behavior change.

Garbage collection

  • Drop metering: going through deleteMaterialization({ dropTables }) meters recordDropTables("success") on a successful physical-table drop and ("failure") when the drop throws — and the record is still deleted (a best-effort drop failure never surfaces to the caller). Asserted via the OTel in-memory harness.
  • Staging cleanup edges: the original build error propagates when the staging-cleanup drop itself fails (leak logged), and a cancelled mid-run build drops its staging table like a failure — no _staging orphan. (Build-SQL failure was already covered.)
  • Cascade deletes are records-only: deleteByEnvironmentId / deleteByPackage issue a single DELETE and no DDL. A materialized table intentionally outlives its record on env/package delete — physical GC is the caller's responsibility (same contract as deleteMaterialization's default). Locked in with SQL-shape tests + a contract comment on both repo methods, so a future refactor can't silently add a DROP there. This is the "explicit decision, not silent" the feat(materialization): standalone scheduler, environment-scoped listing, and schedule management UI #888 review asked to keep visible; the optional standalone cascade-drop is tracked in Optional: standalone cascade-drop of materialized tables on env/package delete #901.

State-machine transitions (integration, real store)

Complements the per-tick unit guards by driving each transition end to end through REST-created packages the scheduler sweeps:

  • a cron edit between ticks re-arms from now (no stale fire from the old cadence)
  • a package unload/reload prunes arming state and re-anchors without a spurious fire
  • a fired occurrence advances to the next occurrence (fires once per occurrence, not every tick; the advance is outcome-independent, so the FAILED-fire isolation stays unit-covered — a genuine failed build isn't reliably reproducible without a brittle fixture)
  • maxFiresPerTick caps a tick and the capped package fires on the next tick

Each transition test uses its own environment and tears it down in afterEach, since the scheduler sweeps every loaded environment.

Testing

  • Server unit: 1165 pass / 0 fail (incl. drop-metering, staging-edge, records-only repo tests).
  • Materialization integration: 38 pass / 0 fail (incl. the 4 new transition tests).
  • typecheck + eslint clean on changed files.

Notes

  • All commits are DCO signed-off.
  • No production behavior change — coverage + one documenting comment only.

…cords-only cascade deletes

Hardens garbage-collection coverage for the materialization service (fast-follow
to malloydata#888, split out per review as malloydata#895):

- deleteMaterialization telemetry: a successful physical-table drop meters
  outcome=success; a failed drop meters outcome=failure and still deletes the
  record, so a best-effort drop failure never surfaces to the caller.
- buildOneSource staging cleanup: the original build error propagates when the
  staging-cleanup drop itself fails (leak logged), and a cancelled mid-run build
  drops its staging table like a failure — no _staging orphan.
- Cascade deletes are records-only: deleteByEnvironmentId / deleteByPackage issue
  a single DELETE and no DDL, documenting that physical-table GC is the caller's
  responsibility so a materialized table intentionally outlives its record on an
  env/package delete. Contract comment added on both repo methods.

Covers items 1-3 of malloydata#895; the state-machine transition tests (item 4) follow
once malloydata#888's scheduler lands on main.

Signed-off-by: Girish Jeswani <[email protected]>
…ne transitions

Drives the scheduler's transitions end to end against the real store (malloydata#895,
item 4), complementing the per-tick unit guards:

- a cron edit between ticks re-arms from now (no stale fire from the old cadence)
- a package unload/reload prunes arming state and re-anchors without a spurious fire
- a fired occurrence advances to the next occurrence (fires once per occurrence,
  not every tick; the advance is outcome-independent, so the FAILED-fire
  isolation stays unit-covered)
- maxFiresPerTick caps a tick and the capped package fires on the next tick

Each test uses its own environment and tears it down in afterEach, since the
scheduler sweeps every loaded environment.

Signed-off-by: Girish Jeswani <[email protected]>
The integration version drove the prune + re-anchor transition via a package
delete + same-name re-add, which races the package-install staging rename on
Windows (ENOENT on the staging->final rename; POSIX handles it) — unrelated to
the scheduler logic under test. The transition is pure in-memory arming state,
so it's now a deterministic unit test in materialization_scheduler.spec.ts
(mutating the fake loaded set). The other three transitions stay as integration
tests.

Signed-off-by: Girish Jeswani <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Materialization scheduler: GC + integration state-machine transition coverage (fast-follow to #888)

1 participant