docs(spec): language-neutral behavioral spec for the pipes pipeline

[mo4islona]

What

A full conformance-tier spec suite at spec/ — the language-neutral contract for the pipeline, written against the current TypeScript implementation as reference. Intended to let future implementations (Rust first) be built and verified against the same properties and the same persistent/wire formats, so implementations are interchangeable mid-stream.

Contents

• Wire & format bindings (14): portal protocol (NDJSON stream, 409 fork signal, head headers), all four cursor-state formats (ClickHouse/Postgres/Parquet/BigQuery), cache format, observability HTTP surface, closed error-code registry.
• Sink contract (05, 06): four durability classes (transactional / write-ahead / checkpointed-immutable / append-lagged) with per-class commit protocols, crash windows, and recovery obligations.
• Safety & liveness (07, 08, 09): 29 safety invariants + 10 liveness properties (incl. single-slot ingest prefetch, PF-6/WP-12), fault-family response tables.
• Conformance plan (13): reference-model oracle, 9 test classes with a kill-point matrix, honest traceability matrix from the current vitest inventory, and a dated gap register.
• Decision log: 11 backfilled historical ADRs + 4 proposed ones awaiting ratification (decode-error hook unification, spec location, SLO targets, mandatory class-A repair hook).

Review pass (2026-07-19)

An 11-agent spec-vs-implementation audit verified every wire/format/parameter claim against the code and drove a revision:

• Normative text corrected where it contradicted verified behavior: the deep-fork ancestor rule (WP-42 prose + reference-model pseudocode now match the tested resolveForkCursor semantics, including the previously undocumented floor fallback), per-network cursor timestamp units (tron reports milliseconds), head-only (204) handling, the class-T undo mechanism (after-image trigger, not before-image), and body-stall retryability.
• Parquet coverage model scoped honestly: coverage naming / coverage state map / straddle refusal are marked contract-target (they live in unmerged PR fix(parquet): name files for the block range they cover #123); the traceability matrix is re-pinned to this branch's mainline base a739500, and the coverage row corrected from C to U ⚠ (GAP-17).
• Gap register grown 16 → 34 (8 entries at P1): new entries for the class-T undo holes (GAP-18/19), orphan guards missing outside BigQuery (GAP-20), ClickHouse fork not being a commit point (GAP-21), per-segment lifecycle re-fires on fork (GAP-22), six unbound/unimplemented observability signals (GAP-28), cache codec discrimination (GAP-27), and the open fix PRs fix(runner): respect retry: 0 instead of coercing to default 5 #70/fix(portal-source): end batch spans on empty batches and stream end #71/fix(portal-source): make pipe() id dedup stable for 3+ same-id transformers #73 (GAP-32/33), among others. GAP-10 widened to name both wall-clock-ordered bindings; GAP-11 upgraded from suspected to confirmed (P1).
• Self-consistency fixes: OB↔IB binding annotations, WP-3 vs DEF-9 legacy-key model, RP-31 scoped to migrating bindings, stale GAP citations, OQ-6/OQ-7 added, ADR-14 linked to OQ-4.

Coverage gate

Docs-only change — no runtime code touched, so there is no coverage diff to report. The spec's consistency checker is now committed (spec/check-spec.mjs) and CI-enforced on spec changes (.github/workflows/spec.yml): 428 IDs, 0 dangling references.

🤖 Generated with Claude Code