diff --git a/docs/source/adr/D38-neural-head-deferred-dataset-pack-pivot.rst b/docs/source/adr/D38-neural-head-deferred-dataset-pack-pivot.rst index cef6ac58..7020eb2a 100644 --- a/docs/source/adr/D38-neural-head-deferred-dataset-pack-pivot.rst +++ b/docs/source/adr/D38-neural-head-deferred-dataset-pack-pivot.rst @@ -1,10 +1,26 @@ ADR-D38: Defer neural-head champion; pivot to curated dataset packaging ======================================================================= -:Status: Accepted +:Status: Superseded :Date: 2026-05-25 :Author: Francisco Miguel Pérez Canales :Phase: F-DATA-PACK +:Superseded-by: the sealed board (:doc:`/results`), champion ``d8979601`` + at ``f_micro_w`` 0.4063 on the leakage-free v227 to v230 frame (ADR-D40), + scored through the stacked meta-reranker (ADR-D43) + +.. note:: + + Superseded on the champion designation. Decision point 2 below named the + binary-label LightGBM booster (``prostt5``, k=5, NK+LK Fmax 0.7291 on the + v226 frame) as the methodological champion reported in the thesis. That + designation is retired: the sealed champion is the learned k-WTA retrieval + encoder ``d8979601`` at ``f_micro_w`` 0.4063 on the leakage-free v227 to + v230 frame (ADR-D40), followed by the stacked per-category re-ranker + (ADR-D43). See :doc:`/results`. The neural-head DEFERRAL recorded here + (decision point 1) was also overtaken: the learned-encoder direction was + resumed and became that champion. The F-DATA-PACK deliverable (decision + point 3) shipped and stands. The record is retained for history. Context ~~~~~~~ diff --git a/docs/source/adr/D45-jsonb-blob-feature-governance.rst b/docs/source/adr/D45-jsonb-blob-feature-governance.rst index b83a4dde..bfac0d53 100644 --- a/docs/source/adr/D45-jsonb-blob-feature-governance.rst +++ b/docs/source/adr/D45-jsonb-blob-feature-governance.rst @@ -1,169 +1,210 @@ -ADR-D45: Governing the JSONB-blob feature seam -============================================== +ADR-D45: The producer seam in the reranker feature export +========================================================= -:Status: Proposed -:Date: 2026-06-22 +:Status: Accepted +:Date: 2026-07-10 :Author: Francisco Miguel Pérez Canales :Phase: T-GOBERNANZA -Context -~~~~~~~ +.. note:: + + This ADR was drafted on 2026-06-22 with the seam framed as a + train/serve VALUE skew (the incident recorded below). On 2026-07-10 the + mechanism was traced to code and data, and the true shape is narrower and + sharper: it is a PRODUCER seam, not a value skew and not a storage + problem. Three declared feature families shipped semantically null through + the export while every fingerprint matched. This record now states that + verified mechanism and the decisions that follow from it. The + value-skew framing and the warn-only mitigation already shipped for it are + retained below as supporting context, because both the incident and the + code are real. + +Context: the fingerprint pins names, not values or producers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PROTEA fingerprints its reranker feature set and refuses to score with a -booster whose features have drifted. The canonical feature set is declared -in ``protea-contracts`` (``src/protea_contracts/feature_schema.py``: -``ALL_FEATURES``, ``FEATURE_FAMILIES``) and fingerprinted by -``compute_feature_schema_sha(families, drop)``. At inference the scorer -recomputes the live sha from the active families and raises -``SchemaShaMismatchError`` when it disagrees with the booster's recorded -``RerankerModel.feature_schema_sha`` (see -``protea/core/operations/predict_go_terms/_reranker_scorer.py``, the -``_check_schema_sha`` path around lines 290 to 370, and the families chosen -by ``protea_method.reranker.infer_active_feature_families``). This is the -**governed surface**: a rename, reorder, or addition of a contract feature -forces a SemVer major bump on ``protea-contracts`` and is caught at serve. - -There is a second feature surface that is NOT governed by that sha. Four -families ride the ``GOPrediction.features`` JSONB blob -(``protea/infrastructure/orm/models/embedding/go_prediction.py``, -``features`` column) and are computed post-KNN, each behind its own -``compute_*`` payload flag, in -``protea/core/operations/predict_go_terms/_post_knn_pipeline.py``: - -- **classifier** (``compute_classifier`` -> ``apply_classifier``), -- **self_prior** (``compute_self_prior`` -> ``apply_self_prior``), -- **association** (``compute_association`` -> ``apply_association``), -- **IA / information accretion** (``compute_ia`` -> ``apply_ia``). - -These producers and their flags live in PROTEA core and protea-method -(``infer_active_feature_families`` only knows the contract families: knn, -alignment, taxonomy, anc2vec, emb_pca, go_context, annotation_meta). The -four blob families are never named in ``feature_schema.py``, so they do not -enter ``compute_feature_schema_sha`` and the guard is structurally blind to -them. A change to their VALUES (a producer that was a no-op at train time -becoming populated at serve time, a config change in the producer, a -different IA file) does not change any sha and trips no check. - -The risk is train/serve **value** skew, not schema skew. The schema-sha -guard verifies that the same column NAMES are present; it cannot verify that -the same column VALUES were produced by the same producers under the same -configuration. The blob seam is exactly where that invariant is unenforced. - -Incident (the concrete evidence) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The night of 2026-06-21 an INT-8 native re-run regressed the reranked LAFA -``f_micro_w`` from 0.3745 to 0.3462. The booster trio had been trained when -the ``association`` producer was effectively zero (the cooccurrence the -producer reads was not built for the training t0 sets, so the feature was a -near-constant column at train time). The serve-time run populated -``association`` with real values. The booster therefore consumed an -out-of-distribution column it had learned to ignore, and the score -collapsed by roughly 0.028. The column NAME was unchanged, so -``feature_schema_sha`` matched and the guard passed silently. This is a -textbook train/serve value skew through the ungoverned JSONB-blob seam, of -the same shape as the earlier 0.315 collapse, but invisible to the one -mechanism PROTEA has for catching reranker drift. +booster whose feature schema has drifted. The fingerprint is computed in +``protea-contracts`` (``src/protea_contracts/feature_schema.py``): + +- ``compute_schema_sha(columns)`` (line 253) hashes ``"|".join(sorted(columns))``, + UTF-8 encoded, through ``short_sha`` (``_hashing.py`` line 29, which returns + ``sha256(blob).hexdigest()[:12]``). +- ``compute_feature_schema_sha(families, drop)`` (line 273) builds one part + per family as ``f"{fam}={','.join(sorted(cols))}"`` (lines 292 to 294), + joins the parts with ``"|"``, and hashes that (lines 297 to 298). + +Both digests are functions of column NAMES and, for the family-aware digest, +family membership. Neither can see a value, which producer wrote a column, or +whether a producer ran at all. Identical names plus identical family +membership yield an identical sha regardless of what was actually written into +the cells. + +The six LAFA signals are already declared in the same file. In +``NUMERIC_FEATURES`` they appear as ``classifier_score`` (line 113), +``classifier_present`` (line 114), ``self_prior_score`` (line 116), +``association_total`` (line 119), ``association_cross`` (line 120) and +``association_present`` (line 121). In ``FEATURE_FAMILIES`` they are grouped +as ``classifier`` (line 232), ``self_prior`` (line 233) and ``association`` +(line 234). They are first-class members of the canonical schema. A booster +that selects those families passes the schema-sha guard by construction. + +The export writes a well-defined zero, not the truth +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The leaf-record builder that materialises each ``(query, candidate-GO)`` row +fills all six columns with a constant ``0.0``. The static method +``_lafa_default_fields()`` in ``protea/core/_leaf_record_builder.py`` (defined +at line 342) returns: + +.. code-block:: python + + return { + "classifier_score": 0.0, + "classifier_present": 0.0, + "self_prior_score": 0.0, + "association_total": 0.0, + "association_cross": 0.0, + "association_present": 0.0, + } + +Its own docstring states the intent plainly. ``self_prior_score`` is +overwritten by the native compute when the ``compute_self_prior`` payload flag +is set, and, in the docstring's words, "the ``classifier_*`` and +``association_*`` columns stay zero until later lafa-integrate slices wire +their producers. A well-defined zero, not NaN, matching the lineage +convention." The producers for the ``classifier`` and ``association`` families +were never wired into the export path, so those five columns are emitted as a +constant ``0.0`` for every row of every shard. + +(Note on naming: the six columns are filled by ``_lafa_default_fields()``. A +separate method ``_reranker_default_fields()`` at line 34 of the same file +fills the KNN reranker block for non-KNN rows and does NOT touch the six LAFA +columns. Earlier prose that attributed the zero-fill to +``_reranker_default_fields()`` named the wrong method.) + +The sealed run recorded the consequence. The frozen comparison for the +consolidated champion, +``protea-reranker-lab/results/clean_227230/comparison.json``, carries +``feature_exclusions`` = "association_* and classifier_* (zero-filled in +export) + id/label/category/aspect/snapshot_pair/qualifier/evidence_code/ +taxonomic_relation". The lab excluded the two families precisely because the +export shipped them as constant zeros. + +The live database is not the problem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This is a producer seam in the EXPORT, not a storage defect. A 50,000-row +sample of ``go_prediction.features`` taken on 2026-07-10 shows 0.0% zeros for +``association_total`` and 0.0% zeros for ``classifier_score``: the production +prediction path writes real values into the JSONB blob +(``protea/infrastructure/orm/models/embedding/go_prediction.py``, the +``features`` column at line 115). The seam is that the frozen research dataset, +the artefact the lab trains on, receives a well-defined zero where the live +row holds a real value. The fingerprint matched throughout, so nothing failed. + +Honest scope: the sealed number is intact +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The sealed champion (``f_micro_w`` 0.4063, frame v227 to v230, +config ``d8979601``) was trained WITHOUT the ``classifier`` and +``association`` families, exactly because they were zero-filled and excluded. +The seam therefore did not corrupt the sealed number. What it did is quieter +and worth stating without softening: it silently removed three declared +feature families from the model's reach. The champion was chosen from a +feature space three families smaller than the schema advertised, and no check +ever flagged the absence. Decision ~~~~~~~~ -Do not force the four blob families into ``ALL_FEATURES`` / -``compute_feature_schema_sha``. That would over-couple: the contract sha is -a SemVer-load-bearing fingerprint of the lab parquet column set, and the -blob families are produced inside PROTEA core (not by the lab), are gated -per run, and legitimately vary in presence across runs. Folding them in -would force a contracts major bump and a full booster re-train on every -producer config change, and it still would not capture a value change with a -stable config. The right unit of governance for these is **value -provenance**, not schema identity. - -Adopt instead a documented boundary plus a lightweight provenance marker and -a serve-time check: - -1. **Name the boundary (this ADR).** The governed surface is the contract - ``ALL_FEATURES`` via ``feature_schema_sha``. The ungoverned surface is the - ``GOPrediction.features`` blob families (classifier, self_prior, - association, IA). State this in the data-model docs next to the - ``GOPrediction.features`` column and next to ``RerankerModel`` so the seam - is visible to anyone training or registering a booster. - -2. **Record blob-feature provenance on the training artifact.** When a - ``Dataset`` is exported and a ``RerankerModel`` is registered, record - which blob-feature producers were active and the config they ran under - (the set of ``compute_*`` flags that were on, plus a content marker for - each producer's inputs: for ``association`` the cooccurrence build keyed - to the training t0 sets, for ``IA`` the IA file identity, for self_prior - and classifier the producer revision). This is a small JSON marker on the - training side (a ``blob_feature_provenance`` field on ``Dataset`` / - ``RerankerModel``, paralleling the existing ``feature_schema_sha`` / - ``external_source`` provenance), not a new sha in the contract. - -3. **Warn or refuse on provenance mismatch at serve.** At inference, the - scorer compares the serve-time active blob producers and their config - marker against the booster's recorded ``blob_feature_provenance``. On a - mismatch (for example: ``association`` was a no-op at train time but is - populated now), emit a structured warning event and, behind a strict - flag, refuse with a dedicated error code, mirroring the - ``SchemaShaMismatchError`` path but for VALUE provenance rather than - schema identity. This catches the INT-8 class of regression at registration - or serve, where the schema-sha guard cannot. - -Keep step 1 mandatory and immediate (documentation, this ADR). Steps 2 and 3 -are the proposed mechanism; they are deliberately scoped to a provenance -marker and a comparison, with no change to the contract sha and no forced -booster re-train. +D45 is a PRODUCER seam. Three declared feature families shipped semantically +null through the export while every fingerprint matched. The governance that +follows treats a produced value, its absence, and its provenance as +first-class, rather than trusting name-level identity to imply value-level +truth. + +1. **No silent default.** When no producer ran for a declared column, emit + ``NaN`` (LightGBM already reads ``NaN`` as missing) or refuse to write the + row. A zero is a claim ("this signal was measured and it is zero"); missing + is the truth ("no producer measured this"). The current + ``_lafa_default_fields()`` zero convention makes an absent producer + indistinguishable from a genuine zero, and that is the exact confusion the + seam exploits. + +2. **A degeneracy check at export.** A declared feature family that is + constant across a shard fails the export job loudly instead of shipping. A + family that is present in the schema but never varies is, by definition, + carrying no information, and that condition must be an error at write time, + not a discovery made months later when a comparison file happens to record + it. + +3. **Provenance per family.** Which producer, at which version, over which + snapshot, recorded alongside the dataset. This is the ``SignalConfig.source`` + of the planned signal store: the family's value provenance travels with the + artefact so a consumer can see not only that a column is present but who + produced it and under what inputs. + +What D45 is NOT +~~~~~~~~~~~~~~~ +Promoting the six columns from the JSONB blob to typed columns is a separate, +additive change (on the order of 2.5 GB) that would let the redundant +``features`` JSONB be dropped in a reviewed follow-up step. The blob currently +holds roughly 75 GB of ``go_prediction``'s 101 GB, and about 54 of its 60 keys +duplicate typed columns on the same row (the model mirrors typed columns into +the blob; see the ``build_feature_jsonb`` note next to the ``features`` +column). Typing the columns and dropping the duplicated blob is a real storage +win. It does NOT close D45. A typed ``classifier_score`` column with no +producer wired into the export would still be filled with a default and would +still ship semantically null under a matching fingerprint. D45 is closed by +the three decisions above (no silent default, degeneracy check, per-family +provenance), not by a storage migration. + +Supporting context: the value-skew incident and the shipped mitigation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The same seam produced a train/serve VALUE skew earlier in the campaign. On +2026-06-21 an INT-8 native re-run regressed the reranked LAFA ``f_micro_w`` +from 0.3745 to 0.3462: a booster trained while the ``association`` producer was +effectively a constant column was then served against real ``association`` +values. The column names were unchanged, so ``feature_schema_sha`` matched and +the guard passed silently. That incident and this export producer seam are two +faces of the one invariant the name-level fingerprint cannot enforce: that the +same producers, at the same configuration, filled the same columns. + +A conservative, warn-only mitigation for the value-skew face already ships: + +- ``protea.core.operations.predict_go_terms._blob_provenance`` computes a + stable value-provenance descriptor and a 12-hex digest over the blob families + from the live predict payload (which producers are active plus reachable + config markers, plus a per-family ``BLOB_PRODUCER_VERSIONS`` constant). It is + the value-provenance counterpart to ``feature_schema_sha``. +- ``RerankerScorer.record_blob_provenance`` (in + ``protea/core/operations/predict_go_terms/_reranker_scorer.py``) runs after + the schema-sha guard. It always emits a ``reranker.blob_provenance`` event, + and, when the payload carries an expected provenance that disagrees with the + live one, emits a loud ``reranker.blob_provenance_mismatch`` warning and + proceeds (the expected marker is nullable, so it never refuses). Scoring + numbers are byte-identical when provenance matches or is absent. + +That mitigation makes the seam OBSERVABLE at serve. The three decisions in this +ADR make it a hard error at EXPORT, which is where the three families were lost. Consequences ~~~~~~~~~~~~~ -- The two feature surfaces are named and the asymmetry is no longer - implicit: schema identity is governed by ``feature_schema_sha``; blob - value provenance is governed by the marker. -- A booster trained against a zero (or differently configured) blob - producer can no longer be served silently against a populated one; the - INT-8 0.3745 to 0.3462 regression would have surfaced as a provenance - warning (or a refusal under strict mode) rather than a silent score - collapse. -- The contract sha stays a clean fingerprint of the lab parquet column set; - no over-coupling, no forced contracts major bump for a PROTEA-side producer - change, no full re-train on every IA-file or cooccurrence rebuild. -- The provenance marker is a small additive field on existing rows - (``Dataset`` / ``RerankerModel``); no migration of the prediction hot path, - and legacy rows simply carry a null marker (treated as unknown, warn-only). +- A declared feature family can no longer ship as a constant default while a + matching fingerprint reports health. The export fails loudly (decision 2), + or writes ``NaN`` the booster reads as missing (decision 1). +- The absence of a producer is now recoverable from the artefact itself + (decision 3): the dataset records who produced each family and under what + inputs, so a stranger auditing the frozen dataset can tell a measured zero + from an unwired producer. +- The sealed 0.4063 champion is unaffected: it was trained without the two + zero-filled families, so re-homing them behind real producers is an additive + expansion of the feature space, not a correction to the sealed number. +- The storage question (typed columns, dropping the ~75 GB duplicated blob) is + decoupled from the governance question and can proceed on its own reviewed + schedule without being mistaken for a fix. Resolution ~~~~~~~~~~ -Open (partially implemented). This ADR records the boundary and proposes the -provenance marker plus the serve-time check. The governed surface -(``feature_schema_sha`` in ``feature_schema.py`` and ``_reranker_scorer.py``) -and the ungoverned blob producers (``_post_knn_pipeline.py``) both exist -today; D45 names the seam between them and the incident that proves it is -load-bearing. - -A conservative, warn-only increment of the mechanism is now in place: - -- ``protea.core.operations.predict_go_terms._blob_provenance`` computes a - stable value-provenance descriptor and a 12-hex digest over the four blob - families from the live predict payload (which producers are active plus - their reachable config markers, e.g. ``ia_file``, and a per-family - ``BLOB_PRODUCER_VERSIONS`` constant). It is the value-provenance - counterpart to ``feature_schema_sha``. -- ``RerankerScorer.record_blob_provenance`` (in ``_reranker_scorer.py``) runs - after the schema-sha guard on both the single and per-category scoring - paths. It always emits a ``reranker.blob_provenance`` event recording the - live digest and descriptor (the seam is now observable in the Job event - log), and, when the payload carries an expected provenance that disagrees - with the live one, emits a LOUD ``reranker.blob_provenance_mismatch`` - warning (structured event + stdlib log, ``reason=blob_provenance_mismatch``) - and PROCEEDS. It never raises: unlike the schema-sha guard the expected - marker is nullable, so a hard refuse would take serving down. Warn-only - matches the null-marker decision above. Scoring numbers are byte-identical - when provenance matches or is absent (the guard only adds observability). - -Deferred to the D45 follow-up (the part that closes the loop automatically): -record ``blob_feature_provenance`` on the training artifact -(``Dataset`` / ``RerankerModel``) at registration with a content marker for -each producer's inputs (the ``association`` cooccurrence build keyed to the -training t0 sets, the IA file identity, producer revisions), and carry it to -the batch payload (``reranker_blob_feature_provenance``) so the comparison -above lights up against a real recorded value rather than a null. That step -touches ``protea-contracts`` (the payload field) and the booster-registration -flow, so it is scoped separately; the descriptor, the recording, and the -tested warn-on-mismatch seam it plugs into land here. +Open (partially implemented). The mechanism is verified against code and data +as of 2026-07-10. The warn-only blob-provenance observability described above +is in place. The three decisions (no silent default, export degeneracy check, +per-family provenance via the signal store's ``SignalConfig.source``) are the +governance this ADR adopts; they are additive and land as the signal store is +built. Until they land, treat any ``classifier`` or ``association`` column read +from a legacy frozen dataset as unwired-in-export, not as a genuine zero. diff --git a/docs/source/adr/index.rst b/docs/source/adr/index.rst index c7ada4d1..c5235d3e 100644 --- a/docs/source/adr/index.rst +++ b/docs/source/adr/index.rst @@ -7,20 +7,23 @@ shows the what. ADRs come in two layers: -- **Implementation decisions** (numbered ``001``-``008``): runtime, +- **Implementation decisions** (numbered ``001``-``009``): runtime, data model and operational choices discovered while building PROTEA. They explain trade-offs of concrete code paths (KNN algorithm choice, queue topology, deduplication strategy, retries, etc.). -- **Strategic decisions** (``D1``-``D31``): plan-level decisions - taken in the master plan revision 3 (2026-05-05). They drive the structure - of the project, the deployment story, and the thesis writing - cadence. +- **Strategic decisions** (``D1``-``D45``): plan-level decisions. The + ``D1``-``D31`` block was taken in the master plan revision 3 (2026-05-05) + and drives the structure of the project, the deployment story, and the + thesis writing cadence. Later records (``D34`` onward) were added as the + work progressed and carry their own dates and phases in the table below. + This ``D``-series is a PROTEA-local ledger (see the cross-repo note under + Strategic decisions). Implementation decisions ------------------------ -All implementation ADRs (001-008) follow the MADR template (Status / Context / +All implementation ADRs (001-009) follow the MADR template (Status / Context / Decision / Consequences sections). They are numbered in discovery order, not superseded by the D-series, and remain the authoritative record for the runtime, data model, and operational choices described. @@ -73,10 +76,28 @@ runtime, data model, and operational choices described. Strategic decisions ------------------- -Decisions taken in the master plan revision 3 (2026-05-05). Statuses: +Decisions taken in the master plan revision 3 (2026-05-05), plus later +records added as the work progressed. Statuses: *Accepted*, *Pending* (gate opens at the indicated phase), *Deferred* -(scheduled later in the timeline) or *Obsolete* (superseded by a -later revision). +(scheduled later in the timeline), *Obsolete* (superseded by a +later revision) or *Superseded* (a named later decision replaces it). + +.. admonition:: Cross-repo D-number scope (disambiguation) + :class: important + + This ``D``-series is the PROTEA-local ledger. ``protea-reranker-lab`` + keeps its own independent ``D``-series, and the numbers are NOT a shared + namespace. ``D34`` (selective rerank resurrection) is deliberately the + same decision in both repositories, cross-referenced. But ``D39``, + ``D40`` and ``D41`` collide: in this ledger they are destructive-op + guards, the leakage-free temporal evaluation protocol, and the lean f32 + reference pool; in ``protea-reranker-lab`` the same three numbers denote + FAIR dataset packaging, IA-aligned training, and the universal booster + pipeline. These are not renumbered here because PROTEA ``D39``/``D40``/ + ``D41`` are already cited across the docs, memory, and thesis; a rename + would break those citations for no gain. Read a bare ``D39``/``D40``/ + ``D41`` as the PROTEA decision; qualify it as ``lab-D39`` and so on when + the intended record lives in ``protea-reranker-lab``. .. list-table:: :header-rows: 1 @@ -228,8 +249,8 @@ later revision). - F-AUTH (complete, FARM-AUTH.1-11) * - D38 - :doc:`Defer neural-head champion; pivot to curated dataset packaging (F-DATA-PACK) ` - - Accepted - - F-DATA-PACK + - Superseded + - F-DATA-PACK (champion designation superseded; see :doc:`/results`) * - D39 - :doc:`Defense-in-depth guards on destructive database operations ` - Accepted @@ -255,8 +276,8 @@ later revision). - Proposed - T-PRODUCTO * - D45 - - :doc:`Governing the JSONB-blob feature seam ` - - Proposed + - :doc:`The producer seam in the reranker feature export ` + - Accepted - T-GOBERNANZA .. toctree::