Skip to content

plan(signal-store): D45 is a producer seam, not a storage seam#228

Merged
frapercan merged 1 commit into
mainfrom
fix/d45-producer-seam
Jul 10, 2026
Merged

plan(signal-store): D45 is a producer seam, not a storage seam#228
frapercan merged 1 commit into
mainfrom
fix/d45-producer-seam

Conversation

@frapercan

Copy link
Copy Markdown
Owner

Track B was about to be implemented on a false premise. This corrects the plan before any migration is written.

What the plan said

The six scalars (classifier_score, self_prior_score, association_cross, association_total, classifier_present, association_present) live in the go_prediction.features jsonb outside feature_schema_sha, so their values can drift unnoticed; promoting them to typed columns closes D45.

What the code and the data say

  1. The six are already inside the fingerprint. feature_schema.py declares all of them in NUMERIC_FEATURES and in the classifier, self_prior and association families. compute_feature_schema_sha(families, drop) hashes "<family>=<sorted columns>", i.e. the declared schema. It covers the names.
  2. What it cannot cover is whether a producer ran. A family whose columns are all present and all zero produces exactly the same sha.
  3. The export fills three declared families with a constant 0.0. protea/core/_leaf_record_builder.py::_reranker_default_fields(), whose own docstring reads: "the classifier_* and association_* columns stay zero until later lafa-integrate slices wire their producers. A well-defined zero, not NaN." Only self_prior_score is overwritten, under the compute_self_prior flag.
  4. The sealed run recorded the consequence. results/clean_227230/comparison.json: feature_exclusions: "association_* and classifier_* (zero-filled in export)".
  5. The database is fine. A 50,000-row sample of go_prediction.features finds 0.0% zeros for association_total and 0.0% for classifier_score. The predict path writes real values; the export builds records from scratch and invents zeros.

Consequence

D45 is a producer seam. Three declared families shipped semantically null and every fingerprint matched. Closing it means a failing export, not a column type:

  • no silent default (emit NaN, which LightGBM reads as missing, or refuse the row);
  • a degeneracy check at export (a declared family constant across a shard fails the job);
  • provenance per family (which producer, at which version, over which snapshot). That is SignalConfig.source, and it is why the store is the backend of explainability rather than a disk optimisation.

The six typed columns (~2.5 GB) and the resulting features blob drop (~75 GB of go_prediction's 101 GB; 54 of its 60 keys duplicate typed columns) remain worth doing, verified against the sealed numbers first, as a separate storage-motivated step under the additive-only invariant.

Not verified, and not claimed

No local parquet carries these columns (the bench and reranker-vnext datasets predate the schema). The sealed dataset lives in MinIO and was not read. The three lines of evidence above stand on their own; the parquet would be a fourth.

Files

  • SIGNAL-STORE.md: new section 1.2 (the seam, traced); section 1 and 1.1 corrected; step 0 rewritten.
  • AUTONOMOUS-RUN.md: Track B preamble and step 0 corrected.

Traced to code and data. The six scalars are already declared in
feature_schema.py, so compute_feature_schema_sha already covers their
names; what it cannot cover is whether a producer ever ran. The export
path fills the classifier and association families with a constant 0.0
(_leaf_record_builder._reranker_default_fields, whose docstring says
their producers were never wired in), the sealed comparison.json records
it as 'zero-filled in export', and a 50,000-row sample of the live blob
shows 0.0% zeros. The database is fine; the export invents the zeros.

Closing D45 therefore means a failing export (no silent default, a
degeneracy check, provenance per family), not a column type. The six
typed columns and the ~75 GB blob drop remain worth doing, as a separate
storage-motivated step.
@frapercan frapercan merged commit 05d99f4 into main Jul 10, 2026
7 checks passed
@frapercan frapercan deleted the fix/d45-producer-seam branch July 10, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant