plan(signal-store): D45 is a producer seam, not a storage seam#228
Merged
Conversation
Traced to code and data. The six scalars are already declared in feature_schema.py, so compute_feature_schema_sha already covers their names; what it cannot cover is whether a producer ever ran. The export path fills the classifier and association families with a constant 0.0 (_leaf_record_builder._reranker_default_fields, whose docstring says their producers were never wired in), the sealed comparison.json records it as 'zero-filled in export', and a 50,000-row sample of the live blob shows 0.0% zeros. The database is fine; the export invents the zeros. Closing D45 therefore means a failing export (no silent default, a degeneracy check, provenance per family), not a column type. The six typed columns and the ~75 GB blob drop remain worth doing, as a separate storage-motivated step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track B was about to be implemented on a false premise. This corrects the plan before any migration is written.
What the plan said
The six scalars (
classifier_score,self_prior_score,association_cross,association_total,classifier_present,association_present) live in thego_prediction.featuresjsonb outsidefeature_schema_sha, so their values can drift unnoticed; promoting them to typed columns closes D45.What the code and the data say
feature_schema.pydeclares all of them inNUMERIC_FEATURESand in theclassifier,self_priorandassociationfamilies.compute_feature_schema_sha(families, drop)hashes"<family>=<sorted columns>", i.e. the declared schema. It covers the names.0.0.protea/core/_leaf_record_builder.py::_reranker_default_fields(), whose own docstring reads: "theclassifier_*andassociation_*columns stay zero until later lafa-integrate slices wire their producers. A well-defined zero, not NaN." Onlyself_prior_scoreis overwritten, under thecompute_self_priorflag.results/clean_227230/comparison.json:feature_exclusions: "association_* and classifier_* (zero-filled in export)".go_prediction.featuresfinds 0.0% zeros forassociation_totaland 0.0% forclassifier_score. The predict path writes real values; the export builds records from scratch and invents zeros.Consequence
D45 is a producer seam. Three declared families shipped semantically null and every fingerprint matched. Closing it means a failing export, not a column type:
NaN, which LightGBM reads as missing, or refuse the row);SignalConfig.source, and it is why the store is the backend of explainability rather than a disk optimisation.The six typed columns (~2.5 GB) and the resulting
featuresblob drop (~75 GB ofgo_prediction's 101 GB; 54 of its 60 keys duplicate typed columns) remain worth doing, verified against the sealed numbers first, as a separate storage-motivated step under the additive-only invariant.Not verified, and not claimed
No local parquet carries these columns (the bench and reranker-vnext datasets predate the schema). The sealed dataset lives in MinIO and was not read. The three lines of evidence above stand on their own; the parquet would be a fourth.
Files
SIGNAL-STORE.md: new section 1.2 (the seam, traced); section 1 and 1.1 corrected; step 0 rewritten.AUTONOMOUS-RUN.md: Track B preamble and step 0 corrected.