Add the static scorer, which answers only where the data can (M12, ADR-015)

Precomputes the flagship model's decision surface into results/scorer_grid.json;
the page picks a cell. No server, no inference at request time.

The naive version of this idea lies, and finding that out reshaped it. The raw
grid is 60% empty -- 258 of 432 iso x resource x service combos have zero
projects -- and the emptiest cells score HIGHEST. The first build gave
"ERCOT | Nuclear | Other | 5 MW" a 99.1% chance of being built off zero
projects: the model extrapolating a one-hot combination it never saw jointly.
Shipping that would be the shareable-but-fictional artifact ADR-013 refuses,
sitting on top of the repo whose whole pitch is refusing them.

So a cell carries a score only where the data supports one -- 957 of 6,480.
Extrapolation turned out to have two dimensions, and fixing one left the other:
with the categorical floor in place the model still answered "5 MW coal plant in
ERCOT" with 89.6%, when real ERCOT coal runs a 660 MW median and is 45.9% built.
It had learned "smaller completes more" and was asked about a size that does not
exist. Each combo now also carries the size range it actually spans (p5-p95), and
out-of-range sizes are unscored. The two rejections say different things -- "no
projects like this" vs "none this size; they run 17-885 MW" -- because the
difference is the interesting part.

Also fixed while building it:
- service NA became an explicit "Not specified" level. ERCOT and NYISO report
  service for ZERO projects, so requiring a value silently dropped two whole ISOs
  -- the same reporting-artifact trap as ADR-012's developer field.
- base rate was computed on all cohorts (11.2%) where the published figure is the
  eval cohorts' (8.3%). Same statistic, different population: ADR-014 again. It
  now reads the committed blob.

Honesty constraints, all tested: completion only (no P(withdraw) control ever --
AUC 0.58 is a coin flip dressed as a number); is_hybrid derived from the resource,
never picked (verified 100% agreement, 0 mismatches); congestion pinned at each
ISO's median and published; no aggregate over cells, since the grid is unweighted.

Gotcha worth the comment it got: p:null came from a pandas float column, which
stores None as NaN -- and json.dumps emits NaN, which is invalid JSON. The live
page died at parse time while all tests passed, because Python's json.loads
ACCEPTS NaN. Round-tripping cannot catch it; both writers now use allow_nan=False.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Author: owgreen-dev

Makefile

@@ -1,4 +1,4 @@
-.PHONY: setup lint test demo benchmark viz viz-assets
+.PHONY: setup lint test demo benchmark viz viz-assets scorer
 
 # All targets run offline and keyless against committed sample fixtures.
 
@@ -23,3 +23,6 @@ viz:  ## render the figures to build/queue.html (set GRIDQUEUE_LBNL_PATH for ful
 
 viz-assets:  ## regenerate the committed README figures (needs: uv sync --extra assets)
 	uv run python -m gridqueue.viz.export
+
+scorer:  ## precompute the model decision surface (set GRIDQUEUE_LBNL_PATH for full data)
+	uv run python -m gridqueue.viz.scorer

docs/assets/drift-composition.png

@@ -345,3 +345,49 @@ explicit `require_fips` switch rather than filtering unconditionally.
 **Guard.** `tests/unify/test_resources.py` asserts the two publishers agree at the precision the repo
 actually prints (4dp — `build_headline` rounds for the payload, so a tighter tolerance fails on the
 rounding, not on a disagreement). Any future battery-drift consumer imports the same fold.
+
+---
+
+## ADR-015 — The scorer answers only where the data can answer
+
+**Decision.** `viz/scorer.py` precomputes the flagship model's decision surface into
+`results/scorer_grid.json`; the page picks a cell. No server, no inference at request time. **A cell
+carries a score only where the data supports one** — 957 of 6,480 cells. The rest are emitted with
+`p: null` and the page says *why*.
+
+**Why the floor exists.** The naive version of this idea lies. The raw grid is **60% empty** (258 of
+432 `iso × resource × service` combos have zero projects; 72% have <10), and **the emptiest cells
+score highest**: the first build gave `ERCOT | Nuclear | Other | 5 MW` a **99.1%** chance of being
+built off **zero** projects — the model extrapolating a one-hot combination it never saw jointly. A
+picker shipping that would be the shareable-but-fictional artifact ADR-013 refuses, sitting on top of
+the repo whose whole pitch is refusing them.
+
+**Extrapolation has two dimensions, and fixing one leaves the other.** The categorical floor (n ≥ 30)
+still let the model answer *"5 MW coal plant in ERCOT"* with **89.6%** — real ERCOT coal runs a
+**660 MW median** and is 45.9% built; the model had learned "smaller completes more" and was asked
+about a size that does not exist. So each combo also carries the size range it actually spans
+(p5–p95, not min/max, so one outlier cannot widen it), and out-of-range sizes are unscored too. The
+two rejections carry **different messages** — "no projects like this" vs "none this size, they run
+17–885 MW" — because the difference is the interesting part.
+
+**Consequences.**
+- **It is the benchmarked model, not a refit** — same features and same train cohorts as
+  `benchmark/run.py`, so the surface explored is the one whose AUC 0.72 is published (ADR-014's
+  one-source rule). A test spot-checks that cells *are* `score_entry`'s output, not an approximation.
+- **`is_hybrid` is derived, never picked** — it is `type_2.notna()`, exactly `"+" in resource_type`
+  (verified: 100% agreement, 0 mismatches on 38k rows). A free control would generate impossible
+  states.
+- **Congestion is pinned, never picked** — `log_mw_ahead` is "MW ahead of you in your ISO on filing
+  day", computed from the whole queue; a filer cannot supply it. Pinned at each ISO's median, and the
+  pinned values are published so the assumption is inspectable.
+- **`service` gained an explicit "Not specified" level.** ERCOT and NYISO report it for **zero**
+  projects, so requiring a value silently dropped two ISOs — the same reporting-artifact trap as
+  ADR-012's developer field. NA is a state the model saw (`_design` encodes unknowns as all-zeros).
+- **No withdrawal control, ever** (AUC 0.58 — a coin flip dressed as a number), and no aggregate over
+  cells: the grid is unweighted (as many nuclear cells as solar, though solar has 13,479 projects and
+  nuclear 211), so a "median cell" would be meaningless. The comparator is the published base rate.
+
+**Gotcha.** `p: null` came from a pandas float column, which stores None as **NaN** — and
+`json.dumps` emits `NaN`, which is **invalid JSON**. The live page died at parse time while every
+test passed, because Python's `json.loads` *accepts* NaN. It cannot be caught by round-tripping; both
+writers now pass `allow_nan=False` and refuse it at build time.

docs/assets/null-decile-line.png

@@ -52,6 +52,8 @@
 RESULTS = ROOT / "results"
 BENCHMARK_FULL = RESULTS / "benchmark_full.json"
 FIGURES_FULL = RESULTS / "figures_full.json"
+# The model's precomputed decision surface for the static scorer (ADR-015).
+SCORER_GRID = RESULTS / "scorer_grid.json"
 
 # The through-year of the LBNL edition backing the labels. Drives the
 # horizon-observability gate (a cohort is eligible only if cohort + H <= this).

docs/decisions.md

@@ -56,7 +56,10 @@ def render(payload: dict) -> str:
     tpl = TEMPLATE.read_text(encoding="utf-8")
     if PLACEHOLDER not in tpl:
         raise ValueError(f"template is missing the {PLACEHOLDER} placeholder")
-    blob = json.dumps(payload, separators=(",", ":"))
+    # allow_nan=False: json.dumps happily emits NaN, which is invalid JSON and kills
+    # the page at parse time. Python's json.loads accepts it, so this cannot be caught
+    # by round-tripping in a test - it has to be refused at write time.
+    blob = json.dumps(payload, separators=(",", ":"), allow_nan=False)
     # </script> inside the JSON would close the host <script> tag early.
     blob = blob.replace("</", "<\\/")
     return tpl.replace(PLACEHOLDER, blob)
@@ -100,7 +103,9 @@ def main() -> None:
     if args.emit_figures:
         fig_path = Path(args.emit_figures)
         fig_path.parent.mkdir(parents=True, exist_ok=True)
-        fig_path.write_text(json.dumps(split_figures(payload), separators=(",", ":")))
+        fig_path.write_text(
+            json.dumps(split_figures(payload), separators=(",", ":"), allow_nan=False)
+        )
         print(f"wrote {fig_path} ({fig_path.stat().st_size / 1e3:.0f} KB, geometry excluded)")
 
     html = render(payload)

results/figures_full.json

@@ -199,6 +199,17 @@ def build_flagship() -> dict:
     }
 
 
+def build_scorer_card() -> dict:
+    """The precomputed scorer grid, if it has been built (`make scorer`).
+
+    Read rather than computed here for the same reason as build_flagship: it needs a model
+    fit, and `make viz` is a 2-second render. Absent -> the card stays hidden.
+    """
+    if not paths.SCORER_GRID.exists():
+        return {}
+    return json.loads(paths.SCORER_GRID.read_text())
+
+
 def build_payload(df: pd.DataFrame | None = None, source_label: str = "") -> dict:
     """Assemble the full figure payload. ``df`` defaults to the canonical loader."""
     from gridqueue.viz.geo import load_county_paths
@@ -212,6 +223,7 @@ def build_payload(df: pd.DataFrame | None = None, source_label: str = "") -> dic
     payload.update(build_drift(prepared))
     payload["headline"] = build_headline(all_entries)
     payload["flagship"] = build_flagship()
+    payload["scorer"] = build_scorer_card()
     payload.update(build_null(prepared))
     payload["meta"] = {
         "source": source_label or str(paths.LBNL_SAMPLE),