gate smoothing on a leave's interval, not a count#99
Merged
Conversation
Smoothing exists to invent a value for a leave too thinly sampled to trust its own average. Deciding which those are is currently a bare count -- fewer than fifty samples and the leave is smoothed -- and fifty is a guess. It is also the wrong question: fifty samples that agree closely pin an average down, fifty that disagree wildly do not, and no count can tell the two apart. Work out the spread instead. WOLGES_GENERATE_SMOOTH_CI names a width, and a leave whose average sits in an interval wider than that is smoothed however many samples it has -- the same precision test the census already applies to its own leaves. The interval TIGHTENS the sample floor rather than replacing it -- both tests must pass. On its own it would be unsound: two samples that happen to agree spread by zero, so the average would look perfectly pinned when two samples say almost nothing. Agreement by luck is exactly the noise smoothing exists to absorb, so the floor stays and the interval only ever smooths more. The spread divides by n-1, matching stats::variance, since dividing by n understates how far a handful of samples really reach. Recovering a spread needs the sum of each sample's equity squared, which the summary does not carry. Put it in a sidecar written beside the summary and sharing its run stamp (summary-sq-<stamp>), NOT in a fourth column of the summary itself: that file's shape is not ours to change. Other things read it, and pooling an old summary with a widened one would fail outright on the mismatched width -- the whole point of the format is that summaries concatenate. The sidecar takes the same shape as a summary, totals line included even though nothing reads a grand total of squares, so english-resummarize pools a pile of sidecars exactly as it pools the summaries they came from -- no second tool, no special case. Its count column repeats the summary's, and a rack whose counts disagree is skipped: a sidecar that does not belong to this summary describes other samples, and an interval built from mismatched parts would be worse than none. Both knobs are off by default, so the output is unchanged. Measured on a 20000-game summary: the floor alone smooths 851010 of 914624 leaves; a 0.5-point interval smooths 863277 and a 0.1-point interval 863839 -- tightening only ever smooths more. Asking for an interval with no sidecar smooths exactly what the floor does, since a spread that is unknown is not a spread of zero. A weighted sample -- one play standing for w draws -- contributes value squared times w, not the weighted sum squared, which would inflate its spread w-fold. Adding a third field to Cumulate meant every caller spelling out the same three-line add, so pool the whole sampled play in one helper instead -- a value, its square and its count have to move together or the spread stops describing the mean. It inserts lazily: or_insert would build a Cumulate on every call, including the common one where the rack is already there and it is dropped unused. The square is x.powi(2), which names the value once and compiles to the identical multiply -- LLVM merges the two into one symbol. Cumulate orders the two sums together with the count last, matching both files, where the summary is (rack, equity, count) and the sidecar (rack, sumsq, count). Playability shares Cumulate but never asks how tightly a word's occurrences agree, so it does not track a spread: NAN says unknown rather than claiming zero, and the scan is spared a multiply per word. Every path that pools samples moves all three numbers together, and the two that pool a whole Cumulate -- merging a thread-local map into the shared one, and english-summarize pooling a repeated rack -- have to as well. Adding a third field to a struct whose adds were written out by hand at each site is exactly how one gets forgotten: the compiler flags a missing field in an initializer but says nothing about a missing +=. Three tests pin it, one of which reproduces the Cauchy-Schwarz bound a real sidecar is checked against -- sum of squares can never fall below the mean squared times the count. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A sample floor decides which leaves to smooth with a bare count, and the count is a guess.
It is also the wrong question: fifty samples that agree closely pin an average down, fifty
that disagree wildly do not, and no count can tell them apart. This adds the real test --
how wide an interval the average sits in -- and the data needed to compute it.
Changes
WOLGES_GENERATE_SMOOTH_CI(a width in points; default 0.0, off) smooths a leave whoseaverage sits in an interval wider than the given width.
WOLGES_GENERATE_SMOOTH_CONF(default 0.99) sets how sure that interval has to be.
(
summary-sq-<stamp>);WOLGES_GENERATE_SMOOTH_SQpoints the decompose at it.at every call site.
Correctness
The interval TIGHTENS the floor rather than replacing it -- both tests must pass. Alone it
would be unsound: two samples that happen to agree spread by zero and would look perfectly
pinned, when two samples say almost nothing. The spread divides by n-1, matching
stats::variance.The squares ride in a sidecar rather than a fourth column of the summary because the
summary's shape is not ours to change -- other things read it, and pooling an old summary
with a widened one would fail on the mismatched width. The sidecar takes a summary's shape,
totals line included, so
english-resummarizepools sidecars with no new tool. Its countcolumn repeats the summary's, so a sidecar that does not belong to its summary is detected
rather than silently mixed in.
Test plan
0.1-point interval 863839 -- tightening only ever smooths more.
a spread of zero).
english-resummarize(no NaN); two summaries stillconcatenate and pool; playability summaries unchanged.
pooling_keeps_value_square_and_count_together,merging_thread_maps_keeps_every_square,pooled_spread_cannot_undercut_its_mean.