From 00182f41b782c42af2af47c6ca4cfa5f307a5c0e Mon Sep 17 00:00:00 2001 From: Andy Kurnia Date: Thu, 16 Jul 2026 01:28:34 +0800 Subject: [PATCH 1/4] reuse census sheets across uneven generations The census values leaves on sampled boards, one board per slot, and each board is fixed by its slot number -- so every generation sees the same boards. Building a board's play sheet is the costly step, so later generations reuse the sheets that earlier ones built. That reuse only kicked in when every generation used the same board count; a growing spec such as 256,512,1024 fell back to rebuilding every generation, doing the board work of the sum (1792) instead of the largest (1024). Track the largest board count seen so far and let each generation reuse every slot below it, building only the new ones on top. A growing spec now costs the largest count once, not the sum. The output is byte-identical -- same slots, same sheets, only the wasted rebuilds removed -- so a cheap early generation can settle the table before a costly last one pins it down. Co-Authored-By: Claude Opus 4.8 (1M context) --- lab/leavegen.txt | 8 ++++++++ src/main_leave.rs | 52 ++++++++++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 21 deletions(-) diff --git a/lab/leavegen.txt b/lab/leavegen.txt index e73e949..8f711d9 100644 --- a/lab/leavegen.txt +++ b/lab/leavegen.txt @@ -34,6 +34,14 @@ well, which is why it is the one to reach for. Those alternatives, and every lever that turned out not to help, are described below; the best current way to run each method is collected at the end of this file. +The 4x256 is shorthand for four generations of 256 boards. You can also write +the counts out, one per generation, to let them differ -- 256,512,1024 runs +three generations at those counts. Because every board is fixed by its position +in the run, the larger generations reuse the boards the smaller ones already +built: 256,512,1024 does the board work of a single 1024-board run, not the sum +of the three. So a cheap early generation can settle the table before a costly +last one pins it down, at no extra board cost. + GENERATING A TABLE BY SELF-PLAY diff --git a/src/main_leave.rs b/src/main_leave.rs index ab95687..60f4311 100644 --- a/src/main_leave.rs +++ b/src/main_leave.rs @@ -5207,19 +5207,18 @@ fn generate_census_leaves-.klv2, so a crash loses no completed gens and // WOLGES_CENSUS_RESUME can continue. One in-process klv2 build per gen (cheap vs the @@ -5588,9 +5587,11 @@ fn generate_census_leaves = if sheet_reuse { (0..max_boards) .map(|_| std::sync::Mutex::new(None)) @@ -6152,6 +6153,10 @@ fn generate_census_leaves start_gen, not > 0). - let reuse_board = sheet_reuse && gen_idx > start_gen; + // sheet-reuse: each slot's board + sheet is cached the first gen + // that reaches it. A slot below the running max of the prior gens + // (prior_max_boards) is already cached, so skip the game replay and + // re-value from the cache; slots at or above it are new this gen and + // get built + cached. On resume the cache starts empty (prior_max 0), + // so the first resumed gen rebuilds. + let reuse_board = sheet_reuse && (b as usize) < prior_max_boards; if !reuse_board { // greedy-play fresh games (from this slot's own rng) until one // reaches this slot's target fill. The target ROUND-ROBINS across @@ -6700,6 +6707,9 @@ fn generate_census_leaves Date: Fri, 17 Jul 2026 08:28:18 +0800 Subject: [PATCH 2/4] cache only the sheets a later generation reads The census values leaves on boards it samples, and building a board's play sheet is the costly step, so a sheet is cached and reused by the generations that follow. But every board cached its sheet -- including the boards of the LAST generation, which no later generation exists to read. A sheet holds one entry per rack the lattice knows, tens of megabytes each, so those dead sheets dominated the run's memory while buying nothing. Work out, per generation, how many boards the generations after it will use: live_after, a suffix maximum of the board counts. A slot at or above it is dead the moment its generation ends, so a generation fills the cache only below it, and the last generation's count is zero, so it caches nothing at all. Size the cache itself the same way -- the highest slot any generation keeps -- so a spec whose largest generation comes last stops parking a slot per board of it: 200,1000,400,300 needs 400 slots, not 1000, and 256,256,256,2048 needs 256, not 2048. The lookahead is a suffix maximum rather than the next generation's count because a slot has to survive a dip: in 400,100,300 the 300-board generation still reads slots 0..300, so the 400-board generation must hold them across the 100. Looking only at the next count would keep 100 and rebuild 200 sheets. Two tests pin this down -- one on the arithmetic, one walking each spec to check that no generation ever reuses a slot that was never cached. A slot is never starved: a generation only reads a slot it also has a board for, so any later reader of slot b forces b below the live_after of the generation that built it. Peak memory for a spec such as 256,256,256,2048 becomes the 256 slots the early generations share, not the 2048 the last one builds. Measured on 64,512: 15.55 GB -> 6.40 GB, with the leaves byte-identical (sheet-reuse on and off both give 08b86888304617a4d88e32f10e141fc7 on 8,32,16,12). Co-Authored-By: Claude Opus 4.8 (1M context) --- src/main_leave.rs | 140 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 126 insertions(+), 14 deletions(-) diff --git a/src/main_leave.rs b/src/main_leave.rs index 60f4311..7a4a6fb 100644 --- a/src/main_leave.rs +++ b/src/main_leave.rs @@ -4878,6 +4878,35 @@ fn wolges_census_scatter() -> error::Returns { // generations (Mutex so the first gen's writer publishes to later gens' readers). type SheetCacheSlot = std::sync::Mutex, Vec)>>; +// Plan the sheet cache for a board-count spec: which slots each generation is worth +// caching, and how many slots the cache needs at all. Returns (live_after, cache_len). +// +// live_after[g] = the largest board count of ALL the generations after g -- a suffix +// maximum, NOT simply the next generation's count. A slot stays worth keeping while ANY +// later generation still has a board for it, so the lookahead cannot stop at the next +// one: in 400,100,300 the 300-board generation still reads slots 0..300, so the +// 400-board generation must hold 300 of them ACROSS the 100-board dip. Comparing only +// against the next count would keep 100 and rebuild 200 slots later. +// +// cache_len = the highest slot any generation actually caches. A generation caches a +// slot only if it both has a board for it (board_counts[g]) and a later generation will +// read it (live_after[g]), so the cache never needs more than the largest of those +// minimums -- 200,1000,400,300 needs 400 slots rather than the largest generation's +// 1000, and 256,256,256,2048 needs 256 rather than 2048, because the last generation's +// sheets are never kept. +fn census_sheet_reuse_plan(board_counts: &[u64]) -> (Vec, usize) { + let gens = board_counts.len(); + let mut live_after = vec![0usize; gens]; + for g in (0..gens.saturating_sub(1)).rev() { + live_after[g] = (board_counts[g + 1] as usize).max(live_after[g + 1]); + } + let cache_len = (0..gens) + .map(|g| (board_counts[g] as usize).min(live_after[g])) + .max() + .unwrap_or(0); + (live_after, cache_len) +} + // Serialize the valued leaves to a klv2 file -- the same DawgOnly/Wolges build as // `buildlex -klv2` runs on the csv, but in-process. The machine word is the // leave's sorted tile bytes; the value is (value_mp(idx) - baseline_mp) points as f32. @@ -5064,8 +5093,9 @@ fn generate_census_leaves 1; // online mini-batch SGD (single-generation only). WOLGES_CENSUS_BATCH = boards @@ -5219,6 +5249,13 @@ fn generate_census_leaves-.klv2, so a crash loses no completed gens and // WOLGES_CENSUS_RESUME can continue. One in-process klv2 build per gen (cheap vs the @@ -5588,17 +5625,16 @@ fn generate_census_leaves = if sheet_reuse { - (0..max_boards) - .map(|_| std::sync::Mutex::new(None)) - .collect() - } else { - Vec::new() - }; + // that reaches it AND that a later gen will read back (live_after), then read by every + // later gen (a uniform spec fills every slot in gen 0; a growing one fills the new + // slots as its board count climbs; the last gen fills nothing). The multi-gen barrier + // orders a gen's writes before any later gen's reads, and each slot is written once, by + // whichever thread pulls it. Sized to sheet_cache_len -- the highest slot any gen + // actually caches, which a spec whose last gen is its largest keeps far below that + // gen's board count. Empty unless sheet_reuse. + let sheet_cache: Vec = (0..sheet_cache_len) + .map(|_| std::sync::Mutex::new(None)) + .collect(); eprintln!("census: {num_threads} threads over {board_counts:?} boards/gen"); std::thread::scope(|s| { @@ -6434,7 +6470,14 @@ fn generate_census_leaves( mod tests { use super::*; + // Every spec below is (board_counts, expected live_after, expected cache_len). + const SHEET_PLANS: &[(&[u64], &[usize], usize)] = &[ + // the lookahead is a suffix maximum, not the next gen's count: the 300-board gen + // still reads slots 0..300, so the 400-board gen keeps 300 of them ACROSS the + // 100-board dip. A next-gen-only rule would keep min(400,100)=100 and rebuild. + (&[400, 100, 300], &[300, 300, 0], 300), + // a dip that is genuinely the end: nothing after the 100 needs more than 100. + (&[400, 300, 100], &[300, 100, 0], 300), + // uniform: gen 0 builds every slot, the rest reuse; the last gen keeps none. + (&[256, 256, 256, 256], &[256, 256, 256, 0], 256), + // growing: each gen adds the new slots on top, the last gen keeps none. + (&[256, 512, 1024], &[1024, 1024, 0], 512), + // the ship recipe: three cheap gens share 256 slots, the 2048 gen keeps nothing. + (&[256, 256, 256, 2048], &[2048, 2048, 2048, 0], 256), + (&[200, 1000, 400, 300], &[1000, 400, 300, 0], 400), + // up, down, up again, down: the 700 must survive the 400 dip for the 700 gen. + (&[200, 1000, 400, 700, 350], &[1000, 700, 700, 350, 0], 700), + // single gen: nothing follows, so nothing is ever cached. + (&[256], &[0], 0), + ]; + + #[test] + fn census_sheet_reuse_plan_looks_past_the_next_generation() { + for &(counts, want_live, want_len) in SHEET_PLANS { + let (live_after, cache_len) = census_sheet_reuse_plan(counts); + assert_eq!(live_after, want_live, "live_after for {counts:?}"); + assert_eq!(cache_len, want_len, "cache_len for {counts:?}"); + } + } + + // Walk each spec the way the run does -- reuse a slot below the prior gens' running + // max, cache one below live_after, free the rest at the boundary -- and check the two + // invariants the plan has to guarantee: a reused slot is always still cached (no + // silent rebuild, no missing sheet), and no cached slot ever sits at or past cache_len. + #[test] + fn census_sheet_reuse_plan_never_reads_an_uncached_slot() { + for &(counts, _, want_len) in SHEET_PLANS { + let (live_after, cache_len) = census_sheet_reuse_plan(counts); + let mut cached = std::collections::HashSet::::new(); + let mut prior_max = 0usize; + let mut high_water = 0usize; + for (g, &n) in counts.iter().enumerate() { + for b in 0..n as usize { + if b < prior_max { + assert!( + cached.contains(&b), + "{counts:?} gen {g} reuses slot {b} but it was never cached" + ); + } else if b < live_after[g] { + cached.insert(b); + high_water = high_water.max(b + 1); + } + } + cached.retain(|&b| b < live_after[g]); + prior_max = prior_max.max(n as usize); + } + assert!( + high_water <= cache_len, + "{counts:?} cached slot {high_water} past cache_len {cache_len}" + ); + // and the size is tight, not merely sufficient. + assert_eq!(high_water, want_len, "cache_len not tight for {counts:?}"); + assert!( + cached.is_empty(), + "{counts:?} kept sheets after the last gen" + ); + } + } + #[test] fn parse_board_counts_expands_repeats() { assert_eq!(parse_board_counts("256").unwrap(), vec![256]); From 4e62b973a126aedee224d682975fcf009936e822 Mon Sep 17 00:00:00 2001 From: Andy Kurnia Date: Fri, 17 Jul 2026 08:37:06 +0800 Subject: [PATCH 3/4] free the sheets no later generation will read A generation now caches only the board slots a later generation will read back, but it still held those sheets until the whole run ended. Once a generation finishes, every slot at or above its live_after is dead -- no generation still to come has that many boards -- so there is no reason to keep paying for it. Free them at the generation boundary. A spec that narrows, say 200,1000,400,300, sheds its tail as it goes: the 400-board generation is the last to want slots 300..400, so those go before the 300-board generation starts. The final generation frees the lot, ahead of the confidence-interval report and the klv2 write. It is safe there because every worker has passed the barrier, so the generation's reads are done, and they wait on the next barrier until the leader has finished. This does NOT lower peak memory, and cannot: live_after only ever falls as the run goes on, so the cache reaches its high-water mark before the first shrink and never grows past it again. Measured on 64,512,256,128, peak is 10.42 GB either way. What it buys is giving the memory back DURING the run instead of at exit, which is what a census sharing a machine with other jobs actually needs. The leaves are byte-identical: 8,32,16,12 / 4x8 / 8,32,12 all match the values from before this change, and sheet-reuse on and off still agree. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/main_leave.rs | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/src/main_leave.rs b/src/main_leave.rs index 7a4a6fb..b2c706c 100644 --- a/src/main_leave.rs +++ b/src/main_leave.rs @@ -5250,9 +5250,9 @@ fn generate_census_leaves = (0..sheet_cache_len) .map(|_| std::sync::Mutex::new(None)) .collect(); @@ -6747,6 +6748,22 @@ fn generate_census_leaves Date: Fri, 17 Jul 2026 10:31:50 +0800 Subject: [PATCH 4/4] document both census recipes, 4x256 and 3x256,2048 The doc has recommended four generations of 256 boards and told the reader that adding boards or generations no longer moves the table. The second half is not supported: it rested on a comparison at two hundred thousand game pairs that came out a tie, which is far too few pairs to resolve an edge this small. The tie was the measurement failing, not the table having settled. With enough pairs the edge shows, and it always points the same way. 4096 boards beat 256 about 50.02 to 49.98 over five million game pairs at roughly 94 percent confidence, and 3x256,2048 edges 4x256 the same way -- 50.0064 at five million, 50.0103 and 50.0098 at one million on two different comparison seeds. Four readings, one side, every time. The edge is about a hundredth of a percentage point and 4x256 is three times faster (about 44 seconds against 142), so neither recipe is simply the answer and the doc now carries both with the trade stated: 4x256 while iterating, 3x256,2048 for a table worth keeping. Calling 4x256 the best would misdirect, since it loses every time it is measured; dropping it would cost the reader a table that is very nearly as good for a third of the wait. The cost gap is structural, not incidental -- every board is fixed by its position, so 4x256 builds 256 boards and reuses them, while 3x256,2048 must build all 2048 for its last generation. Below 2048 boards in the final generation the edge goes away, and a fifth generation measured no better than the third. to size a run for an unfamiliar tile set. Measured, it is not: asked to stop once nine leaves in ten had settled, it stopped after 576 boards and produced a table losing to 3x256,2048 by about 0.14 of a percentage point at almost total confidence. Settling by that measure is not the same as playing as well. Say so, rather than keep recommending it. That run changed two things at once -- the stop also requires WOLGES_CENSUS_RACK_SUMMARY=1, which makes the census write a rack summary to decompose rather than a table directly, and runs as a single generation -- so which of them cost the 0.14 is recorded as unpinned rather than guessed at. Co-Authored-By: Claude Opus 4.8 (1M context) --- lab/leavegen.txt | 81 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 17 deletions(-) diff --git a/lab/leavegen.txt b/lab/leavegen.txt index 8f711d9..367238f 100644 --- a/lab/leavegen.txt +++ b/lab/leavegen.txt @@ -22,17 +22,32 @@ TL;DR -- IF YOU JUST WANT A GOOD TABLE Run this. The census works out every leave's value exactly, one board at a time, and needs no environment variables: - cargo run --release --bin leave -- english-census CSW24.kwg - - 4x256 - -That builds a competitive table with no tuning: 4 generations of 256 boards, -which is enough for the leaves to settle, iterated from the null starting -tables (the two "-"). Every default is already the one to ship. The 256-board -count is measured on this English word list; for a different tile set, let the -adaptive stop described below size the run to the leaves themselves. The census -builds faster than plain self-play or board sampling and plays at least as -well, which is why it is the one to reach for. Those alternatives, and every -lever that turned out not to help, are described below; the best current way to -run each method is collected at the end of this file. + cargo run --release --bin leave -- english-census CSW24.kwg - - 3x256,2048 + +That builds a competitive table with no tuning: three cheap generations of 256 +boards to settle the leaves, then one generation of 2048 boards to pin them +down (3x256 is shorthand for three of them), iterated from the null starting +tables (the two "-"). Every default is already the one to ship. Those counts +are measured on this English word list, not universal constants; a different +tile set wants its own sweep, and the adaptive stop described below is not the +shortcut around it that it looks like. The census builds faster than plain +self-play or board sampling and plays at least as well, which is why it is the +one to reach for. Those alternatives, and every lever that turned out not to +help, are described below; the best current way to run each method is collected +at the end of this file. + +The last generation's board count is the number that matters, and it is the +only reason this costs more than the older recommendation of 4x256 (four +generations of 256). Both are worth knowing. 4x256 finishes in about 44 seconds +to this one's 142, because every board is fixed by its position in the run, so +4x256 builds 256 boards and reuses them while 3x256,2048 must build all 2048 +for its last generation. What the extra wait buys is about a hundredth of a +percentage point: 4096 boards beat 256 by roughly 50.02 to 49.98 over five +million game pairs, and 3x256,2048 edges 4x256 by a similar hair on every seed +tried. Small -- but it has landed the same way every time it has been measured, +so 4x256 is not the best table, just a very close and much faster one. Reach +for 4x256 while iterating, and for 3x256,2048 when the table is one you mean to +keep. Fewer than 2048 boards in the final generation gives the edge back. The 4x256 is shorthand for four generations of 256 boards. You can also write the counts out, one per generation, to let them differ -- 256,512,1024 runs @@ -1777,14 +1792,46 @@ good table, use the census. Census -- exact, and the one to use. It values every leave on a board exactly, then averages over many boards. + cargo run --release --bin leave -- english-census CSW24.kwg - - 3x256,2048 + +Three cheap generations of 256 boards settle the leaves, then one generation of +2048 pins them down. There is a cheaper recipe worth knowing, and which you +want depends on what the table is for: + cargo run --release --bin leave -- english-census CSW24.kwg - - 4x256 -4 generations of 256 boards settles the leaves: adding more boards or -generations no longer moves the table. That 256 is measured on this English -word list, not a universal constant; for a different tile set, let the run -size itself with the adaptive stop (WOLGES_CENSUS_CI_STOP_FRAC, in the -confidence-interval section earlier in this file), which ends a run once the -leaves have settled to a target width instead of at a preset count. +4x256 takes about 44 seconds against roughly 142 for 3x256,2048, because every +board is fixed by its position in the run: 4x256 builds 256 boards and reuses +them for its later generations, while 3x256,2048 must build all 2048 for its +last one. Three times the wait buys about a hundredth of a percentage point -- +4096 boards beat 256 by about 50.02 to 49.98 over five million game pairs, and +3x256,2048 edges 4x256 by a similar hair on every seed tried. That is a small +edge, but it has fallen the same way every time it has been measured, so 4x256 +is not quite the best table, only a very close and much faster one. Take 4x256 +while iterating; take 3x256,2048 for a table you intend to keep. + +Below 2048 boards in the final generation the edge goes away, and more +generations do not pay -- a fifth measured no better than the third. Those +counts are measured on this English word list, not universal constants, so a +different tile set wants its own sweep rather than these numbers taken on +faith. + +The adaptive stop (WOLGES_CENSUS_CI_STOP_FRAC, in the confidence-interval +section earlier in this file) looks like the way to avoid that sweep: it ends a +run once the leaves have settled to a target width rather than at a preset +count. Do not reach for it expecting a free lunch. Asked to stop once nine +leaves in ten had settled, it stopped after 576 boards and produced a table +that lost to 3x256,2048 by about 0.14 of a percentage point -- a real loss, not +noise. Settling by that measure is evidently not the same as playing as well, +so a stop wants a far stricter fraction than nine in ten, and wants checking +against a fixed-count table before being trusted. Note also that the stop +requires WOLGES_CENSUS_RACK_SUMMARY=1, which makes the census write a rack +summary to decompose rather than a table directly, and that it runs as a single +generation -- so it reads from an already-settled table rather than the null +one, since one generation never iterates the leaves. That measurement changed +two things at once (the stop and the rack summary), so which of them cost the +0.14 is not yet pinned down. + WOLGES_POOL_* shift the board window; for dynamic leaves add WOLGES_FULL=1 to keep the full-length values.