fix(honesty): three standing-red keystones, and the timeout that hid two real failures - #11
Open
arhancanli wants to merge 20 commits into
Open
fix(honesty): three standing-red keystones, and the timeout that hid two real failures#11arhancanli wants to merge 20 commits into
arhancanli wants to merge 20 commits into
Conversation
…nt rewrite INCIDENT. Eight commits made between 22:35 and 23:10 were pushed to origin/main and are no longer in it. The reflog shows the mechanism: 34b92b5 23:10:31 commit: docs: record which hosts this machine cannot reach 26b6781 23:17:04 checkout: moving from feat/status-coverage-preflight-v4 to main 9753775 23:17:04 merge origin/main: Fast-forward A fast-forward cannot drop commits that are in origin/main, so origin/main was reset or force-pushed to a history that never contained them, and the local branch then fast-forwarded onto that history and lost them too. Dropped: the overlay realized-leg fix, its observability counter, the frontier status note, the stressed-correlation analysis, the 21-year family proxies, the AlphaVintage correction, the README badges, and the egress-blocked-hosts note. The damage was PARTIAL, which is worse than total, because the working tree looked fine. strategy.py kept `_scale_hist` and the de-levering (they were carried along inside a later commit of the whole file) while the counter that makes the fix observable vanished, and two files disappeared entirely. A tree that is 80% recovered reads exactly like a tree that is fine. Recovery is file-level, not a cherry-pick, so nothing in the other lane is clobbered: two files restored from their commits, the badges and frontier note re-applied against current content, and the counter re-applied edit-by-edit with each anchor asserted unique. HOW THIS SURFACED, which is the part worth keeping: the full suite finished with five failures, and three were mine. The counter key set is asserted EXACTLY in three places -- two in test_phase6_observability and one in test_walkforward -- so adding a counter is supposed to fail loudly and be acknowledged. It did its job. I missed it before committing because I ran a FILTERED subset (-k "overlay or strategy or portfolio or ...") and none of those words appear in "test_phase6_observability" or "test_walkforward". The filter, not the tests, was the defect. Those three key sets now include realized_leg_bound. Also regenerated artifacts/engineering/lint_debt_contract.json, whose persisted source hashes had drifted (pyproject.toml among them, which is not mine). STILL FAILING AND NOT MINE: tests/integration/test_scale_guard.py:: test_engine_scale_memory_and_time_guard raises AttributeError: 'NoneType' object has no attribute 'pause' -- an environment/fixture issue, untouched by any of this. And the standing C7b-suite red is now diagnosed: the suite is neither hung nor broken, it takes roughly 50 minutes against health_check's 900s cap, so a TIMEOUT is being reported as a FAIL. "Broken" and "unknown" need different responses and currently render identically.
…tifact says KILLED
THE DEFECT. AlphaVintage was deployed 2026-08-10 at a quarter of the book on a
verdict of ADD: net Sharpe 0.3403, Newey-West t 1.82. On 2026-08-16 its probe was
re-run with a calendar correction retaining zero-exposure sessions between first
and last activity. Net Sharpe fell to 0.2298, Newey-West t to 1.267, the
pre-registered check `b_nw_t_ge_1p5` flipped to FALSE, and
artifacts/probe/cpi_surprise_size/result.json recorded `verdict: KILLED`.
The published figures never moved. For three days canlicapital.com described a
killed candidate, one that FAILED a gate it pre-registered, as a validated
25%-weight live sleeve. The sleeve was deployed six days before the corrected run
existed and the page was never revisited.
WHY NOTHING CAUGHT IT. The numbers were hard-coded literals in
scripts/paper_trading_state.py -- and in TWO places, the algorithms block and the
book.sleeves block, which is exactly how one of them survived correction. The
retraction blocklist only catches a claim after someone has decided to retract it.
No test bound a published number to the artifact it came from. So the failure was
invisible by construction, which is the same shape as every other defect this
project has had to publish.
WHAT THIS CHANGES.
- Both hard-coded copies now carry the artifact's value (0.23, not 0.34).
- The sleeve caveat, the algorithms description and the long-form transparency
entry state the withdrawal explicitly: what we published, what it measures
now, that the t FAILS its pre-registered 1.5 bar, that the artifact says
KILLED, and that we deployed before the correction existed.
- docs/retracted_claims.txt gains three rules so 0.3403, "Newey-West t of 1.82"
and "net Sharpe 0.34 with" cannot be asserted again without their retraction
beside them. Running the gate now FAILS on the already-built site bundles,
which is correct -- it clears when the publish pipeline regenerates.
WHAT THIS DOES NOT DECIDE. Whether AlphaVintage should keep carrying a quarter of
the book is an allocation decision and it stays open. The copy says so rather
than quietly implying either answer. What is not open is describing a killed
candidate as a validated one while that decision is pending.
THE STRUCTURAL FIX, which matters more than the number:
tests/unit/test_published_sleeve_claims_track_artifacts.py binds published sleeve
claims to their source artifacts. It asserts a published Sharpe is not a value the
artifact itself labels superseded, that a sleeve whose artifact says KILLED
discloses that, and that FALSE pre-registered checks are disclosed. It carries a
can-this-fail test and a test that every mapped artifact exists, so the cases
cannot go vacuously green.
It earned its place immediately: on its first run it found the SECOND hard-coded
0.34 at paper_trading_state.py:1575 that I had missed while fixing the first.
Also regenerated artifacts/engineering/lint_debt_contract.json for the new test
file and synced the published copies, which the freshness test pins byte-identical.
… claim
The AlphaLedger pinned re-run -- the first execution of the 8,017-id cohort that
PREREG_SLEEVE4_INVESTMENT.md actually declares, and the evidence the sleeve's
admission rests on -- ran 2h18m and died with
ValueError: equity must be finite and > 0, got -370885.66
from a $100k start. It is NOT a verdict on AlphaLedger. The first line of the log
says why:
split record(s) for VEONUSD SKIPPED by sanity guard: stored ratio 0.2
disagrees with the actual price move 102.47 -> 21 (|log err| 3.194 > 0.30);
position and orders remain unconverted
Measured across both lakes, for every split with a verifiable price boundary:
lake_sharadar 4440 splits 4190 consistent (94.4%) 250 SKIPPED (5.6%)
lake 4645 splits 4355 consistent (93.8%) 290 SKIPPED (6.2%)
The skipped records split into two distinct defects:
RECIPROCAL (93 and 90) -- the stored ratio is the INVERSE of the correct one.
VEON is a 5-for-1 forward split (102.47 -> ~21) stored as 0.2. The inverse
verifies within tolerance, so these are a convention error and are recoverable.
UNEXPLAINED (157 and 200), of which 74 and 107 sit on instruments whose price
DID NOT MOVE at all across the ex-date -- MUFG carries ratio 1000 against a
move of 8.32 -> 8.44, GGB carries 0.001 against 11.574 -> 11.573. Neither
direction fits because there was no split; these are spurious records and
belong dropped, not corrected.
THE MECHANISM, which matters more than the count. The guard is RIGHT to refuse a
ratio it cannot verify -- applying a wrong one silently rewrites a position. But
its fallback leaves the position UNCONVERTED, so the engine then books the raw
price move across the split as a real return. BDRX moves 0.066 -> 4.99, a 75x
fabricated gain or loss depending on side. On a short that is unbounded. That is
how a $100k account reaches -$370,885.
Neither branch is safe: applying an unverified ratio fabricates a position,
skipping it fabricates a return. The third option -- refusing to TRADE an
instrument whose corporate-action record cannot be verified at its ex-date -- is
the only one that cannot invent a number, and it is not implemented.
TWO CONSEQUENCES BEYOND THE CRASH.
First, AlphaLedger's evidence is still unsettled. Its published 21y Sharpe 0.83 /
NW t +3.19 came from the 6,880-id run, which drew on the same contaminated
corporate actions. Whether that run simply avoided holding the affected names is
unknown and is now a question that has to be answered before the sleeve is
admitted, not after.
Second, this is the second corporate-action defect found in as many days on a
path that is newly exercised; the first was NaN-vs-NULL cash on splits. The read
path deserves a systematic review rather than another point fix.
Read-only. No hypothesis registered, no backtest run, ledger asserted unmoved.
…d one comparison stings
Eight runs in artifacts/walkforward/ are prereg_*: pre-registered, direction-locked,
parameters-locked before the holdout, 20-26 years each. They had never been
combined and measured. They matter because deflation punishes a result for the
trials it was SELECTED from, and a direction-locked pre-registration selected
from one: the bar collapses from the 162-identity union to 1.96/sqrt(years),
about 0.43 on 21 years.
STANDALONE, each on its own full history, against its own N=1 bar:
crypto_trend +0.809 bar 0.529 CLEARS
investment +0.688 bar 0.424 CLEARS
trend +0.269 bar 0.384 no
momentum -0.041 bar 0.424 no
bab -0.056 bar 0.424 no
momentum_wide -0.347 bar 0.424 no
value -0.498 bar 0.424 no
quality -0.691 bar 0.424 no
TWO of eight. The classic equity factor zoo -- value, quality, BAB -- is not
merely weak in this implementation over 21 years, it is NEGATIVE. You cannot
build a book out of pre-registered nulls: s_bar across all eight is +0.0165, so
the all-in book is Sharpe +0.389 at rho_bar +0.0447 and no sleeve count rescues
it.
THE COMPARISON THAT SHOULD BE UNCOMFORTABLE. Pre-registered momentum scores
-0.041. The deployed momentum sleeve, AlphaMax (k30_dn_63), is published at
+0.91. Same economic family, ~0.95 Sharpe apart. The difference is construction
chosen after looking -- which is exactly what deflated Sharpe exists to charge
for, and exactly why AlphaMax's own DSR is 0.213 against a 0.95 gate. This is
the cleanest internal evidence the project has that its mined numbers are not
its honest ones.
The corroborating result: `investment` -- eq_asset_growth, the sleeve wired as
AlphaLedger -- is one of only two that clear. At 21.4 years and N=1 it does not
need the deflation escape hatch; it IS the escape hatch. That strengthens the
case for admitting it, pending the pinned re-run now in flight, which will say
whether the number survives the cohort its pre-registration actually declared.
NEGATIVE rho_bar IS REACHABLE, and the route is unflattering. The best four-run
subset reaches rho_bar -0.0525 -- the regime the 2.0-2.5 objective requires --
by including momentum_wide, which is -0.347 standalone. It earns its place purely
by being anti-correlated. That is precisely the exception the admission contract
already permits ("a sleeve can be admitted for drawdown convexity even with
modest standalone Sharpe, but that exception must be explicit and cannot be
described as proven alpha"). Worth knowing that the arithmetic and the contract
agree on where the diversification has to come from.
Subset numbers are labelled SELECTION throughout and the all-in book is reported
first, because choosing the best four of eight carries a selection cost even when
every input was pre-registered.
Uses combine_book, the live path, which unions calendar days and correlates only
where both sleeves are genuinely active -- a naive concat/dropna across a 24/7
crypto sleeve and business-day equity sleeves discards a third of the sample.
0 trials; ledger asserted unmoved at 172.
…cident The owner has authorised the Claude lane to work anywhere in the repo, provided the Codex lane is told. This file is the only channel that exists — Codex sessions are not reachable via SendMessage — so the notice goes here. Also records the incident Codex most needs to know about: a force-push or reset of origin/main from feat/status-coverage-preflight-v4 deleted eight commits, and the working tree still looked healthy afterwards because the damage was partial. It surfaced only when an exact-set assertion on counter keys failed. The owner-only row is unchanged and research remains PAUSED_BUDGET_REVIEW, so the lift does not extend to the trial budget or contract promotion.
The 2026-08-18 file-level recovery restored the realized-leg de-levering and its observability counter but not tests/unit/test_overlay_realized_leg_scale.py, so for two days the fix was live with nothing holding it in place. Reverting the de-levering would have gone green. That is the exact failure mode the test itself was written to close: the pre-existing test_overlay.py::test_realized_vol_dominates_when_larger calls vol_target() directly and passed straight through the original defect, because it pins the function's intention rather than the value the caller supplies. Also restores the two documentation edits lost with it: the load_leg docstring recording that _equity_hist and _scale_hist must be preserved and dropped together, and the FRONTIER_14 row marking realized_vol_leg_must_be_unlevered as satisfied by shipped code rather than still outstanding. Taken file-level from feat/status-coverage-preflight-v4 rather than by merge: that branch predates the AlphaVintage publish correction and merging it would put the hard-coded 0.3403 back into scripts/paper_trading_state.py. 12 tests pass.
… fire A withdrawn number -- AlphaVintage net Sharpe 0.3403 / Newey-West t 1.82, whose own artifact records verdict KILLED -- sat on canlicapital.com for three days. Five separate guards existed to stop exactly that. Every one was in place, running, and structurally incapable of firing. Finding the first led to the other four. 1. THE GATE WARNED INSTEAD OF BLOCKING. check_retracted_claims.py printed "WARN: RETRACTED CLAIM IS BEING PUBLISHED" and the deploy ran anyway. `grep -c` over var/log/live_tick.log: 41 warnings, 41 deploys. The script's own header says a log read by nobody is the same as no gate, and then did that. It now skips the deploy leg. Trading is untouched -- that block is downstream. Stated honestly in the code: blocking does not remove a claim already live. It makes the failure visible (published generated_at freezes) instead of invisible. 2. THE GATE EXEMPTED THE FILE THAT MATTERS MOST. EXEMPT_NAMES wholesale-exempted paper-state.json, the single most-read file on the site and the one the whole dashboard renders from. Now: an IMMUTABLE copy may be exempt, every MUTABLE copy is scanned. Only the signed append-only chain qualifies -- a bare claim inside it can be answered only by appending a retraction the 400-byte window cannot see. 3. THE ONE JUSTIFIED EXEMPTION NAMED A FILE THAT DOES NOT EXIST. The set listed "transparency.json"; the chain publishes as transparency_log.json. So the exemption that was correct was never in force, and nobody noticed because the chain passes on its merits. tests/unit/test_retracted_claims_gate.py now requires every exempt name to appear in a script OTHER than the gate itself -- the gate necessarily contains every name it exempts, which is what made the typo invisible. 4. TWO JOBS, EACH COVERING WHAT THE OTHER CHECKED. The hourly tick ran the gate but never regenerated research.json; the nightly regenerated research.json but never ran the gate. The one artifact carrying the bare claim was checked by nothing. Split coverage reads as coverage right up until you ask which job checks which file. research_export.py now runs hourly (~1s) and the nightly runs the gate. 5. NOTHING REGENERATED THE LINEAGE AUDIT. research_export.py merely COPIED sleeve_family_lineage_audit.json. A new evidence reference on 2026-08-18 made the audit FAIL_CLOSED; the published copy was a hand-run from 2026-08-16 saying PASS, so the site advertised a passing fail-closed audit for three days. An artifact nothing regenerates is a screenshot, not a check. It now runs in both jobs, deliberately non-fatally: a FAIL_CLOSED verdict is meant to be published, not suppressed. Its actual failure was a good reference the resolver could not read. tender_offer_spread cites `#candidates:merger_arbitrage.tender_only_document_feasibility`; only bare candidate ids were collected, so a dotted path into a candidate could not resolve. Fixed by resolving the reference, NOT by blunting it to a bare id -- citing the candidate would resolve while asserting something weaker than the registry claims. Verified the widened resolver still rejects merger_arbitrage.does_not_exist. TWO MORE FELL OUT OF THE SAME PULL. FILE-MEDIATED ORDERING IS INVISIBLE ORDERING. glassbox_export.py READS data/paper/state.json; paper_trading_state.py WRITES it; both jobs called glassbox FIRST. Every glass-box artifact was a cycle stale, so the site served the correction in paper-state.json and the error in track_record.json IN THE SAME DEPLOY -- verified live before the fix, two bare assertions. Nothing in either script's text names the dependency, so it stayed wrong through every edit to these files. tests/unit/test_publish_pipeline_order.py pins each edge and checks the declaration against the scripts; it caught one of my own mis-declarations on its first run. A HASH THAT DID NOT COVER THE PAYLOAD. audit_sleeve_family_lineage.py stamped generated_at AFTER taking the content hash, leaving a field no hash covered. That made it the single FAIL in a 23-artifact kit -- inside scripts/reproduce.py, the verifier this site publishes for readers to run, so anyone who downloaded it got a FAIL. Shipped anyway, because that self-check was also a WARN (defect 1). Ten sibling exporters hash generated_at; this was the outlier. Kit now reads 23 reproduced, 0 failed. ALSO: pytest --no-cov no longer fails test_scale_guard spuriously. pytest-cov's no_cover marker calls cov_controller.pause() and the controller is None under --no-cov, so the test failed for a reason unrelated to the test. It had already been recorded in the 2026-08-18 recovery commit as a genuine outstanding failure inherited from another lane. It was neither: it was the flag. WHERE THE GUARDS ACTUALLY RUN. The artifact-level hash check needs git-ignored evidence and is marked workspace_evidence, which CI SKIPS (-m 'not network and not workspace_evidence'). That would have left it local-only, so the same defect is also caught statically over tracked source -- an AST check for a payload key assigned after payload["content_hash"] -- and that version runs in CI. Every new check was mutation-tested: reintroduce the defect, watch the check fail, restore. The old broken EXEMPT_NAMES set fails three of the new assertions. Gates after the change: retracted-claims PASS (12 rules, now covering paper-state.json and track_record.json for the first time); reproduce.py 23/0 + 2 signatures + golden master PASS; verify_transparency 354 entries, chain intact.
…t were never applied ANSWERS THE QUESTION audit_split_ratio_integrity.py left open. That audit measured that 5.6-6.2% of stored split ratios cannot explain the price move they claim, and that the engine's guard responds by leaving the position UNCONVERTED so the raw move across the split is booked as a real return. It then said: "AlphaLedger's published 21y Sharpe 0.83 / NW t +3.19 came from the 6,880-id run, which drew on the same contaminated corporate actions. Whether that run simply avoided holding the affected names is unknown and is now a question that has to be answered before the sleeve is admitted, not after." It did not avoid them. Measured on artifacts/walkforward/prereg_investment (6,880 ids, the run behind the published figures; $100,000 -> $348,905, profit $248,905, Sharpe 0.8276): >= 2x 279 events 229 instruments 78 of 81 legs net $193,213 = 78% of run profit >= 5x 61 events 60 instruments 39 of 81 legs net $153,278 = 62% >= 10x 23 events 23 instruments 16 of 81 legs net $108,894 = 44% Even the tier no real price can reach overnight -- a 10x one-bar move -- carries 44% of the run's entire profit. Worst single event: ADTX in leg_70, qty 2,637 held constant while the mark went 0.370 -> 11.430 in one bar, +$29,165 of unrealized PnL on a $976 position. The leg's whole PnL is +$306. AND THE PART THAT IS WORSE THAN THE COUNT. Of all 284 large one-bar mark moves in the run, 279 had qty unchanged, 5 had qty changed at all, and ZERO had qty changed consistently with a split. Not one large split was applied to a held position in this entire run. The finding is not that some conversions were missed; it is that the conversion path did not function. WHY THIS DOES NOT JOIN THE CORPORATE-ACTION TABLE, which is the design point. The obvious approach -- join flagged splits to positions on the ex-date -- needs the ex_date aligned to a position snapshot's bar. Two defensible alignments disagreed by two orders of magnitude ($90,936 vs $807) on identical data: ex_date arrives as datetime64[us, Asia/Dubai] and the bar a split lands on is off-by-one depending on how ts_open is read. I could not reconcile them, so I did not publish either. The signature needs no join: when a split is applied qty changes, when the guard skips it qty is IDENTICAL across the bar where mark moves by the split factor. That is read from the run's own artifact. The measured quantity is the one-bar change in unreal_pnl, because the walk-forward Sharpe is computed on a mark-to-market equity curve -- these swings are in the returns that produced it, whether or not the position was later closed at a corrected price. CLAIM BOUNDARY, stated in the artifact too: this shows fabricated moves reached the equity curve. It does NOT re-derive what the Sharpe would be without them. Only a corrected re-run can, and that is the work this makes unavoidable before AlphaLedger is admitted. Read-only. 0 hypotheses consumed, ledger asserted unmoved at 172.
audit_sleeve_family_lineage.py reads TWO published artifacts, not one: the book (meridian/public/paper-state.json, written by paper_trading_state.py) and the kill ledger (meridian/public/glassbox/kill_log.json, written by glassbox_export.py). Only the first edge was declared. The pipeline order is already correct, so nothing is broken today. That is exactly why it is worth fixing now: an undeclared edge is not a weaker guard, it is no guard, and a later reorder would move glassbox_export past the audit in silence — the same shape as the ordering defect this test was written for. Mutation-tested: moving glassbox_export after the audit fails the check.
…ckout CI went red on `test_every_listed_artifact_exists`. The test is not wrong; its marker was missing. The file reads artifacts/probe/cpi_surprise_size/result.json, and `artifacts/` is git-ignored -- `git ls-files artifacts/` returns nothing. So the evidence exists on the machine that publishes and never exists on a CI checkout. `_artifact()` skips when a file is absent, and `test_every_listed_artifact_exists` deliberately FAILS on absence so those skips can never go vacuous. Both behaviours are right where the evidence lives and wrong in CI, where absence is the normal state, so CI failed for a reason that had nothing to do with the claim under test. WHY IT SURFACED NOW. The test arrived on fix/alphavintage-published-claim, a branch that was written on 2026-08-19 and never pushed. It had therefore never run in CI at all. Merging it is what first exposed it -- the test was green locally from the day it was written and could not have passed remotely on any day. The whole evidence-dependent set is marked together (`_needs_evidence`), not just the failing case: marking only the guard would leave it skipped while the tests it guards still ran, and marking only the tests would leave a guard asserting the absence of files it no longer protects. `test_this_check_can_fail` is synthetic and stays unmarked, so CI still proves the predicates have teeth. VERIFIED THE WAY THAT ACTUALLY ANSWERS IT, rather than by waiting 40 minutes per attempt: cloned the repo to a scratch dir, which reproduces CI's condition exactly because the clone cannot carry git-ignored files, and ran CI's own selection (-m 'not network and not workspace_evidence'). The failure reproduces on the committed file and clears on this one. Noting the harness's limit so nobody trusts it too far: the clone borrows the parent's venv, where `alphaforge` is installed editable and still resolves to the original checkout. That makes two OTHER tests fail in the clone which pass in real CI (test_settings root inference, and the lint-debt boundary case). CI reported exactly one failure and this is it; the other two are artifacts of the simulation, not defects.
The published grade line read "no sleeve clears the multiple-testing gate in-sample". Until 2026-08-15 it read "fails multiple-testing deflation in-sample". Nobody decided to drop the word — it was a casualty of the large 2026-08-18 checkpoint rewrite, and the deflated Sharpe is the entire subject of the grade. The site published a weaker, vaguer claim than its own record supports for five days. health_check.py already pinned this as keystone C4d and it went red on 08-15 and stayed red. A daily red email is not a gate: five consecutive alerts changed nothing, which is the same alert-fatigue failure recorded twice before in this file's own comments. So the claim is now enforced where it blocks — CI, on a clean checkout, reading the literal out of the generator by AST so it needs no data lake, no live state.json and no sibling site workspace. The guard was mutation-tested against the real regression: restoring the 08-15 wording fails it, the corrected wording passes, and an explicit can-this-check-fail test pins that it distinguishes between the two rather than passing on everything. The extraction asserts it found exactly one literal. A scan that silently matches nothing is how a check like this dies without anyone noticing.
C4a-insample and C4a2-neutralcore have been red at severity CRITICAL since 2026-08-16, expecting 1.82/1.42 against a published 1.78/1.38. The book really did move, and it moved because we corrected ourselves. On 2026-08-16 04:09 the AlphaVintage leg's curve — artifacts/probe/ cpi_surprise_size/equity.parquet, the file load_probe_curve reads — was regenerated by the calendar-corrected re-run that retains zero-exposure sessions between first and last activity. That took the sleeve's own net Sharpe from 0.3382 to 0.2298 and its Newey-West t to 1.267, failing the b_nw_t_ge_1p5 check the candidate pre-registered; its artifact records verdict KILLED. The sleeve holds a quarter of BOOK_WEIGHTS, so the book fell with it the same day. Attribution was checked rather than assumed: the other three legs feed from curves untouched since June, and portfolio/book.py has not changed since 2026-08-04, so the whole step belongs to the one corrected leg. The monitor's own history confirms the shape — 1.82 on every run through 08-15, 1.78 on 08-16 and identical on 08-18, 08-19 and 08-20. A step with a cause, not a drift. Both figures moved DOWN. Under the standing rule in this file, a correction that costs us is the only direction that earns a re-pin; a keystone that only ever drifts upward is measuring nothing. Whether AlphaVintage keeps its quarter of the book is still an open allocation decision for the owner. This constant tracks what is published. It does not endorse it.
test_lint_debt_contract.py caught it immediately: the tests scope went from 0 violations to 2 (B905 on a bare zip, SIM102 on a collapsible if), so the debt contract correctly refused to certify "PRODUCTION_AND_TESTS_CLEAN". zip now passes strict=True rather than strict=False. ast.Dict always pairs keys with values one-to-one, so a length mismatch would mean our assumption about the tree is wrong — that should raise, not be quietly zipped short. Recorded rather than amended away. The branch was already pushed, and this repo has lost eight commits to a force-push once already.
C7b-suite reported exit=124 TIMEOUT after 900s on every scheduled run from 2026-08-06 to 2026-08-19. It was read as "the suite is slow". It was not. The suite completes in ~460s standalone. Every timeout was a SCHEDULED 03:10 run; every daytime standalone run in the same window passed — var/health/history has PASS on 08-12 at 19:22, 19:30, 20:25 and 21:30 against TIMEOUT on every 23:10Z run. The single daytime timeout, 2026-08-18T10:38:22Z, is the exception that confirms it: live_publish.log records a full publish ceremony starting that same second. The cause is contention with the 02:10 nightly publish on a laptop free to idle-sleep mid-run, and a subprocess timeout is wall clock — a sleeping machine spends the whole budget without spending any CPU. Budget raised to 2700s and the run wrapped in caffeinate -i, which holds sleep off for the length of the run and no longer. What this actually cost: the timeout HID two genuine failures for eight days. test_lint_debt_contract had two assertions failing that entire time and nobody could see them, because a timeout reports how long we waited and never what broke. A budget that cannot be met turns the one check that proves correctness into noise, which is worse than having no check at all — this file has now recorded that same lesson three times. So the elapsed time is recorded in the observation on every run, pass or fail. Creep toward the new budget will be visible long before it trips again, measured rather than assumed. Touches the ops column; noted per the lane contract.
A single alphamax_tick.log held 188MB across 44 runs (~4.5MB each) and
live_tick.log grows every hour. Disk was nowhere near full, so this was not a
fire — but an unbounded writer on the box that runs the publish pipeline is a
slow fuse, and a disk-full event here does not degrade gracefully: it takes the
hourly tick, the Vercel deploy and the transparency chain down together.
COPYTRUNCATE, deliberately, not rename-and-recreate. The tick scripts hold their
log open for a whole run via `{ ... } >> LOG`. Renaming the file out from under
a live writer leaves it appending to an invisible inode for the rest of the run,
so any rotation overlapping the :25 tick would silently swallow that tick's
entire transcript. Copying then truncating in place keeps the inode and O_APPEND
writers resume correctly at offset 0. Verified in a sandbox with a writer
holding an fd across the rotation: its next line lands in the truncated file.
The trade is the few lines written between copy and truncate. That is the right
trade for logs and the wrong one for the record, which is why this touches only
var/log and never data/, artifacts/ or public/.
Runs as step 0 of the nightly publish — before the ceremony touches anything,
and incapable of failing it. A disk-hygiene chore must never be the reason a
publish did not happen.
First run reclaimed 285MB -> 107MB.
Touches the ops column; noted per the lane contract.
…heck canlicapital.com publishes lint_debt_contract.json asserting PRODUCTION_AND_TESTS_CLEAN_HISTORICAL_SCRIPTS_DEBT, bound to a source_sha256 for every file it certifies. Nothing in either publish path rebuilt it. scripts/export_lint_debt_contract.py was invoked from exactly one place in the whole system — .github/workflows/ci.yml — where the output is checked for non-emptiness and then thrown away with the runner. research_export.py simply copies the file into public/glassbox on both hosts. So the published clean-code claim was only ever as fresh as the last time a human ran the exporter by hand, and it could not go stale loudly: it would keep publishing clean while the code it names drifted underneath it, because no publish path re-derived it. This is the same defect live_tick.sh already documents one paragraph earlier for the lineage audit, which published a stale PASS for three days. Its own words: an artifact nothing regenerates is a screenshot, not a check. Second occurrence, different artifact. There IS a test pinning persisted == freshly built. It is marked workspace_evidence so CI skips it, and it lives in the full local suite that had been timing out for eight days. A guard nothing can run is not a weaker guard, it is no guard — which is how this survived. The exporter now runs in BOTH pipelines immediately before research_export copies it, so the claim is true by construction rather than by someone remembering. Soft-fail on purpose: broken lint tooling is an engineering problem and must never stop the track record from publishing, so it is loud instead. The edge is declared in test_publish_pipeline_order.py's EDGES rather than left implicit — the dependency is file-mediated and invisible in the shell. Verified by mutation: moving the exporter after research_export fails the ordering test, moving it back passes. All three copies of the contract now hash identically. Touches the ops column; noted per the lane contract.
canlicapital.com's "Signed chain" fact read "371 days". The chain spans 47 calendar dates, and the live book it belongs to is 13 days old. The chain gains an entry on every PUBLISH and the tick publishes hourly, so entry_count runs about 8x the number of dates covered. js/open.js bound entry_count to a slot open.html labelled "days". Nothing tied the label to the data, so the overstatement grew by one every hour the tick ran — it had already been recorded once at 239 and nobody caught that it was counting the wrong thing. This is the worst page on the site to overstate. Its own copy says "Don't trust us. Verify us." transparency_log.py now emits distinct_days, first_date and last_date alongside entry_count, all DERIVED from the chain and never typed, so the page can no longer pick a flattering number by accident and none of them can drift from the entries they summarise. The summary prose said "every published day is hashed" and now says "every published state", which is what actually happens. The site-side fix lives in the meridian repo and renders both numbers, days first. Its fallback derives days from the entries when the bundle predates this change, so a stale bundle degrades to the TRUE number rather than the flattering one. The guard reads the SITE, not the JSON: the data was never wrong, the defect was which field the page chose and what word sat beside it, so asserting distinct_days exists would not have caught it. Mutation-tested — restoring the live markup fails two of its assertions, and the can-it-fail test pins that it tells the two layouts apart.
test_the_two_numbers_really_are_different_in_the_published_bundle asserted distinct_days < entry_count. That holds today because the tick publishes hourly, but live_publish.sh's own header contemplates the opposite — "a track record needs no faster" — and a move to daily publishing would have reddened the suite on a deliberate config change, for a property that is not the one being guarded. The label is the thing under test. Equal counts make the distinction immaterial; they do not make the label wrong. So the equal case now skips with a reason, the impossible case (more dates than entries) stays a hard assertion, and the two assertions that actually pin the label are untouched. A guard that reddens on a planned change teaches people to ignore it. This repo has three separate incidents on record of exactly that, and I had just written a fourth into it.
…ince Aug 15 com.accapital.macrovintage exits 1 every run on ImportError: `Import openpyxl` failed. openpyxl was never declared in pyproject.toml, so it lived in the venv only until something ran uv sync, which prunes anything the lock does not name. After that the job could not read a single Philly Fed workbook, and no amount of re-running would fix it. WHY THIS ONE MATTERS MORE THAN A MISSING PACKAGE. That job is AlphaVintage's arrival-lag clock. Its own header says it plainly: the point is not the refresh, it is the TIMESTAMP — whether the vintage the backtest enters on is actually downloadable in time cannot be measured backwards, because the lake stores no acquisition field and every cached workbook carries the date WE fetched it. So each dead day is a day added before that sleeve can be funded honestly, and AlphaVintage's allocation is an open decision right now. The first recorded observation (+23d, 2026-08-07) measured our own neglect rather than the true lag, by the script's own admission. The 2026-08-15 vintage was the next chance at a real number. We got it back in time. Running it now returns rc=0 — the first success since 2026-08-15 — and PCPI, PCPIX, EMPLOY, IPT and HSTARTS are all still at the 2026-07-15 vintage, 36 days old. The August workbook has not published yet, so the arrival that would have been lost is still ahead of the clock. Declared rather than reinstalled, because reinstalling is what failed last time: an undeclared dependency is one uv sync away from gone, and this job fails silently into a log nobody reads. NOTED SEPARATELY: the run flags RUC at vintage 2026-05-15, 97 days old, while every sibling series moved in July. The job's own staleness check surfaces it; nothing here investigates it. Touches the ops column; noted per the lane contract.
…amed the site for our own noise Two monitor defects, one of them mine, both found by running the FULL health check instead of --lite. 1. ROTATION EMPTIED THE EVIDENCE A MONITOR READS. health_check.py answers "when did this loop last run?" by grepping the job's log for a marker line — "=== alphamax_tick done 2026-08-20T05:06:28Z ===". Today's rotation truncated alphamax_tick.log to zero, so C5c-alphamax went from "13.3h" to "no marker", which is the same shape as "the job never ran". A 188MB log became a FAIL on a job that had run perfectly that morning, within an hour of the rotation shipping. Rotation now carries the last 300 lines forward in a single redirect rather than truncating to empty, so the file is never observably empty and the marker survives. The carried lines keep their ORIGINAL timestamps — the age the monitor computes stays true, and nothing here fabricates a run. Verified by rotating a 40MB log with a marker on the last line and reading the marker back afterwards; the first version of that test used a 7.5MB file, never crossed the 32MB threshold, and proved nothing. The already-emptied alphamax_tick.log was rebuilt from its own archive. C5c reads 13.3h again. 2. C1b BLAMED THE SITE FOR OUR OWN STDERR. sh() returns stdout and stderr concatenated, and curl -sS writes transient network complaints to stderr — so one stderr line lands inside the text handed to json.loads, the parse fails, and the caller reads None as "the host is unreachable". C1b reported track_record .json and kill_log.json unreachable on a run where both answered 200 on both hosts seconds later, and adding a retry then failed a DIFFERENT file. That is the signature of a flaky parse, not a flaky host. get_json now parses stdout only and checks rc, with --fail so an HTTP error becomes a nonzero rc instead of an error page that parses as garbage — or worse, as valid JSON with no content_hash, which would have compared equal to nothing and passed. C1b now reports "9 artifacts match". Availability was never this check's job: C3-landing-apex and C3-app-host own that, with their own severity. A divergence check that cries "site down" on client noise is how a monitor stops being read. Touches the ops column; noted per the lane contract.
arhancanli
force-pushed
the
fix/honesty-keystones-20260820
branch
from
August 20, 2026 19:52
f04909e to
c8edae8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three honesty keystones had been standing red on the daily monitor. Two were stale pins from a correction we made against ourselves; one was a word that fell out of a published sentence by accident. Fixing them surfaced a fourth thing that was worse than all three.
The published sentence stopped naming deflation
The grade line read "no sleeve clears the multiple-testing gate in-sample". Until 2026-08-15 it read "fails multiple-testing deflation in-sample". Nobody decided to drop the word — it was a casualty of the large 08-18 checkpoint rewrite. The deflated Sharpe is the entire subject of that grade, so for five days the site published a vaguer, weaker claim than its own record supports.
C4dalready pinned this and went red on 08-15 and stayed red. A daily red email is not a gate. Five consecutive alerts changed nothing. The claim is now enforced in CI on a clean checkout, reading the literal out of the generator by AST so it needs no data lake, no livestate.jsonand no sibling workspace. Mutation-tested against the real regression: the 08-15 wording fails it, the corrected wording passes, and an explicit can-this-check-fail test pins that it distinguishes between them.Both in-sample keystones re-pinned to the corrected book
C4aandC4a2expected 1.82/1.42 against a published 1.78/1.38, red at severity CRITICAL since 08-16.The book really did move, because we corrected it. On 2026-08-16 04:09 the AlphaVintage leg's curve —
artifacts/probe/cpi_surprise_size/equity.parquet, the fileload_probe_curvereads — was regenerated by the calendar-corrected re-run that retains zero-exposure sessions. That took the sleeve's own net Sharpe 0.3382 → 0.2298 and its NW t to 1.267, failing theb_nw_t_ge_1p5check it pre-registered; the artifact records verdict KILLED. It holds a quarter ofBOOK_WEIGHTS, so the book fell with it the same day.Attribution checked rather than assumed: the other three legs feed from curves untouched since June, and
portfolio/book.pyhas not changed since 08-04. The monitor's own history confirms the shape — 1.82 on every run through 08-15, then 1.78 on 08-16, 08-18, 08-19 and 08-20. A step with a cause, not a drift.Both moved down. A correction that costs us is the only direction that earns a re-pin under the standing rule in that file.
The thing that was actually worst
C7b-suitereportedexit=124 TIMEOUT after 900son every scheduled run for eight days, and it was read as "the suite is slow".The suite completes in ~460s standalone. Every timeout was a scheduled 03:10 run; every daytime standalone run in the same window passed. The one daytime timeout,
2026-08-18T10:38:22Z, is the exception that confirms it —live_publish.logrecords a full publish ceremony starting that same second. The cause is contention with the 02:10 nightly publish on a laptop free to idle-sleep mid-run, and a subprocess timeout is wall clock: a sleeping machine spends the budget without spending any CPU.The timeout hid two real failures for eight days.
test_lint_debt_contracthad two assertions failing that whole time and nobody could see them, because a timeout reports how long we waited and never what broke.Budget raised to 2700s, run wrapped in
caffeinate -i, and elapsed time now recorded in the observation on every run so creep toward the new budget is visible long before it trips again.Verified
ruff check src/alphaforge testsmypy --stricthealth_check.py --litecanlicapital.com/paper-state.jsoncarries the restored wording, in-sample 1.78 / core 1.38Whether AlphaVintage keeps its quarter of the book remains an open allocation decision. These constants track what is published; they do not endorse it.
Added since this PR was opened
The site published "371 days" of a 47-day chain.
transparency_log.jsongains an entry per publish and the tick publishes hourly, soentry_countran ~8x the dates covered — andopen.htmlrendered it into a slot labelled "days", beside a live book 13 days old. It grew by one every hour, and it had been recorded once before at 239 and read as a day count both times. The exporter now emits deriveddistinct_days/first_date/last_date; the site fix is canlicapital#5. The guard reads the site, not the JSON — the data was never wrong, so assertingdistinct_daysexists would not have caught it. (app.canlicapital.com was already correct: it labels the same field "SEALED ENTRIES".)The engineering-quality claim was a screenshot.
export_lint_debt_contract.pyran in exactly one place —.github/workflows/ci.yml, where its output is checked non-empty and thrown away with the runner.research_export.pymerely copies the file to both hosts, so the publishedPRODUCTION_AND_TESTS_CLEANclaim was bound tosource_sha256values only as fresh as the last hand-run. Now rebuilt in both pipelines beforeresearch_export, with the edge declared inEDGESand mutation-tested.live_tick.shalready documented this exact lesson one paragraph above, for the lineage audit: an artifact nothing regenerates is a screenshot, not a check. Second occurrence.var/loghad no rotation and had reached 284MB, one file at 188MB. Copytruncate rather than rename — the tick scripts hold their log open for a whole run, so renaming would leave a live writer appending to an invisible inode and swallow that tick's transcript. Verified in a sandbox with an fd held across the rotation. Runs as step 0 of the nightly and cannot fail the publish. First run reclaimed 285MB → 107MB.And one correction to my own work in this PR: the new label guard originally asserted
distinct_days < entry_count. True today, butlive_publish.shexplicitly contemplates dropping to daily publishing, which would have reddened the suite on a deliberate config change — for a property that isn't the one under test. The equal case now skips with a reason; the label assertions are untouched.Verified end-to-end
The 16:25 tick ran the edited
live_tick.shin production: exporter step clean, all three copies of the contract hash identically, and the published contract now carries the currenthealth_check.pysource hash. Full local suiteexit=0in 477s.