fix(publish): AlphaVintage is published as validated while its own artifact says KILLED - #5
Open
arhancanli wants to merge 2 commits into
Open
fix(publish): AlphaVintage is published as validated while its own artifact says KILLED#5arhancanli wants to merge 2 commits into
arhancanli wants to merge 2 commits into
Conversation
…nt rewrite INCIDENT. Eight commits made between 22:35 and 23:10 were pushed to origin/main and are no longer in it. The reflog shows the mechanism: 34b92b5 23:10:31 commit: docs: record which hosts this machine cannot reach 26b6781 23:17:04 checkout: moving from feat/status-coverage-preflight-v4 to main 9753775 23:17:04 merge origin/main: Fast-forward A fast-forward cannot drop commits that are in origin/main, so origin/main was reset or force-pushed to a history that never contained them, and the local branch then fast-forwarded onto that history and lost them too. Dropped: the overlay realized-leg fix, its observability counter, the frontier status note, the stressed-correlation analysis, the 21-year family proxies, the AlphaVintage correction, the README badges, and the egress-blocked-hosts note. The damage was PARTIAL, which is worse than total, because the working tree looked fine. strategy.py kept `_scale_hist` and the de-levering (they were carried along inside a later commit of the whole file) while the counter that makes the fix observable vanished, and two files disappeared entirely. A tree that is 80% recovered reads exactly like a tree that is fine. Recovery is file-level, not a cherry-pick, so nothing in the other lane is clobbered: two files restored from their commits, the badges and frontier note re-applied against current content, and the counter re-applied edit-by-edit with each anchor asserted unique. HOW THIS SURFACED, which is the part worth keeping: the full suite finished with five failures, and three were mine. The counter key set is asserted EXACTLY in three places -- two in test_phase6_observability and one in test_walkforward -- so adding a counter is supposed to fail loudly and be acknowledged. It did its job. I missed it before committing because I ran a FILTERED subset (-k "overlay or strategy or portfolio or ...") and none of those words appear in "test_phase6_observability" or "test_walkforward". The filter, not the tests, was the defect. Those three key sets now include realized_leg_bound. Also regenerated artifacts/engineering/lint_debt_contract.json, whose persisted source hashes had drifted (pyproject.toml among them, which is not mine). STILL FAILING AND NOT MINE: tests/integration/test_scale_guard.py:: test_engine_scale_memory_and_time_guard raises AttributeError: 'NoneType' object has no attribute 'pause' -- an environment/fixture issue, untouched by any of this. And the standing C7b-suite red is now diagnosed: the suite is neither hung nor broken, it takes roughly 50 minutes against health_check's 900s cap, so a TIMEOUT is being reported as a FAIL. "Broken" and "unknown" need different responses and currently render identically.
…tifact says KILLED
THE DEFECT. AlphaVintage was deployed 2026-08-10 at a quarter of the book on a
verdict of ADD: net Sharpe 0.3403, Newey-West t 1.82. On 2026-08-16 its probe was
re-run with a calendar correction retaining zero-exposure sessions between first
and last activity. Net Sharpe fell to 0.2298, Newey-West t to 1.267, the
pre-registered check `b_nw_t_ge_1p5` flipped to FALSE, and
artifacts/probe/cpi_surprise_size/result.json recorded `verdict: KILLED`.
The published figures never moved. For three days canlicapital.com described a
killed candidate, one that FAILED a gate it pre-registered, as a validated
25%-weight live sleeve. The sleeve was deployed six days before the corrected run
existed and the page was never revisited.
WHY NOTHING CAUGHT IT. The numbers were hard-coded literals in
scripts/paper_trading_state.py -- and in TWO places, the algorithms block and the
book.sleeves block, which is exactly how one of them survived correction. The
retraction blocklist only catches a claim after someone has decided to retract it.
No test bound a published number to the artifact it came from. So the failure was
invisible by construction, which is the same shape as every other defect this
project has had to publish.
WHAT THIS CHANGES.
- Both hard-coded copies now carry the artifact's value (0.23, not 0.34).
- The sleeve caveat, the algorithms description and the long-form transparency
entry state the withdrawal explicitly: what we published, what it measures
now, that the t FAILS its pre-registered 1.5 bar, that the artifact says
KILLED, and that we deployed before the correction existed.
- docs/retracted_claims.txt gains three rules so 0.3403, "Newey-West t of 1.82"
and "net Sharpe 0.34 with" cannot be asserted again without their retraction
beside them. Running the gate now FAILS on the already-built site bundles,
which is correct -- it clears when the publish pipeline regenerates.
WHAT THIS DOES NOT DECIDE. Whether AlphaVintage should keep carrying a quarter of
the book is an allocation decision and it stays open. The copy says so rather
than quietly implying either answer. What is not open is describing a killed
candidate as a validated one while that decision is pending.
THE STRUCTURAL FIX, which matters more than the number:
tests/unit/test_published_sleeve_claims_track_artifacts.py binds published sleeve
claims to their source artifacts. It asserts a published Sharpe is not a value the
artifact itself labels superseded, that a sleeve whose artifact says KILLED
discloses that, and that FALSE pre-registered checks are disclosed. It carries a
can-this-fail test and a test that every mapped artifact exists, so the cases
cannot go vacuously green.
It earned its place immediately: on its first run it found the SECOND hard-coded
0.34 at paper_trading_state.py:1575 that I had missed while fixing the first.
Also regenerated artifacts/engineering/lint_debt_contract.json for the new test
file and synced the published copies, which the freshness test pins byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A live false claim on canlicapital.com.
AlphaVintage was deployed 2026-08-10 at 25% of the book on verdict ADD (net Sharpe 0.3403, NW t 1.82). On 2026-08-16 its probe re-ran with a calendar correction: net Sharpe 0.2298, NW t 1.267, check
b_nw_t_ge_1p5FALSE,verdict: KILLED.The published figures never moved. For three days the site described a killed candidate — one that failed a gate it pre-registered — as a validated live sleeve.
Why nothing caught it
The numbers were hard-coded literals, in two places (algorithms block and book.sleeves), which is how one survived correction. The retraction blocklist only catches a claim after someone decides to retract it. No test bound a published number to its artifact. Invisible by construction.
Changes
0.3403,Newey-West t of 1.82andnet Sharpe 0.34 withcannot be re-asserted bare. The gate now fails on already-built site bundles — correct, and clears on next publish.Not decided here
Whether AlphaVintage keeps its quarter of the book. That is an allocation decision and stays open; the copy says so rather than implying either answer.
The structural fix
tests/unit/test_published_sleeve_claims_track_artifacts.pybinds published sleeve claims to source artifacts — no superseded values published, KILLED verdicts disclosed, FALSE pre-registered checks disclosed. Carries a can-this-fail test and an artifact-existence test so nothing goes vacuously green.It found the second hard-coded 0.34 on its first run, which I had missed.
53 publish-pipeline tests pass.