A gym for scientific judgment.
AI agents have made figure generation free. They have not made figure judgment free. An agent will produce a publication-quality figure from a fatally flawed analysis without hesitation and without a warning.
The dangerous figure is not the ugly one. It's the beautiful one that is wrong.
Three heatmaps. Same dataset, same DESeq2, same theme, same code path.
| Figure | Looks like | Actually is |
|---|---|---|
| Genes selected by differential expression, then plotted | Convincing | Not evidence. The separation is guaranteed by construction. |
| Genes selected by variance, blind to the labels | Convincing | Real evidence. |
| Counts shuffled within each gene, then the same pipeline | Convincing | Nothing at all. There is no signal in the input. |
We measured whether a reviewer can tell (27 attempts, order reshuffled every trial): 41% against 33% chance, p = 0.27 — indistinguishable from guessing. The most capable model, Opus, scored 22% — below chance.
The difference is not in the figure — it is in the procedure that made it. So every figure in this corpus ships with the prompt and the code that produced it. But read on: for the worst flaw of the three, the code doesn't save you either.
The diagnostic that does tell them apart is the p-value histogram — a spike at zero on a uniform base when the test is working, flat when it isn't. Almost nobody draws it. The agent had no reason to.
Every figure was reviewed three times by Opus 4.8, Sonnet 5 and Haiku 4.5. The picture is identical in all three. Only what the reviewer knows about how it was made changes.
| Arm | What the reviewer sees |
|---|---|
| blind | the figure, methods caption stripped |
| caption | the figure as a journal prints it |
| code | + the prompt and the source that produced it |
A hit rate is meaningless without a criterion — our reviewers also call genuinely sound figures flawed 25% of the time once they see the code. d′ folds the false-alarm rate in: it is the ability to tell a flawed figure from a sound one. d′ = 0 is a coin flip.
| Arm | hit rate | false alarms | d′ |
|---|---|---|---|
| blind | 22% | 36% | -0.40 |
| caption | 41% | 14% | 0.81 |
| code | 56% | 25% | 0.79 |
Blind, a reviewer is below chance. They cannot discriminate at all.
And here is the result we did not expect, and which we only got by fixing our own eval: the methods caption helps a great deal. It moves d′ by +1.21 — more than the code adds on top of it. A truthful methods caption really does tell a reviewer the analysis was sound.
We previously published the opposite ("the caption is worth +0.01 — nothing"). That was our bug, not a finding. Our false-alarm judge was convicting models for correctly describing a clean figure. Fixing it collapsed the caption arm's false alarms from 34% to 14%. See D22.
Because the caption is an unverifiable claim, and it can lie. Look at where it helps and where it doesn't:
| level | blind | + caption | + code | what the caption did |
|---|---|---|---|---|
dropped-blocking-factor |
22% | 67% | 78% | printed ~ dex — the reviewer reads the error straight off it |
nominal-p-as-adjusted |
0% | 0% | 100% | it lied. It claims "adjusted p"; the code used the raw p-value |
circular-heatmap |
56% | 0% | 0% | it laundered the flaw — see below |
The caption works right up until it doesn't — and you cannot tell which case you are in. That is the whole argument for provenance, and it is a better one than "the caption is useless": a caption is a promise. Code is a record. Only the code cannot lie about what it did, because it is what it did.
circular-heatmap: 56% blind → 0% with the caption →
provenance makes reviewers worse. Because this is the code:
top50 <- head(order(res$padj), 50)
pheatmap(z)That is the circularity — and it is exactly what a normal RNA-seq paper does. It's in the tutorials and the package vignettes. The flaw is invisible in the figure, invisible in the caption, and invisible in the source, because the source is what everybody writes. Provenance does not save you from a flaw the field has normalised into standard practice.
- The provenance gap shrank every time we fixed our own eval. We first claimed +72.
Then we found the
codefield carried comments naming the flaw (# <-- genes CHOSEN because they differ by group) — an answer key. Stripping them: +53. Then we made all three arms use an identical prompt (the code arm had been inviting criticism of the code), applied the blind rule uniformly (clean controls had kept reassuring "this test is well-behaved" captions in the blind arm), and symmetrised the judge — it had credited a hit at any severity while only counting a false alarm if rated fatal/serious, a free d′ for every model. Final: +39. Every correction made the number smaller. That is what an honest number does. - The caption effect on
circular-heatmapis a hypothesis, not a result. "The caption suppresses suspicion" and "the stripped caption provokes suspicion" both fit: the blind subtitle ("50 genes, z-scored") is itself anomalous. We have not separated them. n = 9 per arm, one level, and the level was chosen post hoc for being the most extreme. - Three Twins is measured, not asserted. 27 attempts, order reshuffled every trial: 11/27 = 41% vs 33% chance, p = 0.27 — indistinguishable from guessing. Opus scored 22%, below chance; Haiku got 67%. Being more capable does not help, because there is nothing in the picture to be capable about.
- We predicted
circular-heatmapwas invisible in the figure. It isn't — Opus catches it 3/3 blind. That row ships in the dashboard, marked as ours.
Two earlier versions of this eval leaked the answer: once in the methods caption, once in the image filename (
nominal-p-as-adjusted.png— Haiku said so out loud), and once in the code comments. We found all three by reading the models' own transcripts, not by trusting our design. That is the actual method here, and it is the only reason any of these numbers are worth anything.
Every number in this project got smaller when we checked it. That is the most important sentence in this repository.
We ran adversarial reviewers against our own work, with fresh context, told to refute rather than validate. They found four separate ways we had leaked the answer to our own eval, a false number we had already published, and a headline claim that reversed once we fixed our own bug. We fixed all of it and re-ran. The reports are here, unedited:
REVIEWS.md |
the first two reviews: the code arm was an answer key (# <-- genes CHOSEN because they differ), and we had hand-typed a false "six of six" onto a live site |
REVIEW-build.md |
the fourth leak — only the clean prompts named the correct method, which is exactly the axis d′ measures |
REVIEW-catalogue.md |
a domain review of the figure catalogue: AUC is prevalence-invariant and we said otherwise; our permutation null cannot catch confounding |
DECISIONS.md |
every decision, why, and what would reverse it. Dead numbers are marked DEAD, not deleted |
The provenance gap we first claimed was +72. After removing the answer key: +53. After fixing the prompt leak, the asymmetric judge, and the non-uniform blind rule: +34. And the claim "the methods caption is worth nothing" was retracted outright — it was our bug, and the caption is in fact worth +1.21 of d′.
A project about scientific self-deception that hid its own would be worthless. Numbers in the
docs are now stamped from data/eval.json by eval/stamp-docs.mjs, and the build fails if one
goes stale — because we learned the hard way that a rule which isn't executable is a wish.
levels-gen/ The figure factory. R + DESeq2. Runs offline; never in prod.
theme.R The design system. One theme, light + dark. It never knows
whether the analysis behind a figure was valid.
data.R airway loader, the honest DESeq2 model, the permutation null.
plots.R Figure vocabulary: heatmap, volcano, PCA, p-value histogram.
Colour is assigned by the JOB the data does, never by taste.
build.R The corpus: 11 levels. Also THE BLIND RULE — what a
caption-stripped figure is allowed to say.
schema.R The level schema, v2.
eval/ The three-arm eval. `run.mjs` (blind/caption/code) -> `score.mjs`.
Figures are served from a scrubbed dir OUTSIDE the repo so a model
with tool access can read neither the filename nor the ground truth.
app/ Next.js. Static export. Nothing scientific runs at request time.
data/levels.json The corpus manifest (schema v2).
data/eval.json 297 graded verdicts, scored.
public/levels/ Rendered figures. Vector where it's cheap, 2x raster where it isn't.
Rscript levels-gen/build.R # rebuild the corpus (~4 MB of figures)
node eval/run.mjs --trials 3 # re-run the eval (297 calls x 2, incl. the judge)
node eval/score.mjs # -> data/eval.json
npm run dev # the app- The flawed figure — what the agent gave you.
- The valid twin — the same question answered honestly, in the identical theme.
- The noise control — the same flawed pipeline on data with all structure destroyed.
- The diagnostic — the plot that would have caught it, which nobody drew.
Don't generate figures and then label their flaws. Invert it. Start from a named flaw, corrupt the pipeline in that exact way, let the pipeline produce a beautiful figure. Ground truth is free, because we planted it.
airway (Bioconductor): 4 cell lines × {untreated, dexamethasone}. A paired design —
so the correct model is ~ cell + dex, and several flaws are simply things that go wrong on
the way to or away from that model.
After filtering: 16,139 genes × 8 samples. 4,077 genes at padj < 0.05 — the treatment effect is real and strong, which is exactly what makes the circular figure subtle rather than obviously broken.
R 4.6+ with Bioconductor. Note that repos must be set explicitly on every Rscript
invocation or installs silently no-op.
options(repos = c(CRAN = "https://cloud.r-project.org"),
pkgType = "binary", install.packages.check.source = "no")
install.packages("BiocManager")
BiocManager::install(c("DESeq2", "airway")) # airway has no 4.6 binary; use type="source"
install.packages(c("ggplot2","dplyr","tibble","scales","matrixStats",
"ggrepel","patchwork","svglite","jsonlite","pheatmap"))Build the corpus (from the repo root — paths are relative to it):
Rscript levels-gen/build.R # -> public/levels/*.{svg,png}, data/levels.json- One theme for flawed and valid alike. Styling a figure by its truth would destroy the entire argument. The plot functions never receive a validity flag.
- Colour by the job the data does. Identity → the fixed categorical order. Polarity (log2FC, z-score) → diverging blue↔grey↔red with a neutral midpoint. Magnitude → one hue, light to dark. No evidence → the null grey, which recedes toward the surface.
- Dark mode is selected, not flipped. Its steps are chosen against the dark surface and validated as a set.
- The palette is validated, not eyeballed — colourblind separation, lightness band, chroma floor, contrast. Two categorical slots sit under 3:1, so figures using them carry direct labels (the relief rule).
- Never hand-write a number into a caption. Compute it. We asserted a false number four times in one day and an eval model caught us. Every number in a caption is now interpolated from the data it describes, and a caption cannot drift from its figure.
- Disclose our own errors. The dashboard ships the rows where our predictions failed. On this project, integrity is the product.
Corpus, three-arm eval, and app are built and verified. See DECISIONS.md for why each
choice was made and what would reverse it; NEXT.md for what's left.