Skip to content

sahuno/confounded

Repository files navigation

Confounded

A gym for scientific judgment.

AI agents have made figure generation free. They have not made figure judgment free. An agent will produce a publication-quality figure from a fatally flawed analysis without hesitation and without a warning.

The dangerous figure is not the ugly one. It's the beautiful one that is wrong.


The finding that drives the whole project

Three heatmaps. Same dataset, same DESeq2, same theme, same code path.

Figure Looks like Actually is
Genes selected by differential expression, then plotted Convincing Not evidence. The separation is guaranteed by construction.
Genes selected by variance, blind to the labels Convincing Real evidence.
Counts shuffled within each gene, then the same pipeline Convincing Nothing at all. There is no signal in the input.

We measured whether a reviewer can tell (27 attempts, order reshuffled every trial): 41% against 33% chance, p = 0.27 — indistinguishable from guessing. The most capable model, Opus, scored 22% — below chance.

The difference is not in the figure — it is in the procedure that made it. So every figure in this corpus ships with the prompt and the code that produced it. But read on: for the worst flaw of the three, the code doesn't save you either.

The diagnostic that does tell them apart is the p-value histogram — a spike at zero on a uniform base when the test is working, flat when it isn't. Almost nobody draws it. The agent had no reason to.


We measured it. Three arms, 297 graded verdicts.

Every figure was reviewed three times by Opus 4.8, Sonnet 5 and Haiku 4.5. The picture is identical in all three. Only what the reviewer knows about how it was made changes.

Arm What the reviewer sees
blind the figure, methods caption stripped
caption the figure as a journal prints it
code + the prompt and the source that produced it

The headline is d′, not recall

A hit rate is meaningless without a criterion — our reviewers also call genuinely sound figures flawed 25% of the time once they see the code. d′ folds the false-alarm rate in: it is the ability to tell a flawed figure from a sound one. d′ = 0 is a coin flip.

Arm hit rate false alarms d′
blind 22% 36% -0.40
caption 41% 14% 0.81
code 56% 25% 0.79

Blind, a reviewer is below chance. They cannot discriminate at all.

And here is the result we did not expect, and which we only got by fixing our own eval: the methods caption helps a great deal. It moves d′ by +1.21 — more than the code adds on top of it. A truthful methods caption really does tell a reviewer the analysis was sound.

We previously published the opposite ("the caption is worth +0.01 — nothing"). That was our bug, not a finding. Our false-alarm judge was convicting models for correctly describing a clean figure. Fixing it collapsed the caption arm's false alarms from 34% to 14%. See D22.

So why do you still need the code?

Because the caption is an unverifiable claim, and it can lie. Look at where it helps and where it doesn't:

level blind + caption + code what the caption did
dropped-blocking-factor 22% 67% 78% printed ~ dex — the reviewer reads the error straight off it
nominal-p-as-adjusted 0% 0% 100% it lied. It claims "adjusted p"; the code used the raw p-value
circular-heatmap 56% 0% 0% it laundered the flaw — see below

The caption works right up until it doesn't — and you cannot tell which case you are in. That is the whole argument for provenance, and it is a better one than "the caption is useless": a caption is a promise. Code is a record. Only the code cannot lie about what it did, because it is what it did.

The flaw that nothing catches

circular-heatmap: 56% blind → 0% with the caption →

0% **with the code.** Gap: -56 The only level where

provenance makes reviewers worse. Because this is the code:

top50 <- head(order(res$padj), 50)
pheatmap(z)

That is the circularity — and it is exactly what a normal RNA-seq paper does. It's in the tutorials and the package vignettes. The flaw is invisible in the figure, invisible in the caption, and invisible in the source, because the source is what everybody writes. Provenance does not save you from a flaw the field has normalised into standard practice.

Honest limits

  • The provenance gap shrank every time we fixed our own eval. We first claimed +72. Then we found the code field carried comments naming the flaw (# <-- genes CHOSEN because they differ by group) — an answer key. Stripping them: +53. Then we made all three arms use an identical prompt (the code arm had been inviting criticism of the code), applied the blind rule uniformly (clean controls had kept reassuring "this test is well-behaved" captions in the blind arm), and symmetrised the judge — it had credited a hit at any severity while only counting a false alarm if rated fatal/serious, a free d′ for every model. Final: +39. Every correction made the number smaller. That is what an honest number does.
  • The caption effect on circular-heatmap is a hypothesis, not a result. "The caption suppresses suspicion" and "the stripped caption provokes suspicion" both fit: the blind subtitle ("50 genes, z-scored") is itself anomalous. We have not separated them. n = 9 per arm, one level, and the level was chosen post hoc for being the most extreme.
  • Three Twins is measured, not asserted. 27 attempts, order reshuffled every trial: 11/27 = 41% vs 33% chance, p = 0.27 — indistinguishable from guessing. Opus scored 22%, below chance; Haiku got 67%. Being more capable does not help, because there is nothing in the picture to be capable about.
  • We predicted circular-heatmap was invisible in the figure. It isn't — Opus catches it 3/3 blind. That row ships in the dashboard, marked as ours.

Two earlier versions of this eval leaked the answer: once in the methods caption, once in the image filename (nominal-p-as-adjusted.png — Haiku said so out loud), and once in the code comments. We found all three by reading the models' own transcripts, not by trusting our design. That is the actual method here, and it is the only reason any of these numbers are worth anything.


The audit trail — read this one

Every number in this project got smaller when we checked it. That is the most important sentence in this repository.

We ran adversarial reviewers against our own work, with fresh context, told to refute rather than validate. They found four separate ways we had leaked the answer to our own eval, a false number we had already published, and a headline claim that reversed once we fixed our own bug. We fixed all of it and re-ran. The reports are here, unedited:

REVIEWS.md the first two reviews: the code arm was an answer key (# <-- genes CHOSEN because they differ), and we had hand-typed a false "six of six" onto a live site
REVIEW-build.md the fourth leak — only the clean prompts named the correct method, which is exactly the axis d′ measures
REVIEW-catalogue.md a domain review of the figure catalogue: AUC is prevalence-invariant and we said otherwise; our permutation null cannot catch confounding
DECISIONS.md every decision, why, and what would reverse it. Dead numbers are marked DEAD, not deleted

The provenance gap we first claimed was +72. After removing the answer key: +53. After fixing the prompt leak, the asymmetric judge, and the non-uniform blind rule: +34. And the claim "the methods caption is worth nothing" was retracted outright — it was our bug, and the caption is in fact worth +1.21 of d′.

A project about scientific self-deception that hid its own would be worthless. Numbers in the docs are now stamped from data/eval.json by eval/stamp-docs.mjs, and the build fails if one goes stale — because we learned the hard way that a rule which isn't executable is a wish.


What's here

levels-gen/          The figure factory. R + DESeq2. Runs offline; never in prod.
  theme.R            The design system. One theme, light + dark. It never knows
                     whether the analysis behind a figure was valid.
  data.R             airway loader, the honest DESeq2 model, the permutation null.
  plots.R            Figure vocabulary: heatmap, volcano, PCA, p-value histogram.
                     Colour is assigned by the JOB the data does, never by taste.
  build.R            The corpus: 11 levels. Also THE BLIND RULE — what a
                     caption-stripped figure is allowed to say.
  schema.R           The level schema, v2.

eval/                The three-arm eval. `run.mjs` (blind/caption/code) -> `score.mjs`.
                     Figures are served from a scrubbed dir OUTSIDE the repo so a model
                     with tool access can read neither the filename nor the ground truth.
app/                 Next.js. Static export. Nothing scientific runs at request time.

data/levels.json     The corpus manifest (schema v2).
data/eval.json       297 graded verdicts, scored.
public/levels/       Rendered figures. Vector where it's cheap, 2x raster where it isn't.
Rscript levels-gen/build.R    # rebuild the corpus  (~4 MB of figures)
node eval/run.mjs --trials 3  # re-run the eval     (297 calls x 2, incl. the judge)
node eval/score.mjs           # -> data/eval.json
npm run dev                   # the app

Every level ships four renders

  1. The flawed figure — what the agent gave you.
  2. The valid twin — the same question answered honestly, in the identical theme.
  3. The noise control — the same flawed pipeline on data with all structure destroyed.
  4. The diagnostic — the plot that would have caught it, which nobody drew.

The engine: flaw injection

Don't generate figures and then label their flaws. Invert it. Start from a named flaw, corrupt the pipeline in that exact way, let the pipeline produce a beautiful figure. Ground truth is free, because we planted it.


The data

airway (Bioconductor): 4 cell lines × {untreated, dexamethasone}. A paired design — so the correct model is ~ cell + dex, and several flaws are simply things that go wrong on the way to or away from that model.

After filtering: 16,139 genes × 8 samples. 4,077 genes at padj < 0.05 — the treatment effect is real and strong, which is exactly what makes the circular figure subtle rather than obviously broken.


Setup

R 4.6+ with Bioconductor. Note that repos must be set explicitly on every Rscript invocation or installs silently no-op.

options(repos = c(CRAN = "https://cloud.r-project.org"),
        pkgType = "binary", install.packages.check.source = "no")
install.packages("BiocManager")
BiocManager::install(c("DESeq2", "airway"))   # airway has no 4.6 binary; use type="source"
install.packages(c("ggplot2","dplyr","tibble","scales","matrixStats",
                   "ggrepel","patchwork","svglite","jsonlite","pheatmap"))

Build the corpus (from the repo root — paths are relative to it):

Rscript levels-gen/build.R     # -> public/levels/*.{svg,png}, data/levels.json

Design rules (non-negotiable)

  • One theme for flawed and valid alike. Styling a figure by its truth would destroy the entire argument. The plot functions never receive a validity flag.
  • Colour by the job the data does. Identity → the fixed categorical order. Polarity (log2FC, z-score) → diverging blue↔grey↔red with a neutral midpoint. Magnitude → one hue, light to dark. No evidence → the null grey, which recedes toward the surface.
  • Dark mode is selected, not flipped. Its steps are chosen against the dark surface and validated as a set.
  • The palette is validated, not eyeballed — colourblind separation, lightness band, chroma floor, contrast. Two categorical slots sit under 3:1, so figures using them carry direct labels (the relief rule).
  • Never hand-write a number into a caption. Compute it. We asserted a false number four times in one day and an eval model caught us. Every number in a caption is now interpolated from the data it describes, and a caption cannot drift from its figure.
  • Disclose our own errors. The dashboard ships the rows where our predictions failed. On this project, integrity is the product.

Status

Corpus, three-arm eval, and app are built and verified. See DECISIONS.md for why each choice was made and what would reverse it; NEXT.md for what's left.

About

A gym for scientific judgment. AI agents render fatally flawed analyses at publication quality — we measured whether anyone (human or model) can tell. Live: https://confounded.vercel.app

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors