Scaffold schema v2

.env.example

@@ -0,0 +1,12 @@
+# Example secrets file. Copy to `.env` and fill in real values.
+# Real `.env` is gitignored. On Alpine, prefer sourcing `~/.fmharness/secrets`
+# in sbatch headers rather than placing a `.env` on the cluster.
+
+# HuggingFace personal access token (for Tahoe-x1 weights)
+HUGGINGFACE_TOKEN=
+
+# Synapse personal access token (for Soragni 2024 download)
+SYNAPSE_PAT=
+
+# Optional override of the Alpine scratch path used for tranche caching
+FMHARNESS_SCRATCH=

.github/workflows/ci.yml

@@ -0,0 +1,26 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+      - name: Install Python 3.11
+        run: uv python install 3.11
+      - name: Sync project
+        run: uv sync --extra dev
+      - name: Ruff lint
+        run: uv run ruff check .
+      - name: Ruff format check
+        run: uv run ruff format --check .
+      - name: Pyright
+        run: uv run pyright
+      - name: Pytest
+        run: uv run pytest --cov=src/fmharness --cov-report=term-missing

.gitignore

@@ -216,3 +216,10 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+
+# fmharness project-specific
+data/tranches/        # cached tranche artifacts (large)
+reports/              # generated reports
+containers/*.sif      # built Apptainer images
+.fmharness/           # local user secrets/state
+

.pre-commit-config.yaml

@@ -0,0 +1,31 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.4.10
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml
+      - id: check-merge-conflict
+      - id: detect-private-key
+      - id: check-added-large-files
+        args: [--maxkb=1024]
+
+  - repo: https://github.com/RobertCraigie/pyright-python
+    rev: v1.1.367
+    hooks:
+      - id: pyright
+        pass_filenames: false
+        additional_dependencies:
+          - pydantic>=2.7
+          - numpy>=1.26
+          - pandas>=2.2
+          - hypothesis>=6.100
+          - pytest>=8.0

README.md

@@ -1,2 +1,34 @@
 # fm-pdo-evaluator
- Realizing the benefits of foundation models requires careful evaluations that map the boundaries of generalization.
+
+Foundation-model evaluation harness for patient-derived tumor organoid (PDTO) drug-response prediction. Realizing the benefits of foundation models requires careful evaluations that map the boundaries of generalization.
+
+The harness produces a registry-backed report comparing three models against three out-of-distribution split strategies on two PDTO datasets, with negative/positive controls, bootstrap confidence intervals, and a pretraining-leakage exposure profile per result.
+
+See [docs/fm-pdo-evaluator-plan.md](docs/fm-pdo-evaluator-plan.md) for the 3-week plan.
+
+## Quickstart
+
+```bash
+# Install uv (Python package manager) if you don't already have it
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Sync dependencies (creates .venv and uv.lock)
+uv sync --extra dev
+
+# Run the tests
+uv run pytest
+```
+
+## Datasets
+
+- **Soragni 2024** sarcoma PDTOs ([Synapse PDTOSarcoma](https://www.synapse.org/PDTOSarcoma))
+- **Yang 2024** primary liver cancer PDOs ([Cancer Cell](https://www.cell.com/cancer-cell/fulltext/S1535-6108(24)00089-8))
+
+
+## Affiliation
+
+Greene Laboratory, University of Colorado Anschutz Medical Campus.
+
+## License
+
+BSD-2-Clause Plus Patent License (see [LICENSE](LICENSE)).

containers/fmharness.def

@@ -0,0 +1,37 @@
+Bootstrap: docker
+From: python:3.11-slim
+
+%labels
+    org.opencontainers.image.title fmharness
+    org.opencontainers.image.description Foundation-model evaluation harness for PDTO drug response
+    org.opencontainers.image.licenses BSD-2-Clause-Patent
+
+%files
+    pyproject.toml /opt/fmharness/pyproject.toml
+    uv.lock /opt/fmharness/uv.lock
+    src /opt/fmharness/src
+    README.md /opt/fmharness/README.md
+    LICENSE /opt/fmharness/LICENSE
+
+%post
+    set -eux
+    apt-get update
+    apt-get install -y --no-install-recommends git curl ca-certificates
+    rm -rf /var/lib/apt/lists/*
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+    export PATH="/root/.local/bin:${PATH}"
+    cd /opt/fmharness
+    uv sync --frozen --no-dev
+
+%environment
+    export PATH="/root/.local/bin:/opt/fmharness/.venv/bin:${PATH}"
+    export PYTHONDONTWRITEBYTECODE=1
+    export CUBLAS_WORKSPACE_CONFIG=:4096:8
+
+%runscript
+    exec "$@"
+
+%help
+    Skeleton Apptainer image for the harness. The Tahoe-x1 and STATE
+    wrappers extend this image (containers/tahoe.def, containers/state.def)
+    with the CUDA/torch versions matching the Alpine GPU driver.

docs/adapter_contract.md

@@ -0,0 +1,124 @@
+# Adapter contract
+
+How a model joins the harness. Every model — linear baseline, Tahoe-x1, STATE, and any future addition — implements the same `ModelAdapter` Protocol so the rest of the pipeline (splits, probe, metrics, registry, leakage scan) is model-agnostic.
+
+The contract has three required surfaces (`embed`, `metadata`, `version`) and one optional surface (`predict_native`). Day 7 implements the Protocol and the `linear_baseline` and `MockAdapter` reference implementations; later days add `tahoe_x1` (Day 8) and `state` (Day 11) against the same contract.
+
+## 1. The interface
+
+```python
+from typing import Protocol, runtime_checkable
+import numpy as np
+from anndata import AnnData
+
+
+@runtime_checkable
+class ModelAdapter(Protocol):
+    """Wrap a foundation model (or baseline) so the harness can use it."""
+
+    def version(self) -> str:
+        """Stable identifier, e.g. `[email protected]`. Embedded in PredictionRecord."""
+
+    def metadata(self) -> "ModelMetadata":
+        """Pretraining provenance for the leakage scan."""
+
+    def embed(self, adata: AnnData) -> np.ndarray:
+        """Encode samples (rows of `adata`) into a dense embedding matrix.
+
+        Returns an array of shape ``(adata.n_obs, embedding_dim)``. Rows are
+        aligned with ``adata.obs_names``. The harness owns input ordering;
+        adapters must not reorder rows.
+        """
+
+    def predict_native(
+        self, adata: AnnData, drug_ids: list[str]
+    ) -> np.ndarray | None:
+        """Optional native drug-aware head. Return ``None`` if the adapter
+        does not expose one. Shape: ``(adata.n_obs, len(drug_ids))``.
+        """
+```
+
+`ModelMetadata` (a pydantic model declared in `schema/`):
+
+| Field | Required | Purpose |
+|---|---|---|
+| `pretraining_corpus` | yes | Free-form name (`"tahoe_100m"`, `"none"` for baselines) |
+| `pretraining_cutoff_date` | yes | ISO date the corpus was frozen; used to flag dataset-leakage risk |
+| `task_signal_in_pretrain` | yes | One of `"none"`, `"adjacent"`, `"direct"` — declares whether the corpus contained drug-response labels |
+| `model_weights_hash` | yes for FM models | sha256 of the loaded checkpoint, captured into `EnvironmentSnapshot` |
+| `container_digest` | yes for FM models | Apptainer image digest the adapter expects to run inside |
+
+## 2. The probe-based prediction pipeline (default path)
+
+All four matrix rows (linear baseline, Tahoe-x1, STATE, plus the metadata-only control) share an identical probe so the comparison isolates "what the encoder captures":
+
+```
+  sample (RNA-seq)  --[ encoder ]-->  embedding  --[ concat drug feat ]-->  [ probe ]  --> P(responder)
+                                                                              ^
+                                                                              |
+                                                                       trained per split-fold
+```
+
+- **Encoder** is model-specific. For the linear baseline the encoder is `StandardScaler` (a passthrough; "embedding" == scaled expression). For Tahoe-x1 / STATE it is the pretrained transformer encoder.
+- **Drug feature** is a one-hot over the drug crosswalk's canonical IDs at MVP; richer drug descriptors (Morgan fingerprint, ATC class) are a deferred extension.
+- **Probe** is a fixed architecture across all models: `StandardScaler → ElasticNetCV` (continuous response) or `LogisticRegressionCV` (binary responder). Declared once in `src/fmharness/probe/linear.py`. The harness — not the adapter — owns the probe.
+
+The adapter's only job is to produce a faithful embedding. Probe training and inference are downstream.
+
+## 3. The native-head path (optional)
+
+Foundation models with a drug-aware head can return a prediction directly:
+
+- Tahoe-x1: trained on perturbation-response prediction; may expose a `predict(baseline_state, drug) → post_state` or scalar head.
+- STATE: the ST (state transition) component is exactly this.
+
+When `predict_native` returns a value, the harness records it as a separate row in the registry tagged `prediction_mode="native"`. The probe-based row (`prediction_mode="probe"`) is always produced for fair comparison; the native row is supplementary.
+
+Adapters that do not expose a native head return `None` and only the probe path runs.
+
+## 4. Required behaviors
+
+Every adapter must:
+
+1. **Be deterministic.** Calling `embed` twice on the same input must produce identical output bits-for-bits when `fmharness.utils.determinism.fix_seeds(seed)` has been called. The determinism check on Day 14 will fail loud otherwise.
+2. **Not reorder rows.** `adata.obs_names[i]` must correspond to `embedding[i]`.
+3. **Declare its container.** GPU adapters run inside the Apptainer image whose digest is returned by `metadata().container_digest`. The harness refuses to record a `PredictionRecord` whose `EnvironmentSnapshot.container_digest` does not match.
+4. **Cache by content.** Embedding caches under `data/tranches/{tranche_id}/embeddings/{model_version}/` are keyed by sha256 of the input AnnData bytes plus `model_version`. Adapters should call into `fmharness.utils.cache` rather than rolling their own.
+5. **Gracefully refuse mismatched inputs.** If the input gene panel does not match the adapter's expected reference, raise `GenePanelMismatch` rather than silently aligning — gene-panel reconciliation is the loader's responsibility (Day 4), not the adapter's.
+
+## 5. Adding a new model
+
+1. Build / pick the Apptainer image (`containers/<name>.def`). Pin the digest in `containers/digests.json`.
+2. Create `src/fmharness/models/wrappers/<name>.py` implementing `ModelAdapter`.
+3. Register the adapter in `src/fmharness/models/registry.py` so the CLI's `--model <name>` flag resolves.
+4. Add `configs/<name>_{soragni,yang}_{id,lpo,lso}.yaml` (6 files) following the existing pattern.
+5. Add a smoke test under `tests/wrappers/test_<name>.py` using the `MockAdapter` round-trip pattern.
+
+## 6. The metadata-only control adapter
+
+The metadata-only baseline is implemented as a `ModelAdapter` too — it lives in `src/fmharness/controls/negative.py` and the same probe is applied. Its `embed()` returns the one-hot concatenation of tissue, subtype, and drug ID; its `metadata()` declares `pretraining_corpus="none"` and `task_signal_in_pretrain="none"`. The leakage scan reports zero overlap for metadata-only rows by construction.
+
+## 7. Reference: the linear baseline
+
+```python
+class LinearBaselineAdapter:
+    def version(self) -> str:
+        return "[email protected]"
+
+    def metadata(self) -> ModelMetadata:
+        return ModelMetadata(
+            pretraining_corpus="none",
+            pretraining_cutoff_date=date(1970, 1, 1),
+            task_signal_in_pretrain="none",
+        )
+
+    def embed(self, adata: AnnData) -> np.ndarray:
+        # The "encoder" is identity; downstream StandardScaler in the probe
+        # handles per-feature standardization.
+        return np.asarray(adata.X, dtype=np.float32)
+
+    def predict_native(self, adata, drug_ids):
+        return None
+```
+
+Anything more elaborate than this for the baseline (e.g., PCA, per-gene z-score, gene selection) belongs in the probe pipeline, not the adapter.

docs/environment.md

@@ -0,0 +1,98 @@
+# Environment contract
+
+This document describes the harness's environment-control contract: which
+parts of the runtime must be reproducible, how each is pinned, and how the
+provenance is captured in every prediction record. Without these guarantees,
+the determinism check is half-blind to node-to-node
+drift and the leakage scan can't be tied to a specific code/data state.
+
+## 1. Containers
+
+Foundation-model inference runs inside Apptainer images pinned by digest in
+`containers/digests.json`:
+
+| Image | Definition | Built on | Wraps |
+|---|---|---|---|
+| `fmharness` | `containers/fmharness.def` | Day 1+2 (skeleton); rebuilt Day 8 | core Python deps |
+| `tahoe` | `containers/tahoe.def` | Day 8 | Tahoe-x1 + torch + CUDA |
+| `state` | `containers/state.def` (built only if needed) | Day 11 | STATE + torch + CUDA |
+
+STATE reuses the Tahoe container unless torch/CUDA conflicts force a split;
+the decision (and the reason) is recorded on Day 11 in this document.
+
+Every `PredictionRecord` carries `EnvironmentSnapshot.container_digest`. A
+prediction made outside a pinned container fails determinism check #6.
+
+## 2. Deterministic GPU execution
+
+Every CLI entrypoint calls `fmharness.utils.determinism.fix_seeds(seed)`
+before importing torch CUDA functionality. That call:
+
+- seeds `random`, `numpy`, `torch`, `torch.cuda`
+- sets `PYTHONHASHSEED`
+- sets `CUBLAS_WORKSPACE_CONFIG=:4096:8` (required for cuBLAS determinism)
+- calls `torch.use_deterministic_algorithms(True)`
+
+The `CUBLAS_WORKSPACE_CONFIG` env var must be set before CUDA initializes,
+so `fix_seeds` is unsafe to call after model load. Callers that load torch
+modules at import time must invoke `fix_seeds` before any such import.
+
+`EnvironmentSnapshot.cuda_deterministic` records whether this contract was
+active when the prediction ran. Set it to `True` only if `fix_seeds` was
+called and `torch.are_deterministic_algorithms_enabled()` returned `True`.
+
+## 3. Secrets
+
+Two secrets the harness needs are kept out of the repo:
+
+- `HUGGINGFACE_TOKEN` — Tahoe-x1 weights download
+- `SYNAPSE_PAT` — Soragni 2024 dataset access
+
+Layout:
+
+- Local dev: copy `.env.example` to `.env`, fill in values. `.env` is
+  gitignored. Loaded by `pydantic-settings` from the repo root.
+- Alpine: `~/.fmharness/secrets` with `chmod 600`. Slurm sbatch headers
+  source it explicitly: `source ~/.fmharness/secrets`.
+
+`.env` and any path containing `secrets` are caught by the pre-commit
+`detect-private-key` hook and a project-specific token-pattern check
+(added Day 13 when secrets handling is fully exercised).
+
+## 4. Static asset versioning
+
+Static assets (the Tahoe-100M drug list, drug crosswalk tables, gene-panel
+reconciliation tables, reference FASTA + GTF used by the RNA quantification
+pipeline) live under `data/static/` with a `manifest.json` recording sha256
+per file. Loaders verify on read and refuse to proceed on mismatch.
+
+The manifest's hash of the reference genome + GENCODE annotation propagates
+into `EnvironmentSnapshot.data_commit` via the tranche content hash, so a
+mismatch is detectable from any prediction record.
+
+## 5. Pinned Python environment
+
+`uv.lock` (committed) pins every direct and transitive Python dependency.
+CI installs from the lock; Alpine inference jobs install from the lock
+inside the Apptainer build (`uv sync --frozen --no-dev`).
+
+When upgrading a dependency:
+
+1. Edit `pyproject.toml`.
+2. Run `uv lock` to refresh `uv.lock`.
+3. Rebuild any Apptainer image whose `%files` includes the lock.
+4. Commit both files in the same PR.
+
+## 6. `EnvironmentSnapshot` field reference
+
+| Field | Source | Required for det. check |
+|---|---|---|
+| `code_commit` | `git rev-parse HEAD` at run time | yes |
+| `container_digest` | `apptainer inspect --digest <image>` | yes (GPU runs) |
+| `python_version` | `sys.version` | yes |
+| `torch_version` | `torch.__version__` if torch available | yes (GPU runs) |
+| `cuda_version` | `torch.version.cuda` if CUDA available | yes (GPU runs) |
+| `model_weights_hash` | sha256 of the loaded checkpoint bytes | yes |
+| `data_commit` | tranche `content_hash` for the inputs | yes |
+| `seed` | seed passed into the CLI | yes |
+| `cuda_deterministic` | `True` iff `fix_seeds` ran + det algos enabled | yes |