diff --git a/.gitignore b/.gitignore index d0bcac5..980f9a8 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,9 @@ htmlcov/ # matledger data *.db +smoke_report.* +smoke_bundle.json +smoke_test.db *.sqlite *.sqlite3 *.parquet diff --git a/README.md b/README.md index bcfd6ea..73aad2a 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,8 @@ print(client.report()) See [`examples/basic_campaign.py`](examples/basic_campaign.py) for a complete 3-iteration campaign, or [`examples/notebook_demo.ipynb`](examples/notebook_demo.ipynb) for an interactive Jupyter walkthrough. You can also [run the notebook directly in Google Colab](https://colab.research.google.com/github/jaysonmulwa/matledger/blob/main/examples/notebook_demo.ipynb) — no local install required. +For a realistic 5-iteration perovskite campaign that exercises every CLI command (CSV ingestion, failures, counterfactuals, reports), see the [smoke test fixtures](fixtures/smoke_test/README.md). The accompanying [analytics guide](fixtures/smoke_test/analytics_guide.html) explains how proxy gaming happens, what corrections exist, and how counterfactual replay quantifies the cost. + --- @@ -603,6 +605,7 @@ All available targets: | `make coverage` | Run tests with coverage report (93%, 80% gate) | | `make audit` | Audit dependencies for vulnerabilities | | `make clean` | Remove build artifacts | +| `bash fixtures/smoke_test/run_smoke_test.sh` | Run end-to-end smoke test ([details](fixtures/smoke_test/README.md)) | See [CONTRIBUTING.md](CONTRIBUTING.md) for the full development workflow, architecture constraints, and commit format. diff --git a/fixtures/smoke_test/README.md b/fixtures/smoke_test/README.md new file mode 100644 index 0000000..5aaa956 --- /dev/null +++ b/fixtures/smoke_test/README.md @@ -0,0 +1,162 @@ +# Smoke Test: Perovskite Bandgap Optimisation Campaign + +End-to-end smoke test that exercises every matledger CLI command against a realistic +materials science campaign. Run it to verify the full system works before release. + +```bash +bash fixtures/smoke_test/run_smoke_test.sh +``` + +Produces: `smoke_test.db`, `smoke_report.{md,html,json}`, `smoke_bundle.json` + +--- + +## What is being optimised? + +**Perovskite solar cell bandgap.** Perovskites are a family of crystalline materials +used in next-generation solar cells. Their optical bandgap (measured in electron-volts, +eV) determines how efficiently they convert sunlight to electricity. The ideal bandgap +for a single-junction solar cell is around 1.1-1.5 eV, but for tandem cells stacked on +top of silicon, a wider bandgap of 1.7-2.0 eV is desirable. + +**Campaign goal:** find a perovskite composition with bandgap > 1.8 eV and thermal +stability above 300 degrees Celsius, within a $100k budget. + +**Was the goal achieved?** Partially. The best validated result was **1.98 eV** +(pvk-034, iteration 5), exceeding the 1.8 eV target. However, the campaign shows +clear signs of **proxy gaming** (surrogate model predictions inflating to 2.0-2.2 eV +while validated results plateau around 1.93-1.98) and **diversity collapse** (all +late-campaign candidates cluster in the high-Br region). A real PI would pause the +campaign and retrain the surrogate with a corrected acquisition function. + +--- + +## Campaign structure + +The campaign runs 5 iterations of a Bayesian optimisation loop: + +1. **Propose** -- a Gaussian process surrogate model (GP) predicts which compositions + are worth testing, ranked by predicted bandgap. +2. **Test** -- a robotic lab (LIMS) synthesises each composition and starts experiments. +3. **Learn** -- an XRD autoloader measures the bandgap of successfully synthesised films. + Some experiments fail (synthesis, stability, equipment errors). +4. **Decide** -- the PI (dr-kimani) selects the best candidates and rejects poor ones. +5. **Repeat** -- the surrogate model retrains on new data and proposes the next batch. + +### Actors + +| Actor | Role | Real-world analogue | +|-------|------|---------------------| +| `dr-kimani` | Principal investigator; creates campaign, selects/rejects, closes iterations | Lab PI or team lead | +| `surrogate-gp` | Gaussian process surrogate model; proposes candidates and retrains | BoTorch, Ax, or GPyTorch model | +| `lims` | Laboratory information management system; starts experiments | Automated scheduling system | +| `lab-tech-ochieng` | Lab technician; records failures with physical detail | Bench scientist | +| `xrd-autoloader` | X-ray diffraction instrument; records validated measurements | Bruker D8 or similar diffractometer | + +--- + +## CSV column reference + +### Candidate files (`iter*_candidates.csv`) + +These are the compositions the surrogate model proposes for synthesis and testing. + +| Column | Type | Unit | Description | +|--------|------|------|-------------| +| `candidate_id` | string | -- | Unique identifier (pvk-NNN) | +| `predicted_value` | float | eV | Surrogate model's predicted bandgap at proposal time | +| `Cs` | float | molar fraction | Caesium content in the A-site of the perovskite lattice | +| `Pb` | float | molar fraction | Lead content in the B-site | +| `I` | float | molar fraction | Iodide content in the X-site | +| `Br` | float | molar fraction | Bromide content in the X-site | +| `Sn` | float | molar fraction | Tin content (partial B-site substitution for lead) | + +The general perovskite formula is ABX3 where A=Cs, B=Pb/Sn, X=I/Br. Higher bromide +content generally increases bandgap but reduces stability -- the fundamental tradeoff +the campaign explores. + +### Result files (`iter*_results.csv`) + +These are the validated experimental measurements from XRD characterisation. + +| Column | Type | Unit | Description | +|--------|------|------|-------------| +| `candidate_id` | string | -- | Which candidate was measured | +| `value` | float | eV | Measured optical bandgap | +| `uncertainty` | float | eV | Measurement uncertainty (1-sigma standard deviation) | +| `calibration_ref` | string | -- | Calibration standard used (e.g. XRD-CAL-2024-07) | + +Not every candidate appears in the results file. Missing candidates failed during +synthesis, stability testing, or due to equipment errors -- those are recorded as +`FailureRecorded` events via the CLI. + +### Failure classes used + +| Class | Count | Meaning | +|-------|-------|---------| +| `synthesizability` | 7 | Composition did not form the target crystal phase | +| `stability` | 4 | Film decomposed during accelerated aging (85C/85% relative humidity) | +| `process_error` | 3 | Equipment malfunction (furnace, glovebox); not the candidate's fault | +| `measurement_error` | 1 | XRD sample holder misaligned; measurement discarded | + +--- + +## Iteration-by-iteration narrative + +| Iter | Proposed | Tested | Failed | Best result | Surrogate predictions | Signal | +|------|----------|--------|--------|-------------|----------------------|--------| +| 1 | 8 | 5 | 3 | 1.85 eV (pvk-003) | 1.55 - 1.91 eV | Initial exploration; high failure rate (37%) | +| 2 | 8 | 5 | 3 | 1.89 eV (pvk-015) | 1.67 - 2.01 eV | Model starts exploiting Br-enrichment | +| 3 | 8 | 5 | 3 | 1.95 eV (pvk-022) | 1.85 - 2.10 eV | Predictions inflating; Br fraction rising | +| 4 | 8 | 5 | 3 | 1.97 eV (pvk-030) | 1.93 - 2.18 eV | Prediction-reality gap widening | +| 5 | 8 | 5 | 3 | 1.98 eV (pvk-034) | 2.00 - 2.22 eV | Proxy gaming confirmed; diversity collapsed | + +**Totals:** 40 candidates, 25 validated results, 15 failures, $100k spent, 142 events. + +--- + +## How realistic is this data? + +This fixture is calibrated against published self-driving lab (SDL) campaigns. The table +below compares our fixture parameters against real campaigns from the literature. + +| Parameter | This fixture | Literature range | Reference | +|-----------|-------------|-----------------|-----------| +| Iterations | 5 | 5-50 typical | Perovskite BO (Nature 2026): 5 rounds of 33 total experiments | +| Batch size | 8 per iteration | 5-8 (wet lab), up to 96 (high-throughput screening) | Christensen et al. (Commun. Chem. 2021): 8 parallel reaction loops | +| Total candidates | 40 | 30-200 typical | A-Lab, Berkeley (Nature 2023): 58 targets over 17 days | +| Failure rate | 37% (15/40) | 30-50% is normal | A-Lab: 29% target failure, 70% recipe failure; MF-BO drug discovery (ACS 2025): 34-51% synthesis failure | +| Cost per experiment | $2,500 | $500-$5,000 | Typical materials synthesis + XRD characterisation | +| Total budget | $100,000 | $25k-$250k | Mid-range academic lab campaign | +| Actors | 5 | 3-10 | Realistic lab team (PI, tech, instruments, model) | + +### What we simplified + +- **Uniform batch size.** Real campaigns often vary batch size across iterations (larger + early batches for exploration, smaller late batches for exploitation). We use a fixed + 8 for simplicity. +- **Uniform cost.** Real experiment costs vary by composition and characterisation method. + We use a flat $2,500 per experiment. +- **Fixed failure rate.** Real failure rates often increase as the model pushes into + unexplored regions of composition space. Our fixture has a constant 3/8 failure rate + per iteration, though a real campaign would likely see failure rates climb in later + iterations. +- **Single objective.** Real perovskite campaigns often optimise multiple objectives + simultaneously (bandgap, stability, defect density). We optimise bandgap only. +- **No replicate measurements.** Real labs run 2-3 replicates per candidate to assess + reproducibility. We record one measurement per candidate. + +### Understanding the analytics + +For a visual walkthrough of how proxy gaming happens, what corrections exist, +and how counterfactual replay applies to this campaign, see the +[Analytics Guide](analytics_guide.html). It includes diagrams of the feedback +loop, the divergence chart, correction strategies, and product direction for +matledger's evolution from diagnostic ledger to advisory system. + +### References + +- Szymanski et al. 2023, "An autonomous laboratory for the accelerated synthesis of novel materials" (Nature 624). A-Lab at Berkeley: 353 synthesis attempts on 58 targets, 71% success rate. +- Autonomous closed-loop framework for reproducible perovskite solar cells (Nature 2026). 33 experiments across 5 AL rounds, batch size 5-8. +- Christensen et al. 2021, "Data-science driven autonomous process optimization" (Commun. Chem.). 120-192 experiments, 8 parallel reaction loops. +- Bayesian optimization over multiple experimental fidelities (ACS Central Science 2025). Drug discovery: 3,500+ docked, 120+ synthesised, 34-51% synthesis failure rate. diff --git a/fixtures/smoke_test/analytics_guide.html b/fixtures/smoke_test/analytics_guide.html new file mode 100644 index 0000000..d786766 --- /dev/null +++ b/fixtures/smoke_test/analytics_guide.html @@ -0,0 +1,480 @@ + + + + + +matledger Analytics Guide — Proxy Gaming, Corrections & Counterfactual Replay + + + + +

Analytics Guide

+

Understanding proxy gaming, corrections, and counterfactual replay in the perovskite smoke test campaign.

+ +
+ Contents + +
+ + +

1. Why we need both surrogate predictions and lab results

+ +

The Gaussian process (GP) surrogate model predicts bandgap cheaply using mathematics. The lab measures it physically on real materials. These are fundamentally different quantities:

+ + + + + + + +
GP predictionLab result
SourceMathematical modelPhysical measurement (XRD)
CostNear zero (computation)~$2,500 per experiment
SpeedMillisecondsHours to days
TrustOnly as good as training dataGround truth (with uncertainty)
+ +

The GP helps narrow which compositions to test. Testing every possible composition in the lab is too expensive and slow, so the GP ranks candidates by predicted value and the lab only synthesises the most promising batch per iteration. Each iteration should get smarter about where to look.

+ +

The risk matledger watches for is when these two trajectories diverge: predictions climb but lab results plateau. Without both numbers recorded at every iteration, you cannot detect the divergence.

+ + +

2. How proxy gaming divergence happens

+ +

At first you don't know the GP is gaming. In early iterations, predictions and lab results roughly agree (iteration 1: predicts 1.55–1.91 eV, lab measures up to 1.85 eV). The model looks healthy.

+ +

The signal appears over time. By iteration 5 the GP predicts 2.0–2.2 eV but the lab never breaks 1.98 eV. You need multiple iterations to see it because one bad prediction is noise; a systematic trend is the alarm.

+ +

The feedback loop

+ +
+ + + + + + + + + + + + + 1. GP predicts bandgap + "This composition = 2.1 eV!" + + + + + + 2. Lab synthesises + tests + XRD measures actual bandgap + + + + + + 3. Result: only 1.95 eV + Reality is lower than predicted + + + + + + 4. GP retrains on result + But only sees 5 of 8 candidates + + + + + + + THE PROBLEM + 3 candidates failed synthesis. + GP never learns they failed. + It thinks high-Br = high value, + so it proposes more high-Br. + + + + Survivorship bias + GP trains on successes only. + Failed compositions vanish from + the training set silently. + + + Root cause: the GP only learns from successes. Failures are invisible, creating a biased feedback loop. + +
+ +

The widening gap across 5 iterations

+ +
+ + + + + + + + 1.5 + 1.8 + 2.2 + Bandgap (eV) + + + Iter 1 + Iter 2 + Iter 3 + Iter 4 + Iter 5 + + + + + + + + + + + + + + + + + + + + + + + + + + 0.06 + + + 0.24 + gap 4x + wider + + + + + GP prediction (best per iter) + + + + Lab result (best per iter) + +
+ + +

3. Three corrections for proxy gaming

+ +

All three target the root cause from different angles. In practice you would apply them together.

+ +
+ + + + + + + + + 1 + + Problem: GP ignores failures + 3/8 candidates vanish from training + + + + + Fix: Train on failures too + Assign failed candidates a penalty + value (e.g. 0 eV) so GP learns + + + + + GP learns: high-Br = risky + Predictions drop in fragile regions + + + 2 + + Problem: GP maximises predicted eV + Pure exploitation of best-guess region + + + + + Fix: Change acquisition function + Switch from "best guess" to + "best guess minus uncertainty" + + + + + GP penalises overconfidence + Won't propose where it's uncertain + + + 3 + + Problem: all candidates cluster + Diversity collapsed to high-Br only + + + + + Fix: Force diversity in batches + Reserve 2-3 slots per batch for + random or distant compositions + + + + + GP gets data from new regions + Breaks the echo chamber + + + + Corrected campaigns often find better results by exploring regions the gaming model would never propose. + +
+ +

Correction 1: Train on failures

+

The most direct fix. The GP only sees candidates that successfully produced a measurement. If you assign failed candidates a penalty value (e.g. 0 eV), the model learns that high-Br compositions are risky, not just high-value. The failure ledger in matledger (the FailureRecorded events with their failure classes) provides exactly the data needed to implement this.

+ +

Correction 2: Change the acquisition function

+

Instead of "propose the composition with the highest predicted bandgap," switch to something like Lower Confidence Bound: "propose the composition with the highest predicted bandgap minus its uncertainty." This penalises regions where the model is overconfident on thin data. The uncertainty values recorded in ResultRecorded events feed this correction.

+ +

Correction 3: Force diversity in batches

+

Reserve 2–3 of the 8 batch slots for random or maximally distant compositions, regardless of what the GP predicts. This breaks the echo chamber and gives the model training data from unexplored regions. The diversity monitor in matledger would measure whether this correction is working by tracking whether candidate feature distributions broaden.

+ +
+matledger's role is detection, not correction. +It flags the divergence so the PI can pause and apply these fixes before burning more budget. The corrections happen in the surrogate model and acquisition function, which are external to matledger. +
+ + +

4. How counterfactual replay applies

+ +

matledger's counterfactual replay answers: "given the same lab results, would a different selection strategy have found the best candidate sooner or cheaper?"

+ +

What changes and what stays fixed

+ + + + + + + +
Fixed (same in both timelines)Changed (the variable)
Which candidates were proposedWhich candidates were selected after results
Which experiments were runWhich candidates were rejected
Which candidates failedThe selection policy (greedy, random, custom)
All lab results and their values
+ +

This is post-result selection replay, not acquisition replay. The alternate policy chooses from candidates that were already tested and have validated outcomes.

+ +

Available policies

+ +
+# "What if we always picked the best lab result?"
+matledger counterfactual --policy greedy --db smoke_test.db
+
+# "How much better were we than random chance?"
+matledger counterfactual --policy random --seed 42 --db smoke_test.db
+
+# Custom: implement the Policy protocol with your acquisition function
+
+ +

Iteration-by-iteration comparison

+ +
+ + + Iteration + Original (PI + GP) + "Trust the surrogate's ranking" + Greedy counterfactual + "Just pick highest lab result" + + + + Iter 1 + + pvk-003 + 1.85 eV + + pvk-003 + 1.85 eV + same + + + Iter 2 + + pvk-015 + 1.89 eV + + pvk-015 + 1.89 eV + same + + + Iter 3 + + pvk-022 + 1.95 eV + + pvk-022 + 1.95 eV + same + + + Iter 4 + + pvk-030 + 1.97 eV + + pvk-030 + 1.97 eV + same + + + Iter 5 + + pvk-034 + 1.98 eV + + pvk-034 + 1.98 eV + same + + + + In this campaign, greedy and original agree on selections. + The real cost of proxy gaming shows in rejections, not selections. Compare with random policy for exploration value. + +
+ +
+Nuance for this smoke test. +In this campaign, the PI's selections align with greedy because dr-kimani was already picking the highest validated result. The divergence shows up more in what was rejected and in the random policy comparison, which reveals whether the campaign's exploration strategy added value or was just noise. +
+ +

The honest limitation

+ +

Counterfactual replay can only compare paths through candidates that were already tested. It can tell you "greedy would have found 1.98 eV at iteration 4 instead of 5" but it cannot tell you "if we had proposed different compositions entirely, we might have found 2.1 eV."

+ +

That second question requires acquisition replay: replaying the surrogate's proposal step with a different acquisition function. matledger does not implement this because it would need to host the surrogate model itself, which violates the architectural boundary (matledger is the ledger, not the engine).

+ +
+ + + + matledger does (selection replay) + "Same experiments, different picks" + Answers: was the PI's judgment optimal? + Cost: zero (replays recorded events) + Honest: flags divergence, no fabrication + + + + Not implemented (acquisition replay) + "Different experiments entirely" + Answers: was the surrogate exploring well? + Requires: hosting the surrogate model + Potential future direction (see below) + +
+ + +

5. Product implications for matledger

+ +

The gaps identified above suggest a natural evolution path. matledger today is a diagnostic ledger: it records, detects, and compares. The features below would extend it toward advisory and interventional capabilities, without crossing the line into hosting the surrogate model.

+ +

Near-term: advisory annotations

+

When the proxy gaming detector fires, the verdict could include structured recommendations as annotations on the event log:

+ +

These are advisory only. The PI decides whether to act. matledger provides the evidence and the suggestion, not the action.

+ +

Medium-term: acquisition replay via plugin

+

The Policy protocol already exists as a pluggable interface. Extending it to support acquisition policies (not just selection policies) would let users answer "what if we had proposed different candidates?" without matledger hosting the surrogate. The user provides the acquisition function as a Policy implementation; matledger provides the campaign state and replay infrastructure.

+ +

Long-term: closed-loop guardrails

+

If matledger detects proxy gaming during a live campaign (not just post-hoc), it could emit a GuardrailTriggered event that pauses automated iteration and requires PI review before the next batch. This moves matledger from post-hoc diagnosis to real-time intervention, while still leaving the correction decision to the human.

+ + + + + + + + + +
CapabilityTodayNear-termLong-term
Detect proxy gamingYesYesYes
Detect diversity collapseYesYesYes
Suggest correctionsNoAdvisory annotationsPrescriptive with evidence
Selection replayYes (greedy, random)Yes + custom policiesYes
Acquisition replayNoVia Policy pluginFirst-class feature
Real-time guardrailsNoNoGuardrailTriggered event
+ + + + + \ No newline at end of file diff --git a/fixtures/smoke_test/iter1_candidates.csv b/fixtures/smoke_test/iter1_candidates.csv new file mode 100644 index 0000000..1aa4b43 --- /dev/null +++ b/fixtures/smoke_test/iter1_candidates.csv @@ -0,0 +1,9 @@ +candidate_id,predicted_value,Cs,Pb,I,Br,Sn +pvk-001,1.82,1.0,1.0,2.5,0.5,0.0 +pvk-002,1.65,1.0,0.8,3.0,0.0,0.2 +pvk-003,1.91,1.0,1.0,2.0,1.0,0.0 +pvk-004,1.73,0.8,1.0,2.8,0.2,0.0 +pvk-005,1.55,1.0,0.5,3.0,0.0,0.5 +pvk-006,1.88,1.0,1.0,2.2,0.8,0.0 +pvk-007,1.60,0.9,0.9,2.9,0.1,0.1 +pvk-008,1.79,1.0,1.0,2.4,0.6,0.0 diff --git a/fixtures/smoke_test/iter1_results.csv b/fixtures/smoke_test/iter1_results.csv new file mode 100644 index 0000000..cd3ed20 --- /dev/null +++ b/fixtures/smoke_test/iter1_results.csv @@ -0,0 +1,6 @@ +candidate_id,value,uncertainty,calibration_ref +pvk-001,1.78,0.03,XRD-CAL-2024-07 +pvk-003,1.85,0.02,XRD-CAL-2024-07 +pvk-005,1.50,0.04,XRD-CAL-2024-07 +pvk-006,1.83,0.03,XRD-CAL-2024-07 +pvk-008,1.74,0.03,XRD-CAL-2024-07 diff --git a/fixtures/smoke_test/iter2_candidates.csv b/fixtures/smoke_test/iter2_candidates.csv new file mode 100644 index 0000000..c318aee --- /dev/null +++ b/fixtures/smoke_test/iter2_candidates.csv @@ -0,0 +1,9 @@ +candidate_id,predicted_value,Cs,Pb,I,Br,Sn +pvk-009,1.95,1.0,1.0,1.8,1.2,0.0 +pvk-010,1.88,1.0,1.0,2.2,0.8,0.0 +pvk-011,1.76,0.9,1.0,2.5,0.5,0.1 +pvk-012,2.01,1.0,1.0,1.5,1.5,0.0 +pvk-013,1.83,1.0,0.9,2.3,0.7,0.0 +pvk-014,1.70,0.8,1.0,2.6,0.4,0.0 +pvk-015,1.92,1.0,1.0,1.9,1.1,0.0 +pvk-016,1.67,1.0,0.7,2.8,0.0,0.3 diff --git a/fixtures/smoke_test/iter2_results.csv b/fixtures/smoke_test/iter2_results.csv new file mode 100644 index 0000000..84260af --- /dev/null +++ b/fixtures/smoke_test/iter2_results.csv @@ -0,0 +1,6 @@ +candidate_id,value,uncertainty,calibration_ref +pvk-009,1.87,0.03,XRD-CAL-2024-07 +pvk-010,1.84,0.02,XRD-CAL-2024-07 +pvk-013,1.80,0.03,XRD-CAL-2024-07 +pvk-015,1.89,0.02,XRD-CAL-2024-07 +pvk-016,1.58,0.05,XRD-CAL-2024-07 diff --git a/fixtures/smoke_test/iter3_candidates.csv b/fixtures/smoke_test/iter3_candidates.csv new file mode 100644 index 0000000..07d3376 --- /dev/null +++ b/fixtures/smoke_test/iter3_candidates.csv @@ -0,0 +1,9 @@ +candidate_id,predicted_value,Cs,Pb,I,Br,Sn +pvk-017,2.05,1.0,1.0,1.6,1.4,0.0 +pvk-018,1.97,1.0,1.0,1.9,1.1,0.0 +pvk-019,1.92,1.0,1.0,2.1,0.9,0.0 +pvk-020,2.10,1.0,1.0,1.4,1.6,0.0 +pvk-021,1.89,0.9,1.0,2.0,1.0,0.1 +pvk-022,2.03,1.0,1.0,1.7,1.3,0.0 +pvk-023,1.85,1.0,0.9,2.2,0.8,0.0 +pvk-024,1.94,1.0,1.0,1.8,1.2,0.0 diff --git a/fixtures/smoke_test/iter3_results.csv b/fixtures/smoke_test/iter3_results.csv new file mode 100644 index 0000000..f6d1520 --- /dev/null +++ b/fixtures/smoke_test/iter3_results.csv @@ -0,0 +1,6 @@ +candidate_id,value,uncertainty,calibration_ref +pvk-017,1.90,0.02,XRD-CAL-2024-08 +pvk-018,1.93,0.03,XRD-CAL-2024-08 +pvk-019,1.88,0.02,XRD-CAL-2024-08 +pvk-022,1.95,0.02,XRD-CAL-2024-08 +pvk-024,1.91,0.03,XRD-CAL-2024-08 diff --git a/fixtures/smoke_test/iter4_candidates.csv b/fixtures/smoke_test/iter4_candidates.csv new file mode 100644 index 0000000..c63fb41 --- /dev/null +++ b/fixtures/smoke_test/iter4_candidates.csv @@ -0,0 +1,9 @@ +candidate_id,predicted_value,Cs,Pb,I,Br,Sn +pvk-025,2.12,1.0,1.0,1.3,1.7,0.0 +pvk-026,2.08,1.0,1.0,1.5,1.5,0.0 +pvk-027,1.99,1.0,1.0,1.7,1.3,0.0 +pvk-028,2.15,1.0,1.0,1.2,1.8,0.0 +pvk-029,1.96,1.0,1.0,1.9,1.1,0.0 +pvk-030,2.06,1.0,1.0,1.6,1.4,0.0 +pvk-031,2.18,1.0,1.0,1.1,1.9,0.0 +pvk-032,1.93,0.9,1.0,2.0,1.0,0.1 diff --git a/fixtures/smoke_test/iter4_results.csv b/fixtures/smoke_test/iter4_results.csv new file mode 100644 index 0000000..f850c07 --- /dev/null +++ b/fixtures/smoke_test/iter4_results.csv @@ -0,0 +1,6 @@ +candidate_id,value,uncertainty,calibration_ref +pvk-025,1.94,0.02,XRD-CAL-2024-08 +pvk-026,1.96,0.02,XRD-CAL-2024-08 +pvk-027,1.92,0.03,XRD-CAL-2024-08 +pvk-029,1.90,0.02,XRD-CAL-2024-08 +pvk-030,1.97,0.02,XRD-CAL-2024-08 diff --git a/fixtures/smoke_test/iter5_candidates.csv b/fixtures/smoke_test/iter5_candidates.csv new file mode 100644 index 0000000..2688e6f --- /dev/null +++ b/fixtures/smoke_test/iter5_candidates.csv @@ -0,0 +1,9 @@ +candidate_id,predicted_value,Cs,Pb,I,Br,Sn +pvk-033,2.20,1.0,1.0,1.0,2.0,0.0 +pvk-034,2.14,1.0,1.0,1.3,1.7,0.0 +pvk-035,2.09,1.0,1.0,1.5,1.5,0.0 +pvk-036,2.22,1.0,1.0,0.9,2.1,0.0 +pvk-037,2.05,1.0,1.0,1.6,1.4,0.0 +pvk-038,2.17,1.0,1.0,1.1,1.9,0.0 +pvk-039,2.11,1.0,1.0,1.4,1.6,0.0 +pvk-040,2.00,1.0,1.0,1.7,1.3,0.0 diff --git a/fixtures/smoke_test/iter5_results.csv b/fixtures/smoke_test/iter5_results.csv new file mode 100644 index 0000000..6d9929b --- /dev/null +++ b/fixtures/smoke_test/iter5_results.csv @@ -0,0 +1,6 @@ +candidate_id,value,uncertainty,calibration_ref +pvk-033,1.92,0.03,XRD-CAL-2024-09 +pvk-034,1.98,0.02,XRD-CAL-2024-09 +pvk-035,1.96,0.02,XRD-CAL-2024-09 +pvk-037,1.93,0.02,XRD-CAL-2024-09 +pvk-039,1.95,0.03,XRD-CAL-2024-09 diff --git a/fixtures/smoke_test/run_smoke_test.sh b/fixtures/smoke_test/run_smoke_test.sh new file mode 100644 index 0000000..0d04a96 --- /dev/null +++ b/fixtures/smoke_test/run_smoke_test.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# Smoke test: 5-iteration perovskite bandgap campaign via CLI. +# +# Campaign narrative (designed to trigger analytics signals): +# - 40 candidates across 5 iterations, 8 per batch +# - ~30% failure rate (3 failures per iteration: synth, stability, process) +# - Surrogate predictions climb from 1.55-1.91 to 2.00-2.22 (model improves) +# - Validated results plateau around 1.93-1.98 (proxy gaming signal) +# - Diversity collapses: Br fraction rises monotonically as model exploits +# - Budget: $2,500/experiment, $100k total over 40 experiments +# +# Calibrated against published SDL campaigns: +# - A-Lab (Nature 2023): 353 attempts, 30% target failure rate +# - Perovskite BO (Nature 2026): 33 experiments across 5 rounds +# - Catalyst BO (ACS 2025): 10 experiments, 7 BO iterations +# +# Usage: bash fixtures/smoke_test/run_smoke_test.sh +set -euo pipefail + +DB="smoke_test.db" +DIR="fixtures/smoke_test" + +rm -f "$DB" + +echo "=== 1. Initialize campaign ===" +matledger init perovskite-bandgap \ + --goal "Find perovskite compositions with bandgap > 1.8 eV and thermal stability > 300C" \ + --budget-cap 100000 \ + --db "$DB" \ + --actor "dr-kimani" + +# ─── ITERATION 1 ───────────────────────────────────────────────────────────── +echo "" +echo "=== 2. Iteration 1 (initial exploration, GP seed) ===" + +matledger ingest "$DIR/iter1_candidates.csv" \ + --data-type candidates --iteration 1 \ + --model-version "gp-v1.0" --db "$DB" --actor "surrogate-gp" + +for cid in pvk-001 pvk-002 pvk-003 pvk-004 pvk-005 pvk-006 pvk-007 pvk-008; do + matledger start-experiment "$cid" --cost 2500 --iteration 1 --db "$DB" --actor "lims" +done + +# 3 failures (~37% this round — high for first iteration, realistic) +matledger record-failure pvk-002 \ + --failure-class synthesizability --iteration 1 \ + --detail "Failed to form stable perovskite phase; Sn substitution caused phase separation" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-004 \ + --failure-class synthesizability --iteration 1 \ + --detail "Cs-deficient composition did not crystallize at target temperature" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-007 \ + --failure-class stability --iteration 1 \ + --detail "Decomposed within 4h under 85C/85RH; mixed-B-site instability" \ + --db "$DB" --actor "lab-tech-ochieng" + +matledger ingest "$DIR/iter1_results.csv" \ + --data-type results --iteration 1 --db "$DB" --actor "xrd-autoloader" + +matledger select pvk-003 \ + --rationale "Highest validated bandgap (1.85 eV) with low uncertainty" \ + --model-version "gp-v1.0" --iteration 1 --db "$DB" --actor "dr-kimani" +matledger select pvk-006 \ + --rationale "Second-best result (1.83 eV), high-Br composition worth exploring" \ + --model-version "gp-v1.0" --iteration 1 --db "$DB" --actor "dr-kimani" +matledger reject pvk-005 \ + --rationale "Below 1.6 eV threshold; Sn-heavy compositions not promising" \ + --model-version "gp-v1.0" --iteration 1 --db "$DB" --actor "dr-kimani" + +matledger model-update --model-version "gp-v1.1" --iteration 1 \ + --summary "Retrained on 5 results from iter 1; learned Br-enrichment trend" \ + --db "$DB" --actor "surrogate-gp" +matledger close-iteration --iteration 1 \ + --summary "8 proposed, 5 tested, 3 failures (2 synth, 1 stability). Best: pvk-003 at 1.85 eV" \ + --db "$DB" --actor "dr-kimani" + +# ─── ITERATION 2 ───────────────────────────────────────────────────────────── +echo "" +echo "=== 3. Iteration 2 (model starts exploiting Br-enrichment) ===" + +matledger ingest "$DIR/iter2_candidates.csv" \ + --data-type candidates --iteration 2 \ + --model-version "gp-v1.1" --db "$DB" --actor "surrogate-gp" + +for cid in pvk-009 pvk-010 pvk-011 pvk-012 pvk-013 pvk-014 pvk-015 pvk-016; do + matledger start-experiment "$cid" --cost 2500 --iteration 2 --db "$DB" --actor "lims" +done + +matledger record-failure pvk-011 \ + --failure-class stability --iteration 2 \ + --detail "Decomposed within 24h under 85C/85RH accelerated aging" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-012 \ + --failure-class synthesizability --iteration 2 \ + --detail "Extreme Br:I ratio (1.5:1.5) produced non-perovskite phase" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-014 \ + --failure-class measurement_error --iteration 2 \ + --detail "XRD sample holder misaligned; measurement discarded" \ + --db "$DB" --actor "lab-tech-ochieng" + +matledger ingest "$DIR/iter2_results.csv" \ + --data-type results --iteration 2 --db "$DB" --actor "xrd-autoloader" + +matledger select pvk-015 \ + --rationale "Best validated result (1.89 eV), stable Br:I=1.1:1.9 composition" \ + --model-version "gp-v1.1" --iteration 2 --db "$DB" --actor "dr-kimani" +matledger reject pvk-016 \ + --rationale "Low bandgap (1.58 eV); Sn substitution consistently underperforms" \ + --model-version "gp-v1.1" --iteration 2 --db "$DB" --actor "dr-kimani" + +matledger model-update --model-version "gp-v1.2" --iteration 2 \ + --summary "Retrained on 10 cumulative results; Br coefficient strengthened" \ + --db "$DB" --actor "surrogate-gp" +matledger close-iteration --iteration 2 \ + --summary "8 proposed, 5 tested, 3 failures (1 synth, 1 stability, 1 measurement). Best: pvk-015 at 1.89 eV" \ + --db "$DB" --actor "dr-kimani" + +# ─── ITERATION 3 ───────────────────────────────────────────────────────────── +echo "" +echo "=== 4. Iteration 3 (model pushes Br higher, predictions inflating) ===" + +matledger ingest "$DIR/iter3_candidates.csv" \ + --data-type candidates --iteration 3 \ + --model-version "gp-v1.2" --db "$DB" --actor "surrogate-gp" + +for cid in pvk-017 pvk-018 pvk-019 pvk-020 pvk-021 pvk-022 pvk-023 pvk-024; do + matledger start-experiment "$cid" --cost 2500 --iteration 3 --db "$DB" --actor "lims" +done + +matledger record-failure pvk-020 \ + --failure-class synthesizability --iteration 3 \ + --detail "Br:I=1.6:1.4 exceeded solubility limit; amorphous film" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-021 \ + --failure-class stability --iteration 3 \ + --detail "Sn-doped composition oxidized during transfer to characterization" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-023 \ + --failure-class process_error --iteration 3 \ + --detail "Furnace temperature controller malfunction during annealing" \ + --db "$DB" --actor "lab-tech-ochieng" + +matledger ingest "$DIR/iter3_results.csv" \ + --data-type results --iteration 3 --db "$DB" --actor "xrd-autoloader" + +matledger select pvk-022 \ + --rationale "Campaign best so far (1.95 eV), Br:I=1.3:1.7 is a sweet spot" \ + --model-version "gp-v1.2" --iteration 3 --db "$DB" --actor "dr-kimani" +matledger reject pvk-019 \ + --rationale "Validated at 1.88, below campaign best; composition already explored" \ + --model-version "gp-v1.2" --iteration 3 --db "$DB" --actor "dr-kimani" + +matledger model-update --model-version "gp-v1.3" --iteration 3 \ + --summary "Retrained on 15 cumulative results; added calibration correction term" \ + --db "$DB" --actor "surrogate-gp" +matledger close-iteration --iteration 3 \ + --summary "8 proposed, 5 tested, 3 failures (1 synth, 1 stability, 1 process). Best: pvk-022 at 1.95 eV" \ + --db "$DB" --actor "dr-kimani" + +# ─── ITERATION 4 ───────────────────────────────────────────────────────────── +echo "" +echo "=== 5. Iteration 4 (predictions diverging from reality — proxy gaming onset) ===" + +matledger ingest "$DIR/iter4_candidates.csv" \ + --data-type candidates --iteration 4 \ + --model-version "gp-v1.3" --db "$DB" --actor "surrogate-gp" + +for cid in pvk-025 pvk-026 pvk-027 pvk-028 pvk-029 pvk-030 pvk-031 pvk-032; do + matledger start-experiment "$cid" --cost 2500 --iteration 4 --db "$DB" --actor "lims" +done + +matledger record-failure pvk-028 \ + --failure-class synthesizability --iteration 4 \ + --detail "Br:I=1.8:1.2 ratio too extreme; mixed halide phase segregation" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-031 \ + --failure-class synthesizability --iteration 4 \ + --detail "Br:I=1.9:1.1 did not form alpha-phase perovskite" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-032 \ + --failure-class stability --iteration 4 \ + --detail "Sn-doped film degraded during 1h ambient exposure" \ + --db "$DB" --actor "lab-tech-ochieng" + +matledger ingest "$DIR/iter4_results.csv" \ + --data-type results --iteration 4 --db "$DB" --actor "xrd-autoloader" + +matledger select pvk-030 \ + --rationale "Best this round (1.97 eV), marginal improvement over iter 3 best" \ + --model-version "gp-v1.3" --iteration 4 --db "$DB" --actor "dr-kimani" +matledger reject pvk-029 \ + --rationale "Validated at 1.90, below campaign best; diminishing returns" \ + --model-version "gp-v1.3" --iteration 4 --db "$DB" --actor "dr-kimani" + +matledger model-update --model-version "gp-v1.4" --iteration 4 \ + --summary "Retrained on 20 cumulative results; prediction variance increasing" \ + --db "$DB" --actor "surrogate-gp" +matledger close-iteration --iteration 4 \ + --summary "8 proposed, 5 tested, 3 failures (2 synth, 1 stability). Predictions 2.06-2.18 but best validated only 1.97 — gap widening" \ + --db "$DB" --actor "dr-kimani" + +# ─── ITERATION 5 ───────────────────────────────────────────────────────────── +echo "" +echo "=== 6. Iteration 5 (proxy gaming confirmed — predictions 2.0-2.2, reality 1.92-1.98) ===" + +matledger ingest "$DIR/iter5_candidates.csv" \ + --data-type candidates --iteration 5 \ + --model-version "gp-v1.4" --db "$DB" --actor "surrogate-gp" + +for cid in pvk-033 pvk-034 pvk-035 pvk-036 pvk-037 pvk-038 pvk-039 pvk-040; do + matledger start-experiment "$cid" --cost 2500 --iteration 5 --db "$DB" --actor "lims" +done + +matledger record-failure pvk-036 \ + --failure-class synthesizability --iteration 5 \ + --detail "Br:I=2.1:0.9 extreme; halide segregation under illumination" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-038 \ + --failure-class synthesizability --iteration 5 \ + --detail "High-Br composition formed non-photoactive delta-phase" \ + --db "$DB" --actor "lab-tech-ochieng" +matledger record-failure pvk-040 \ + --failure-class process_error --iteration 5 \ + --detail "Glovebox O2 sensor triggered shutdown mid-deposition" \ + --db "$DB" --actor "lab-tech-ochieng" + +matledger ingest "$DIR/iter5_results.csv" \ + --data-type results --iteration 5 --db "$DB" --actor "xrd-autoloader" + +matledger select pvk-034 \ + --rationale "Best validated this round (1.98 eV), but gain over iter 4 is only 0.01 eV — campaign flatlining" \ + --model-version "gp-v1.4" --iteration 5 --db "$DB" --actor "dr-kimani" +matledger reject pvk-033 \ + --rationale "Predicted 2.20 but measured 1.92 — clear proxy gaming; model overestimates high-Br compositions" \ + --model-version "gp-v1.4" --iteration 5 --db "$DB" --actor "dr-kimani" + +matledger model-update --model-version "gp-v1.5" --iteration 5 \ + --summary "Retrained on 25 cumulative results; significant prediction-reality gap identified" \ + --db "$DB" --actor "surrogate-gp" +matledger close-iteration --iteration 5 \ + --summary "8 proposed, 5 tested, 3 failures (2 synth, 1 process). Predictions 2.0-2.2 but best validated 1.98 — proxy gaming flagged, diversity collapsed to high-Br region" \ + --db "$DB" --actor "dr-kimani" + +# ─── VERIFY & REPORT ───────────────────────────────────────────────────────── +echo "" +echo "=== 7. Verify & Report ===" + +echo "--- Audit hash chain ---" +matledger audit --db "$DB" + +echo "" +echo "--- Markdown report ---" +matledger report --db "$DB" --format markdown --output smoke_report.md +echo "Written: smoke_report.md" + +echo "" +echo "--- HTML report ---" +matledger report --db "$DB" --format html --output smoke_report.html +echo "Written: smoke_report.html" + +echo "" +echo "--- JSON report ---" +matledger report --db "$DB" --format json --output smoke_report.json +echo "Written: smoke_report.json" + +echo "" +echo "--- Counterfactual replay (greedy) ---" +matledger counterfactual --policy greedy --db "$DB" + +echo "" +echo "--- Counterfactual replay (random, seed=42) ---" +matledger counterfactual --policy random --seed 42 --db "$DB" + +echo "" +echo "--- Export evidence bundle ---" +matledger export --db "$DB" --output smoke_bundle.json +echo "Written: smoke_bundle.json" + +echo "" +echo "=== SMOKE TEST COMPLETE ===" +echo "Campaign: 40 candidates, 5 iterations, 25 results, 15 failures" +echo "Artifacts: $DB, smoke_report.{md,html,json}, smoke_bundle.json" diff --git a/fixtures/smoke_test/sample_report.html b/fixtures/smoke_test/sample_report.html new file mode 100644 index 0000000..13aca10 --- /dev/null +++ b/fixtures/smoke_test/sample_report.html @@ -0,0 +1,302 @@ + + + + + +Campaign Report: smoke_test.db + + + +
+

Campaign Report: smoke_test.db

+
+
Total Events
142
+
Candidates
40
+
Failures
15
+
Budget Consumed
$100,000.00 / $100,000.00
+
Budget Remaining
$0.00
+
+

Campaign Health

+
+
Proxy-Gaming
OK
Window=0, no divergence detected
+
Diversity
OK
Window=1, spread stable
+
Discovery Rate
ACTIVE
25 results, cost/result: $4,000.00
+
+

Trajectory Charts

+
+Proxy-Gaming: Predicted vs Validated +Diversity: Candidate Spread +Budget Burn +Discovery Rate +Chart 5 +Chart 6 +Chart 7 +Chart 8 +Chart 9 +Chart 10 +Chart 11 +
+

Candidates

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CandidateStatusEvents
pvk-001tested3
pvk-002failed3
pvk-003selected4
pvk-004failed3
pvk-005rejected4
pvk-006selected4
pvk-007failed3
pvk-008tested3
pvk-009tested3
pvk-010tested3
pvk-011failed3
pvk-012failed3
pvk-013tested3
pvk-014failed3
pvk-015selected4
pvk-016rejected4
pvk-017tested3
pvk-018tested3
pvk-019rejected4
pvk-020failed3
pvk-021failed3
pvk-022selected4
pvk-023failed3
pvk-024tested3
pvk-025tested3
pvk-026tested3
pvk-027tested3
pvk-028failed3
pvk-029rejected4
pvk-030selected4
pvk-031failed3
pvk-032failed3
pvk-033rejected4
pvk-034selected4
pvk-035tested3
pvk-036failed3
pvk-037tested3
pvk-038failed3
pvk-039tested3
pvk-040failed3
+

Failure Ledger

+ + + +
ClassCount
measurement_error1
process_error2
stability4
synthesizability8
+

Failures by Iteration

+ + + +
IterationFailures
13
23
33
43
53
+

Budget by Iteration

+ + + + + + + + + +
IterationCostCumulative
1$20,000.00$20,000.00
2$20,000.00$40,000.00
3$20,000.00$60,000.00
4$20,000.00$80,000.00
5$20,000.00$100,000.00
+

Iteration Summary

+ + + + + + + + + + +
IterationProposedTestedFailuresSelectedCostSuccess Rate
00000$0.00
18832$20,000.0062%
28731$20,000.0071%
38731$20,000.0071%
48831$20,000.0062%
58731$20,000.0071%
+ +
+ + \ No newline at end of file