Make eval workflow reproducible with minimal dataset#21
Open
brianhenn wants to merge 2 commits into
Open
Conversation
Cache only the reduced, plot-ready inputs each manuscript figure
consumes instead of the full gridded intermediates, so the figures can
be regenerated from a ~140 MB bundle (26 files) rather than the ~42 GB
cache / ~1.8 TB raw dataset. This backs a new code+data Zenodo version
that runs end-to-end from the bundled data.
Notebooks (E1-E5):
- Each RESET_CACHE write cell now computes the reduced products
(bias/RMSB, global-mean annual means + trends, ENSO error/RMSE,
daily-variability error + global-mean tables, perturbation response)
and caches those; RESTORE_CACHE loads them directly. Map figures get
small tas/pr slices; bar/series figures get scalar tables.
- Default to RESTORE_CACHE=True / RESET_CACHE=False and add markdown
sections (How to run / Data loading & processing / Restore minimal
cache / Supplementary diagnostics / Manuscript figures) so the
cache-only vs regenerate-from-raw paths are documented rather than
gated per-cell.
- E2 now emits both trend_series_tas_ens_false and ens_true.
aimip_data_utils.py:
- Move compute_global_mean / compute_decadal_trend / compute_annual_mean
/ linear_fit out of the E2 notebook into the shared module.
Add evaluations/build_repro_archive.py (writes cached/MANIFEST.md,
stages the tree + minimal cache, tars it) and evaluations/
REPRODUCIBILITY.md. Ignore .ipynb_checkpoints/.
Verified: regeneration reproduces the cached-data path (0-pixel diff),
and the modified figure cells render identically before/after.
Track the 26 reduced cache files (~134 MB) under evaluations/notebooks/
cached/ so a tagged GitHub release — and the Zenodo archive built from
it — ships the reproducibility data automatically, with no manual upload
or git-LFS. The files are all <40 MB (under GitHub's per-file limit).
whitelist exactly the 26 reduced products, keeping the ~45 GB of full
gridded intermediates ignored.
- Remove build_repro_archive.py: the standalone-tarball step is
superseded by the auto-release path. Its file->figure manifest now
lives in REPRODUCIBILITY.md.
- REPRODUCIBILITY.md: document that the data ships in-repo and is carried
by the release/Zenodo integration; add the cache-file -> figure table.
Fix ENSO regeneration (enso_index.py): build the Nino3.4 index time
coordinate as datetime64[ns]. Newer numpy converts datetime.datetime to
datetime64[us]; interpolating a [us] index onto the [ns] model/ERA5 time
axis mismatched resolutions and produced all-NaN ENSO coefficients when
regenerating E3 from raw. Verified: interp NaN 99.8% -> 1.6%.
Notebooks (E2-E5): move the config/flags cell to the top for easy
RESET_CACHE/RESTORE_CACHE toggling, refresh regeneration outputs, and fix
a typo in the E3 processing cell. Regeneration from raw was verified to
reproduce the committed cache (0 diff for E1/E2/E4/E5; fp round-off for
E3) and all 29 manuscript figures (0% pixel diff vs the cache path).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the evaluation notebooks reproducible from the repo state by committing a minimal subset of datasets. This is essentially the smallest version of the dataset (reductions applied) that can reproduce the figures; the users still have the option of running the notebooks from the full dataset if they have access to it.