[NO MERGE]: torch-2.13+cu130 and flash-attn-3 on santis - #2664
Draft
florianscheidl wants to merge 78 commits into
Draft
[NO MERGE]: torch-2.13+cu130 and flash-attn-3 on santis#2664florianscheidl wants to merge 78 commits into
florianscheidl wants to merge 78 commits into
Conversation
This reverts commit 973f3ef.
Co-authored-by: Tharmeekan <[email protected]>
…cmwf#2414) * Handle scalar datetimes in time encoders * Keep single target times one-dimensional --------- Co-authored-by: Christian Lessig <[email protected]>
…2412) * ensure samples_per_mini_epoch >= world_size*batch_size * updated warning log
* Fix logging when NaNs occur * fix nan variables being included in terminal logged loss, replace nan by 0 * Removing incorrect reset --------- Co-authored-by: Sebastian Hickman <[email protected]>
…fied projection logic (ecmwf#2401) * added arctic region for plotting in stereographic projection and modified projection logic * new projection logic in plotter * restored attention and positional_encoding * added icon, arome, uwc-west regions * removing accidental file --------- Co-authored-by: Matteo Broccoli <[email protected]> Co-authored-by: iluise <[email protected]>
* Align usage of stream configs in model with dict semantics. * parse stream configs into dictconfig instead of list * Align data sampling to dict semantics, wrap self.streams into self.streams_datasets * change list to dictconfig for streams * fixes * ruffed * fix backward compatibility. * address comments * linter * Adjust incoming fresh code. * fix unit tests, rephrase comment
* multiple ranks in evaluation * shorter version * shorter version * Minor correction to ranks PR (ecmwf#2420) * Fix minor bug * Minor correction * Linting --------- Co-authored-by: Savvas Melidonis <[email protected]>
* remove dead code from train_logger * fix plot_loss_per_run * fix filter logic * address comments --------- Co-authored-by: Christian Lessig <[email protected]>
* Changes to allow for score animations. Addition possibility for log-scale colorbar * Linting * Edit config files * Change to score map function name, move score function to new utils file, generalize single_animation function * Minor edits * Linting * Minor corrections to clarify the sa variable --------- Co-authored-by: iluise <[email protected]>
Co-authored-by: iluise <[email protected]>
* Delete unnecessary code * Linting
…text in legend entries (ecmwf#2433) * Made legend font size and columns configurable and removed redundant string from plot legend entries and moved this to title * Reverting unnecessary changes * Put losses per run behind command line arg (typically not used currently)
…data (ecmwf#2422) * Custom data reader for operational analysis that accounts for actual availability * Reduce mem footprint during training; add DataReaderAnemoiOperan * Config files * Fixed corner case: sample 0 that has no precedessor * Added additional check * Move OperAn data reader to extras readers * Cleaning up * Remove incorrect general setting for base permuation * Remove incorrect time_window_step settings * Linting * Improved encapsulation
* Fixes to forcing fields in io * Fixed linting
…on (ecmwf#2436) * time series scores * add offset for lead time without source * add fstep to ratio plots * fixed range for ratio plots * if the source is not available as a group in the zarr, instead use source interval attribute from target or prediction * add comments * lint * fix label in ratio plot * fix init time --------- Co-authored-by: Sebastian Hickman <[email protected]>
* Move data reader fesom to extra package * ruff --------- Co-authored-by: Christian Lessig <[email protected]>
* remove dead code from train_logger * fix plot_loss_per_run * fix filter logic * address comments * Smaller decoder for SYNOP * Custom data reader for operational analysis that accounts for actual availability * Reduce mem footprint during training; add DataReaderAnemoiOperan * Config files * Reduce spoofing to two tokens * Fixed corner case where there is no t_idx-1: very first sample * Fixes needed to run inference and write output with spoofed streams * Save soure_raw in val mode * Add additional sanity check * Reformatting * Updated config * Updated configs * Configurable plot_train * remove clim align when not necessary * Iluise/develop/init time plot (ecmwf#2457) * Sgrasse/develop/342 streams as dict (ecmwf#2242) * Align usage of stream configs in model with dict semantics. * parse stream configs into dictconfig instead of list * Align data sampling to dict semantics, wrap self.streams into self.streams_datasets * change list to dictconfig for streams * fixes * ruffed * fix backward compatibility. * address comments * linter * Adjust incoming fresh code. * fix unit tests, rephrase comment * combine inference from multiple ranks in evaluation (ecmwf#2400) * multiple ranks in evaluation * shorter version * shorter version * Minor correction to ranks PR (ecmwf#2420) * Fix minor bug * Minor correction * Linting --------- Co-authored-by: Savvas Melidonis <[email protected]> * Sgrasse/develop/2353 plot train (ecmwf#2354) * remove dead code from train_logger * fix plot_loss_per_run * fix filter logic * address comments --------- Co-authored-by: Christian Lessig <[email protected]> * Score animations with fixed colorbar (ecmwf#2402) * Changes to allow for score animations. Addition possibility for log-scale colorbar * Linting * Edit config files * Change to score map function name, move score function to new utils file, generalize single_animation function * Minor edits * Linting * Minor corrections to clarify the sa variable --------- Co-authored-by: iluise <[email protected]> * apply scale_z_channels (ecmwf#2417) Co-authored-by: iluise <[email protected]> * Fix bug in the evaluation wegen_reader.py (ecmwf#2431) * Delete unnecessary code * Linting * time series scores * add offset for lead time without source * add fstep to ratio plots * Made legend font size and columns configurable and removed redundant text in legend entries (ecmwf#2433) * Made legend font size and columns configurable and removed redundant string from plot legend entries and moved this to title * Reverting unnecessary changes * Put losses per run behind command line arg (typically not used currently) * Add data reader for Oper-an that accounts for actual availability of data (ecmwf#2422) * Custom data reader for operational analysis that accounts for actual availability * Reduce mem footprint during training; add DataReaderAnemoiOperan * Config files * Fixed corner case: sample 0 that has no precedessor * Added additional check * Move OperAn data reader to extras readers * Cleaning up * Remove incorrect general setting for base permuation * Remove incorrect time_window_step settings * Linting * Improved encapsulation * Fixes to forcing fields in io (ecmwf#2442) * Fixes to forcing fields in io * Fixed linting * Fix spoofing so that it only uses 2 tokens to avoid large memory footprint (ecmwf#2449) * fixed range for ratio plots * if the source is not available as a group in the zarr, instead use source interval attribute from target or prediction * add comments * lint * fix label in ratio plot * fix init time --------- Co-authored-by: Simon Grasse <[email protected]> Co-authored-by: iluise <[email protected]> Co-authored-by: Savvas Melidonis <[email protected]> Co-authored-by: Julius Polz <[email protected]> Co-authored-by: iluise <[email protected]> Co-authored-by: Sebastian Hickman <[email protected]> * lint * Revert "Merge branch 'clessig/develop/smaller_decoder_synop' into iluise.develop/fix-climatology" This reverts commit 9176522, reversing changes made to b7b6c9d. * julius comments --------- Co-authored-by: Simon Grasse <[email protected]> Co-authored-by: Simon Grasse <[email protected]> Co-authored-by: lessig2 <[email protected]> Co-authored-by: Savvas Melidonis <[email protected]> Co-authored-by: Julius Polz <[email protected]> Co-authored-by: Sebastian Hickman <[email protected]>
* first working implementation of spectra * fix zonal (ukmet) psd * remove old psd implementation * remove dead code * shorter functions * implement inverse transform and tests * lint * add var name to psd plot title * PR comments * add comments about limitations * lint * automatically detect grids * fix eval config * implement comments * lint
* initial casestudy implementation * initial linting * added some docstrings and comments * encapsulated dependencies, added instructions * changed path to relative, added README --------- Co-authored-by: buschow1 <[email protected]>
Change default value of 'rank' from 0 to 'all'.
* Fix missing type of rnaks when specified as list * Fix import alias for omegaconf in wegen_reader.py --------- Co-authored-by: iluise <[email protected]>
* Implement regular subsampling * ruff * Fix handling of holes in dataset * Update stream config * Remove 0 as filler value --------- Co-authored-by: Savvas Melidonis <[email protected]>
* Replace cf.rank==0 with utils.distributed.is_root * Restore streams_directory replacement semantics --------- Co-authored-by: wang85 <[email protected]> Co-authored-by: wang85 <[email protected]>
Co-authored-by: kasravi1 <[email protected]> Co-authored-by: Christian Lessig <[email protected]>
* score bug fixes * Align score-map ensemble handling with adjusted-scores branch Use the robust ens-detection variant in _plot_score_maps_per_stream so the block is identical (apart from the branch-specific tag string) to mh/full-pipeline-diffusion-adjusted-scores, minimising future merge conflicts: - restore ens labels via assign_coords(ens=preds.ens.values) (positional) instead of plot_metrics["ens"] = preds.ens (index-aligned) - gate has-ens detection on the ens *dimension* (all_ens) rather than the coordinate - compute per-metric ens iteration via metric_has_ens / ens_values Behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 <[email protected]> * renames --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
* score bug fixes * Align score-map ensemble handling with adjusted-scores branch Use the robust ens-detection variant in _plot_score_maps_per_stream so the block is identical (apart from the branch-specific tag string) to mh/full-pipeline-diffusion-adjusted-scores, minimising future merge conflicts: - restore ens labels via assign_coords(ens=preds.ens.values) (positional) instead of plot_metrics["ens"] = preds.ens (index-aligned) - gate has-ens detection on the ens *dimension* (all_ens) rather than the coordinate - compute per-metric ens iteration via metric_has_ens / ens_values Behaviour is unchanged. Co-Authored-By: Claude Opus 4.8 <[email protected]> * implemented spread adj and ssr adj * add latitude weighting * call calc_latitude_weights on p in ssr functions * Refactor latitude weight calculation to use static method and improve coordinate handling * move calc_latitude_weights to score utils * wire lat weights to config * add optional lat weighting to more scores * refactor: introduce _weighted_mean helper * unify ssr and ssr_adj via param switch * rename use_latitude_weights to latitude_weighting * move lat weights to verified data * add warning if no latitude was found * make adjusted ssr the default * lint * renames --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: jpolz <[email protected]> Co-authored-by: iluise <[email protected]>
* Require geoinfo_channels for obs streams * Switch from raise to assert * Linting --------- Co-authored-by: Christian Lessig <[email protected]>
…on when config settings change (ecmwf#2598) * do not recompute scores when unnecessary * better get eval settings * lint * add forecast_step to eval_settings dict * restore csv * update config reference section "Score caching (JSON files)" (ecmwf#2645) * Update cache invalidation details in documentation Clarified cache invalidation process and updated parameters for recomputation. --------- Co-authored-by: Julius Polz <[email protected]>
Co-authored-by: iluise <[email protected]>
* initial fix with no real coordinate * hacked in the coordinate * coordinate ordering * sugggestions * linting * more changes * testing complete --------- Co-authored-by: buschow1 <[email protected]> Co-authored-by: Sorcha Owens <[email protected]> Co-authored-by: Sorcha Owens <[email protected]>
* Update filename in synop.yml for new data version * Update filenames and tokenize_spacetime settings Need tokenize_spacetime False as now obs are near continuous * Change tokenize_spacetime from True to False * Update synop.yml
* Synop int. test, token_size 512 tok_spctm = False Change tokenize_spacetime to False for integration test for multi streams. v2 -> v5 SurfaceCombined changes the sampling/averaging of observations. v2 had observations averaged to hourly, while v5 keeps them continuous. This makes tokenize_spacetime True unworkable as we now have many, many unique timestamps in a 6 hour window. Also increase token_size to avoid exceeding max num tokens. * Change tokenize_spacetime from True to False
* change colorbars * lint --------- Co-authored-by: Christian Lessig <[email protected]>
….com:florianscheidl/WeatherGenerator into exp/try-pytorch-213-flash-attn-diff-versions
florianscheidl
changed the title
Contributor
|
Perfect, will test FSDP2 without spoofing now if that works, we should then continue with some science experiments to see how much we should fast track merging this upgrade! |
github-actions
Bot
added
data
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Not meant for merge - demonstrates that we can run Pytorch 2.13 with CUDA 13.0 and Flash Attention 3.0 on Santis.
@sophie-xhonneux
Issue Number
Is this PR a draft? Mark it as draft.
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60