Workflow to store models permanently

[clessig]

Describe the task. It can be a feature, documentation, etc.

Currently, model artefacts are stored on SCRATCH (with auto-deletion) and can be pushed to the EWC. However, some models (release and other baselines) need to be available permanently (slurm, models, results, log dirs). We can achieve this by symlinking from a location on a perm file system (same structure in terms of dirs; slurm ideally also directly under shared_work on PERM to make it easier to find) to the default locations on SCRATCH. Symlinks are not affected by the auto-delete policy (has been checked with the CSCS and JSC).

We need tooling (a script) that moves a model from SCRATCH to the perm location on the respective HPC. Ideally, we also want to do this across HPCs (maybe we can use ATOS as bridge and then ALPS-B)?

Hedgedoc URL, if you are keeping notes, plots, logs in hedgedoc.

No response

Area

• datasets, data readers, data preparation and transfer
• model
• science
• infrastructure and engineering
• evaluation, export and visualization
• documentation
• performance

[tjhunter]

The intent of this issue is to protect certain slurm dirs, models, results (inference runs?) from deletion policies.

For models: this script should solve the issue if people care enough to run it before launching their runs (we can automate that):
#2699

For slurm dirs: the bigger problem is that they are becoming work directories. they were meant to be a staging area for isolating all the parts of a run.

For results: do we need something formal right now? The inference data seems very run- or user-specific.

The 2 bigger issues:

• The PERM storage is limited and we will end up with the same storage issues as before to clean it
• We build a federated storage (each HPC has a partial copy of the permanent data). It is easier to have a centralized place for data management and visibility purpose.