Skip to content

Remote pull: stage temp files in per-user /tmp; GUI shot deletion#70

Open
sguizzo wants to merge 2 commits into
developfrom
bugfix/remote-out-dir-staging
Open

Remote pull: stage temp files in per-user /tmp; GUI shot deletion#70
sguizzo wants to merge 2 commits into
developfrom
bugfix/remote-out-dir-staging

Conversation

@sguizzo

@sguizzo sguizzo commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Two related data-layer/GUI changes for the DIII-D remote fetch path.

1. Stage the remote pull's temp .h5 in per-user /tmp (off the home quota)

The remote (cluster/toksearch) backend ran the pull on omega and wrote its
compact .h5 under ~/magnetics_fetch/out/ before rsync'ing it home. That file
is ~80 MB and accumulated across runs, so it eventually blew the GA home disk
quota
— the pull died with OSError [Errno 122] Disk quota exceeded on the
cluster-side h5py.File create.

The staged file is purely transient (written → copied back → never reused), so:

  • Where toksearch puts its temporary files changed: it now stages the .h5
    in node-local /tmp/magnetics_out_<user>/ instead of the home dir. Only the
    small package sync still lives under remote_dir (~/magnetics_fetch).
  • The staged copy is deleted right after the copy-back, so /tmp doesn't
    accumulate either.
  • The dir is per-user (_<username> suffix, a top-level entry in sticky-bit
    /tmp) so concurrent pulls from different users on the shared login node don't
    collide. The username comes from the known GA username, falling back to
    id -un on the cluster (works even when $USER/$LOGNAME are unset in a
    non-login ssh command shell). Base dir overridable via run_remote(out_stage=...).

Verified live: a real pull staged to /tmp/magnetics_out_<user>/shot_<n>.h5,
copied home, and cleaned up — quota error gone.

2. GUI: delete a shot (× per row) + clear-all, removing its data

Adds data-management controls to the left-rail shot list:

  • an × on each fetched shot row deletes that shot, and a Clear all in the
    section header deletes every fetched shot;
  • both remove the underlying HDF5 file(s) on the backend, not just the GUI row.

Backend: h5source.delete_shot() removes every file for a shot (matched on
the stored shot attr, so multi-file shots all go), delete_all_shots() clears
them all; both refresh the file index. New DELETE /api/machines/{shot} and
DELETE /api/machines routes also drop the shot's cached STFT/QS state; the
per-shot route 404s when nothing matches (e.g. a mock machine).

Frontend: store.removeMachine/clearMachines call the routes, re-list, and
fix the selection. The × and Clear-all only render for real shots against a live
backend (mock demo machines have nothing to delete), each behind a confirm()
since the data loss is irreversible.

Testing

  • New tests/test_delete.py (5 tests) covers delete_shot/delete_all_shots and
    the DELETE routes against an isolated temp data dir.
  • Full suite green: 309 passed, 5 skipped; frontend 29 passed; ty check src/magnetics, ruff check, ruff format --check all clean.
  • Both changes verified manually in the running GUI.

🤖 Generated with Claude Code

sguizzo and others added 2 commits July 1, 2026 16:42
The remote toksearch pull staged its ~80 MB compact .h5 under
~/magnetics_fetch/out/ before copying it home; across runs this blew
the GA home quota (OSError [Errno 122] Disk quota exceeded on the
cluster-side h5py.File create).

Stage the transient file in node-local /tmp instead (it is written,
rsync'd back, and never reused), keeping only the small package sync
under remote_dir, and delete the staged copy after copy-back so /tmp
doesn't accumulate.

Make the stage dir per-user (/tmp/magnetics_out_<user>) so concurrent
pulls on the shared login node don't collide: the username is put in
the top-level /tmp entry (each user owns their own) rather than a
subdir under a shared parent. The name comes from the known GA
username, falling back to `id -un` on the cluster (works when
$USER/$LOGNAME are unset in a non-login ssh shell). Stage dir base is
overridable via run_remote(out_stage=...).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add data-management controls to the shot list: an × on each fetched
shot deletes it, and a "Clear all" in the section header deletes every
fetched shot. Both remove the underlying HDF5 file(s) on the backend,
not just the GUI row.

Backend: h5source.delete_shot() removes EVERY file for a shot (matched
on the stored `shot` attr, so multi-file shots — different windows/fmax
or a bench artifact — all go), delete_all_shots() clears them all; both
refresh the file index. New DELETE /api/machines/{shot} and DELETE
/api/machines routes also drop the shot's cached STFT/QS state; the
per-shot route 404s when nothing matches (e.g. a mock machine).

Frontend: store.removeMachine/clearMachines call the routes, re-list,
and fix the selection; the × and Clear-all only render for real shots
against a live backend (mock demo machines have nothing to delete), each
behind a confirm() since the data loss is irreversible.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant