Problem
When a remote job (runplz brev/runplz ssh) is mid-flight and needs to be stopped — e.g. wall-time bug, a planned bigger refactor before the run finishes, OOM error that survived the orchestrator's exit, or just an unsupervised retry loop spinning — there's no first-class way to cancel it. The current workflow is:
ssh <instance> (sometimes through a proxy)pgrep -af <job-keyword> to find pidspkill -TERM -f <pattern> (often survives), then pkill -KILL -f <pattern>- Verify GPU is idle (
nvidia-smi) so a relaunch doesn't OOM
This is fragile — you have to remember the right keyword, and when the orchestrator's bash supervisor has already died (heartbeat dead, only worker spawn_main processes left, all parented to init), pkill -P doesn't help and you fall back to pkill -KILL.
Desired feature
A subcommand that, given a runplz run identifier (or the --instance flag plus a run id from ~/runplz-runs/<run_id>/), cleanly stops the job: kills the orchestrator + all spawned workers + any forked DataLoader children, frees device memory, writes a final event=killed_by_user to events.ndjson. Sketch:
runplz kill --instance mhcflurry-release-exact <run-id>
runplz kill --host my-ssh-alias <run-id> # via ssh backend
Or using the existing run-id discovery:
runplz ls --instance mhcflurry-release-exact # list active runs
runplz kill --instance mhcflurry-release-exact --latest
Bonus / nice-to-have
runplz kill --signal TERM first, escalate to KILL after a configurable timeout (e.g. 10 sec) if procs survive.- Print final disposition (procs killed, GPUs freed, heartbeat last-seen, log-tail).
runplz status --instance ... to peek without killing — useful for a "is anything still running?" check before relaunch.- Idempotent: re-running on an already-dead run should exit 0 with a message, not error.
Context
I'm running mhcflurry release training and minibatch sweeps via runplz brev against MassedCompute 8×A100-80G boxes. After several rounds of "stop the sweep, fix a perf bug, re-launch", the manual SIGTERM/SIGKILL/nvidia-smi-check dance is the most error-prone step in the workflow. A first-class kill would also make CI-style cancel-on-superseded retries clean.
Problem
When a remote job (
runplz brev/runplz ssh) is mid-flight and needs to be stopped — e.g. wall-time bug, a planned bigger refactor before the run finishes, OOM error that survived the orchestrator's exit, or just an unsupervised retry loop spinning — there's no first-class way to cancel it. The current workflow is:ssh <instance>(sometimes through a proxy)pgrep -af <job-keyword>to find pidspkill -TERM -f <pattern>(often survives), thenpkill -KILL -f <pattern>nvidia-smi) so a relaunch doesn't OOMThis is fragile — you have to remember the right keyword, and when the orchestrator's bash supervisor has already died (heartbeat dead, only worker spawn_main processes left, all parented to init),
pkill -Pdoesn't help and you fall back topkill -KILL.Desired feature
A subcommand that, given a
runplzrun identifier (or the--instanceflag plus a run id from~/runplz-runs/<run_id>/), cleanly stops the job: kills the orchestrator + all spawned workers + any forked DataLoader children, frees device memory, writes a finalevent=killed_by_usertoevents.ndjson. Sketch:Or using the existing run-id discovery:
Bonus / nice-to-have
runplz kill --signal TERMfirst, escalate toKILLafter a configurable timeout (e.g. 10 sec) if procs survive.runplz status --instance ...to peek without killing — useful for a "is anything still running?" check before relaunch.Context
I'm running mhcflurry release training and minibatch sweeps via
runplz brevagainst MassedCompute 8×A100-80G boxes. After several rounds of "stop the sweep, fix a perf bug, re-launch", the manual SIGTERM/SIGKILL/nvidia-smi-check dance is the most error-prone step in the workflow. A first-class kill would also make CI-style cancel-on-superseded retries clean.