feat(gms): add pluggable snapshot transfer backends

[galletas1712]

Summary

• split GMS snapshot restore byte movement behind a TransferBackend interface while keeping allocation layout, metadata restore, and commit semantics in GMSStorageClient
• keep the default CPU-staged restore path and add selectable aio, nixl-gds, cufile-gds, local-ssd-striped, and local-ssd-pinned backends
• pass GMS transfer/backend tuning env vars from restore/checkpoint workloads into gms-loader / gms-saver, including local SSD roots and shard sizing
• support separate GMS artifact storage from the snapshot control PVC while preserving snapshot-agent-visible sentinel semantics
• reduce restore loader startup overhead by avoiding eager NIXL/PyTorch imports and discovering devices from checkpoint layout before falling back to NVML
• add transfer timing logs and focused tests for the storage-client/backend boundary and backend transfer setup

Validation

• PYTHONPATH=lib python3 -m py_compile lib/gpu_memory_service/common/utils.py lib/gpu_memory_service/snapshot/disk.py lib/gpu_memory_service/snapshot/transfer.py lib/gpu_memory_service/snapshot/storage_client.py lib/gpu_memory_service/cli/storage_runner.py lib/gpu_memory_service/cli/snapshot/loader.py lib/gpu_memory_service/cli/snapshot/saver.py
• PYTHONPATH=/home/schwinns/dynamo-gms-nixl-pr8833/lib /home/schwinns/dynamo-pr8833/lib/gpu_memory_service/.venv/bin/python -m pytest -c /dev/null /home/schwinns/dynamo-gms-nixl-pr8833/lib/gpu_memory_service/tests/test_snapshot_transfer.py -q -o cache_dir=/tmp/pytest-cache-gms-pr8900 (8 passed)
• go test ./internal/checkpoint ./internal/controller from deploy/operator

nscale benchmark notes

AIO smoke

Same-node nscale DRA node cluster-0967a26d-pool-14bee067-prctr-s2877, GMS_TRANSFER_BACKEND=aio, GMS_LOAD_WORKERS=8.

Artifacts: /home/schwinns/blog-post-bench/results/gms-aio-samenode-s2877-20260430T061030Z

model	default preadv GMS load	AIO GMS load	result
Qwen3-0.6B	0.530s	0.700s	coherent, length-capped
Qwen2.5-72B	20.210s	19.590s	coherent, finish_reason=stop

The AIO 72B loader logs confirmed modes=odirect-libaio:41, so the run used native AIO rather than buffered fallback. The result is only a small/noisy improvement over the default CPU-staged path.

Local SSD pinned matrix

Same-node nscale DRA node cluster-0967a26d-pool-14bee067-prctr-l9nsv, GMS_TRANSFER_BACKEND=local-ssd-pinned, GMS_{SAVE,LOAD}_WORKERS=8, 1 GiB shards, /mnt/nvme2..9/gms.

Artifacts: /home/schwinns/blog-post-bench/results/gms-local-ssd-pinned-posix-full-l9nsv-20260503T010828Z/

model	GMS load	Phase B	Phase B BW	result
Qwen3-0.6B	0.740s	0.243s	4.84 GiB/s	coherent, stop
Qwen3-8B	1.270s	0.638s	24.01 GiB/s	coherent, stop
Qwen3-14B	1.590s	0.987s	27.95 GiB/s	coherent, stop
Qwen3-32B	2.450s	1.729s	35.34 GiB/s	coherent, stop
Qwen2.5-72B	4.690s	3.949s	34.34 GiB/s	coherent, stop
GPT-OSS-120B	2.980s	2.254s	32.86 GiB/s	coherent, stop
Llama 3.3 70B FP8	2.680s	1.966s	34.50 GiB/s	coherent, stop

The earlier Qwen2.5-72B 7.52s local-SSD-pinned row reproduced as noise: rerunning the same code path gave 4.93s, the previous image gave 5.14s, and the final posix-buffer image gave 4.69s.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.