Skip to content

Run worker fingerprint capture from mounted script - #256

Draft
alec-flowers wants to merge 1 commit into
NVIDIA:mainfrom
alec-flowers:codex/fingerprint-runtime-script
Draft

Run worker fingerprint capture from mounted script#256
alec-flowers wants to merge 1 commit into
NVIDIA:mainfrom
alec-flowers:codex/fingerprint-runtime-script

Conversation

@alec-flowers

@alec-flowers alec-flowers commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • extract worker-container fingerprint capture from an embedded Python heredoc into runtime_scripts/fingerprint.py
  • invoke the existing /srtctl-runtime mount with a short, shell-quoted command
  • keep framework and CPU parsing definitions shared between host and worker capture paths
  • preserve best-effort probe isolation and the existing fingerprint JSON contract
  • retain srun command logging at its existing DEBUG level while removing the stale heredoc explanation

Why

Worker startup previously generated an entire Python program as a string and passed it through bash, srun, container bash, and Python. The nested quoting made the capture path difficult to review and required tests to extract and parse generated source code.

runtime_scripts/ is already mounted into every worker container at /srtctl-runtime. Running a normal packaged script makes the execution path direct and keeps the generated worker command small without changing fingerprint filenames or consumers.

User impact

There is no configuration or JSON schema change. Workers continue writing fingerprint_<mode>_w<index>.json under /logs, and fingerprint failures remain non-fatal.

Validation

  • ruff check src/srtctl/
  • ruff format --check src/srtctl/
  • modified-file ty check
  • pytest -q tests --ignore=tests/test_integration_status.py: 828 passed, 2 skipped, 6 deselected
  • direct standalone-script execution and malformed-probe regression coverage
  • wheel build verified srtctl/runtime_scripts/fingerprint.py is packaged
  • deep and general review findings addressed before commit

Live canary

Job 21231 completed COMPLETED 0:0 in 55 seconds on Watchtower GB200 node watchtower-blue-cn01, using a checkout synced from this PR commit.

The generated connection command included:

.../src/srtctl/runtime_scripts:/srtctl-runtime

The custom benchmark failed unless /logs/fingerprint_agg_w0.json existed, parsed as JSON, and contained CPU and framework mappings. It completed with:

LIVE_FP_PATH=/logs/fingerprint_agg_w0.json
LIVE_FP_CPU_MODEL=ARM CPU implementer 0x41 part 0xd4f (architecture 8, variant 0x0, revision 0)
LIVE_FP_FRAMEWORKS=dynamo,tensorrt_llm
LIVE_FP_STATUS=ok

Canary output: /mnt/lustre01/users/slurm-shared/aflowers/srt-slurm-fingerprint-runtime-script-smoke/outputs/21231

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.96537% with 74 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@b07c990). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/srtctl/runtime_scripts/fingerprint.py 65.56% 73 Missing ⚠️
src/srtctl/cli/mixins/worker_stage.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #256   +/-   ##
=======================================
  Coverage        ?   66.91%           
=======================================
  Files           ?       71           
  Lines           ?     9189           
  Branches        ?        0           
=======================================
  Hits            ?     6149           
  Misses          ?     3040           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants