Skip to content

Gate SkillsBench launch on runner readiness#2543

Merged
huangruiteng merged 1 commit into
mainfrom
codex/skillsbench-launch-readiness-preflight-20260726
Jul 25, 2026
Merged

Gate SkillsBench launch on runner readiness#2543
huangruiteng merged 1 commit into
mainfrom
codex/skillsbench-launch-readiness-preflight-20260726

Conversation

@huangruiteng

Copy link
Copy Markdown
Owner

Summary

  • let the existing runner-profile CLI probe task-free SSH from the launcher's current environment
  • require a ready connectivity receipt before any non-dry-run remote preflight or supervisor start
  • preserve dry-run as a no-network plan while reporting the pending readiness gate

Boundary

  • emits only the existing compact public connectivity receipt on failure
  • reads no task material and records no SSH output, destination, or profile values
  • validation launches no benchmark job
  • does not change scoring, task semantics, submissions, uploads, or permission boundaries

Validation

  • pytest -q tests/test_skillsbench_turn_launcher.py tests/test_skillsbench_runner_profile.py tests/test_skillsbench_reverse_tunnel_supervisor.py (36 passed)
  • python examples/skillsbench-runner-profile-smoke.py
  • python -m mypy
  • ruff check
  • bash -n scripts/skillsbench-launch-goal-xhigh.sh
  • loopx check on all changed public surfaces
  • loopx canary premerge --from-git-diff: all direct checks, 6 catalog canaries, 8 risk-profile smokes, and the public boundary check passed; the generic benchmark_sensitive hold is covered by the standing owner authorization for narrow benchmark helper/runtime PR self-merge

@huangruiteng

Copy link
Copy Markdown
Owner Author

No actionable findings.

Open questions/assumptions:

  • Exit code 3 remains the launcher's compact preflight-blocked class; the JSON schema identifies runner connectivity versus later sandbox preflight failure.
  • --current-environment accepts only the SSH destination and option keys needed by the existing task-free probe.

Product/architecture judgment:

  • Motivation: a typed readiness receipt is insufficient if the real launcher can bypass it and enter remote preflight or supervisor startup while task-free SSH is rejected.
  • Solved: the existing runner-profile CLI owns current-environment probing, while the launcher only enforces the receipt and preserves dry-run planning.
  • Operator impact: failed remote acceptance now returns one public-safe blocker before supervisor state or benchmark work exists; unrelated LoopX todos can continue.
  • Main risk: one additional task-free SSH probe on non-dry-run launch. It replaces ambiguous downstream failure with an explicit gate and reuses the launcher's existing destination/options.
  • Design: the shell remains thin, and readiness semantics stay in the existing adapter rather than being duplicated in the launcher or heartbeat.

Validation:

  • 36 focused tests and the runner-profile smoke passed
  • mypy, Ruff, shell syntax, diff hygiene, and public/private boundary checks passed
  • premerge canary passed all direct checks, 6 catalog canaries, 8 risk-profile smokes, and the public boundary check
  • the remaining generic benchmark_sensitive hold is acknowledged under the standing owner authorization for narrow benchmark helper/runtime PR self-merge

Merge decision: self-merge after GitHub Python Tests pass.

@huangruiteng
huangruiteng merged commit 8a8490d into main Jul 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant