Vendor coding and cua templates under environments/ - #512
Conversation
Bring the coding (v6 rebuild) and cua starter templates into the SDK repo as canonical examples of v6 environment authoring. Each is a standalone uv project with its own lockfile, Dockerfile, and tests. environments/* stays ignored except the vendored templates, so local scratch checkouts are unaffected. The templates are excluded from the built wheel/sdist; their nested ruff config keeps root lint isolated, and root pyright/pytest scoping already excludes them. Follow-ups: CI matrix running template tests against the local SDK, release-gate integrity runs, vendored hud init presets, and mirroring back to the standalone template repos.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ea60e77. Configure here.
| else \ | ||
| git clone ${REPO_URL} /app; \ | ||
| fi \ | ||
| && chown -R 1000:1000 /app |
There was a problem hiding this comment.
Token leaked into image layers
High Severity
The BuildKit secret keeps the token file out of the layer, but embedding it in the clone URL writes it into /app/.git/config. That config is part of the image filesystem, so anyone with the image can recover CODING_GITHUB_TOKEN from the layer even after runtime vaulting moves .git.
Reviewed by Cursor Bugbot for commit ea60e77. Configure here.
| await git(repo, "add", "-A") | ||
| await git(repo, "commit", "-q", "--allow-empty", "-m", "hud: final snapshot") | ||
| _, final, _ = await git(repo, "rev-parse", "HEAD") | ||
| _, diff, _ = await git(repo, "diff", setup_commit, final.strip(), timeout=1200.0) |
There was a problem hiding this comment.
Binary changes break generic grading
Medium Severity
capture_agent_diff runs git diff without --binary, so binary edits become non-applyable "Binary files differ" stubs. apply_diff then fails the whole patch, scoring 0 even when the text portion of the fix is correct. The SWE-bench path strips those hunks; the generic path does not.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ea60e77. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea60e77aee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else: | ||
| _LOCAL_ROOT = Path(tempfile.gettempdir()) / "hud-coding" | ||
| REPO_DIR = _LOCAL_ROOT / f"repo-{os.getpid()}" | ||
| VAULT_DIR = _LOCAL_ROOT / f"vault-{os.getpid()}" |
There was a problem hiding this comment.
When --runtime local runs as an ordinary user on macOS or Linux without bwrap, Workspace.shell_uid neither changes the UID nor isolates the filesystem, so this predictable same-user path is readable from the agent shell. Because the prompt exposes REPO_DIR and therefore the PID suffix, an agent can inspect vault-<pid>/git to recover hidden-test and golden refs before grading, invalidating local evaluation results; local mode should fail closed without isolation or keep the answer key behind a genuinely inaccessible boundary.
Useful? React with 👍 / 👎.
| grading_dir = LOGS_DIR / "pr-checkout" | ||
| shutil.rmtree(grading_dir, ignore_errors=True) | ||
| LOGS_DIR.mkdir(parents=True, exist_ok=True) | ||
| await repo_lib.run("git", "clone", "-q", str(REMOTE_DIR), str(grading_dir), cwd=LOGS_DIR) |
There was a problem hiding this comment.
When a target repository relies on ignored or setup-generated artifacts such as node_modules, a repo-local virtualenv, or compiled outputs, cloning the remote creates a tracked-source-only grading tree and discards the prepared artifacts that remain available in REPO_DIR. The subsequent test command therefore fails independently of the submitted patch; grade the PR head in the prepared worktree or reproduce the repository setup in this checkout.
Useful? React with 👍 / 👎.
| # so agent edits to them never survive into grading. | ||
| await repo_lib.git(REPO_DIR, "checkout", test_ref, "--", *(test_files or ["."])) | ||
|
|
||
| command = test_command.format(test_files=" ".join(test_files or [])) |
There was a problem hiding this comment.
If test_command contains braces for any other valid shell syntax—for example ${VAR}, an awk action, or inline JSON—str.format treats them as additional fields and raises before the grader runs. The template contract only requires expanding {test_files}, so replace that exact token rather than formatting the entire command; the duplicate formatting call in the SDLC path has the same failure mode.
Useful? React with 👍 / 👎.
| total = sum(c.get("weight", 1.0) for c in (bash_checks or [])) | ||
| total += 1.0 if grading_criteria else 0.0 |
There was a problem hiding this comment.
When the authored bash weights do not sum to 1.0, adding a fixed judge weight of 1.0 does not produce the documented 50/50 bash-versus-judge split. For example, the README's single bash check weighted 0.3 receives about 23% of the result while the judge receives about 77%, allowing the judge to dominate unexpectedly; normalize the bash checks within their half of the score before assigning the judge's half.
Useful? React with 👍 / 👎.
1 participant
What
Brings the coding template (the v6 rebuild with diff-based grading) and the cua template into the SDK repo under
environments/, as canonical, in-repo examples of v6 environment authoring. Each is a standalone uv project with its own lockfile, Dockerfile, ruff config, and tests.This is step 1 of making the SDK repo the source of truth for starter templates: follow-up PRs add a CI matrix that runs each template's tests against the SDK checkout (so SDK PRs that break a template fail at the PR boundary), release-gate oracle/null integrity runs, vendored
hud initpresets, and a mirror action back to the standalone template repos for the platform flow.Repo integration
.gitignore:environments/becomesenvironments/*+ negations for the vendored templates, so untracked local environment checkouts stay ignored. Each template's own.gitignorere-includes itsuv.lock(the root ignore is repo-wide).pyproject.toml:environments/added to hatch build excludes; root pyright/pytest scoping already excludes it, and ruff picks up each template's nested[tool.ruff].hudpins aligned at>=0.6.11; standalone-repo tooling (pre-commit) dropped from dev extras.Verification
uv run pytestwith the local SDK installed editable over the pin — 20 passed, 1 skipped (test_integration.pyneeds a built SWE-bench Pro instance image).test_local_rollout.pycovers the full serve/connect/run/grade path (golden=1.0, baseline=0.0) against the checkout.ruff check .andruff format --check .clean.Known follow-ups in the templates (deliberate, not drive-by)
pytestis a runtime dep;fastmcpused but undeclared; a few dead-defensive branches and a test asserting a private helper.Note
Low Risk
Mostly additive example/template code outside the core SDK package; no production runtime changes in the diff shown. Operational risk is limited to maintainers running Docker/SWE image builds or eval flows from the new directory.
Overview
Vendors the coding HUD v6 environment template into
environments/coding/as a standalone uv project (lockfile, Dockerfile, ruff, tests) and adjusts repo ignores so onlyenvironments/coding/andenvironments/cua/are tracked while other local environment checkouts stay ignored.The template implements diff-based grading with git history vaulting (
coding/repo.py), three task flavors inenv.py(coding-task,sdlc-taskwith mock GitHub MCP tools, SWE-bench Pro when instance assets are baked), sample rows intasks.py, andswe_tasks.pyfor fetching instances and building per-instance images viaDockerfile.hud. Tests cover offline contracts, hermetic local rollouts, and optional Docker integration against built SWE images.Reviewed by Cursor Bugbot for commit ea60e77. Bugbot is set up for automated code reviews on this repo. Configure here.
HUD-2276