Vendor coding and cua templates under environments/

.gitignore

@@ -61,7 +61,9 @@ hud/rl/checkpoints_test/
 
 docs/internal
 
-environments/
+environments/*
+!environments/coding/
+!environments/cua/
 
 experiments/
 .memories/

environments/coding/.dockerignore

@@ -0,0 +1,8 @@
+# The build context is this directory; keep it to what the images COPY, plus
+# the Dockerfile itself (remote builders read it from the context).
+*
+!Dockerfile.hud
+!env.py
+!coding
+!instances
+**/__pycache__

environments/coding/.gitignore

@@ -0,0 +1,36 @@
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.coverage
+.venv
+.env
+build/
+dist/
+*.egg-info/
+
+# OS / IDE
+.DS_Store
+.vscode/
+.idea/
+*.swp
+
+# HUD
+hud.lock.yaml
+.hud
+.hud_local/
+.hud_eval.toml
+.hud-images.json
+
+# SWE-bench Pro instance assets are fetched per-user (swe_tasks.py) and carry
+# the golden patches; only the empty .none dir (generic image builds) ships.
+instances/*
+!instances/.none/
+
+# The SDK repo ignores uv.lock at its root; this template pins its own so the
+# example resolves reproducibly.
+!uv.lock
+
+# Logs
+*.log

environments/coding/Dockerfile.hud

@@ -0,0 +1,73 @@
+# syntax=docker/dockerfile:1
+# One image definition for both flavors.
+#
+# Generic (default) — the repo-clone stage bakes your target repo into /app:
+#   docker build -f Dockerfile.hud --build-arg REPO_URL=https://github.com/you/repo .
+#   (private repos: --secret id=CODING_GITHUB_TOKEN,env=CODING_GITHUB_TOKEN)
+#
+# SWE-bench Pro (driven by swe_tasks.py) — BASE selects the instance's
+# prebuilt image (repo already at /app) and INSTANCE_ID bakes its assets:
+#   docker build -f Dockerfile.hud --build-arg BASE=jefzda/sweap-images:<tag> \
+#     --build-arg INSTANCE_ID=<id> .
+ARG BASE=repo-clone
+
+# ─── generic head: toolchain + target repo at /app ───
+FROM ubuntu:24.04 AS repo-clone
+
+RUN apt-get update -y \
+  && apt-get install -y --no-install-recommends \
+  bash build-essential ca-certificates curl git openssl \
+  python-is-python3 python3 python3-dev python3-pip python3-venv util-linux \
+  && rm -rf /var/lib/apt/lists/* \
+  && update-ca-certificates
+
+# The sample tasks' test command uses the image's python; adjust per repo.
+RUN pip3 install --break-system-packages pytest
+
+ARG REPO_URL="https://github.com/hud-evals/coding-template-sample"
+# chown in the same RUN as the clone (a separate layer would duplicate /app):
+# env.py serves agent shells as uid 1000, so the baked tree must be theirs.
+# Doing it at build makes it free at runtime, whatever the repo's size.
+RUN --mount=type=secret,id=CODING_GITHUB_TOKEN \
+    if [ -f /run/secrets/CODING_GITHUB_TOKEN ]; then \
+        auth="$(printf 'x-access-token:%s' "$(cat /run/secrets/CODING_GITHUB_TOKEN)" | openssl base64 -A)"; \
+        git -c "http.extraHeader=Authorization: Basic ${auth}" clone "${REPO_URL}" /app; \
+    else \
+        git clone "${REPO_URL}" /app; \
+    fi \
+    && chown -R 1000:1000 /app
+# Add a build step for the target repo here if it needs one (deps, compile).
+
+# ─── the served image: BASE + hud serving layer (+ instance assets) ───
+FROM ${BASE}
+
+# ".none" is a committed empty dir, so generic builds bake no instance and
+# env.py serves only the generic coding-task template.
+ARG INSTANCE_ID=".none"
+COPY env.py /hud/env.py
+COPY coding /hud/coding
+COPY instances/${INSTANCE_ID}/ /hud/instance/
+
+# Self-contained hud venv under /hud: bootstrap uv (with its own managed
+# Python), never touching the base image's Python or site-packages. chmod 700
+# keeps the vault and instance assets out of the uid-dropped agent's reach.
+RUN <<'INSTALL'
+set -eu
+export PATH="$HOME/.local/bin:$PATH" UV_INSTALL_DIR="$HOME/.local/bin"
+command -v uv >/dev/null 2>&1 || {
+  { command -v curl >/dev/null 2>&1 || command -v wget >/dev/null 2>&1; } \
+    || { apt-get update -qq && apt-get install -y -qq curl ca-certificates; } \
+    || apk add --no-cache curl ca-certificates
+  { command -v curl >/dev/null 2>&1 && curl -LsSf https://astral.sh/uv/install.sh | sh; } \
+    || { command -v wget >/dev/null 2>&1 && wget -qO- https://astral.sh/uv/install.sh | sh; } \
+    || pip install -q -U uv
+}
+uv python install 3.12
+uv venv /hud/venv --python 3.12
+uv pip install --python /hud/venv/bin/python 'hud>=0.6.11'
+chmod -R go-rwx /hud
+INSTALL
+ENV REPO_DIR=/app PYTHONPATH=/hud
+EXPOSE 8765
+ENTRYPOINT []
+CMD ["/hud/venv/bin/hud", "serve", "/hud/env.py", "--host", "0.0.0.0", "--port", "8765"]

environments/coding/README.md

@@ -0,0 +1,99 @@
+# Coding Environment (HUD v6)
+
+A coding environment: the agent gets a git repo over a sandboxed **`ssh` workspace** (the
+harness brings its own bash/file tools), and grading is diff-based — capture the agent's
+changes, reset to the pre-agent snapshot, re-apply the diff, bring in the hidden tests, run
+them. Generic tasks, SDLC workflow tasks, and SWE-bench Pro are flavors on this core.
+
+The agent never sees the answer key. Task setup moves the repo's real `.git` — which may hold
+solution branches or the fix commit — into a vault outside the workspace and leaves a fresh
+single-commit repo: git works normally, but there is no history, no refs, and no remotes.
+Grading discards the agent's `.git`, restores the vault, and checks hidden tests out *after*
+the agent's diff. In images, the vault and instance assets live under `/hud` (root, mode 700)
+and agent shells drop to a non-root uid via `setpriv`.
+
+This directory is a self-contained uv project — run every command below from it (`hud init`
+hands you a copy of it as your own environment package).
+
+## Layout
+
+- `env.py` — the environment: workspace wiring plus the three task templates.
+- `coding/repo.py` — the shared repo lifecycle: vault, snapshot, diff capture, reset, apply.
+- `coding/github.py` — mock GitHub for SDLC tasks: issue/PR store served as `github_*` tools.
+- `coding/swe_bench_pro.py` — the SWE-bench Pro grading pipeline.
+- `tasks.py` — sample generic and SDLC tasks.
+- `swe_tasks.py` — the SWE-bench Pro task source: fetches instances and builds their images
+  when run; task rows when imported.
+- `Dockerfile.hud` — the image definition for both flavors: `BASE` selects the generic
+  repo-clone head or a prebuilt instance image.
+
+## Generic tasks (`coding-task`)
+
+Point the env at a repo (`REPO_URL`; locally it clones per process, or bake it with
+`Dockerfile.hud`) and parameterize the template: a `base_ref` to start from, a `test_ref` whose
+`test_files` are the hidden tests (checked out from the vaulted history at grade time), and a
+`test_command` scored by exit code. Tasks follow the 3-branch convention — `{task}_baseline` /
+`{task}_test` / `{task}_golden`; `tasks.py` ships four sample bugs on
+[coding-template-sample](https://github.com/hud-evals/coding-template-sample):
+
+```bash
+uv sync
+hud set HUD_API_KEY=your-key-here
+hud eval tasks.py claude --task-ids sentry-fix -y --runtime local
+```
+
+## SDLC tasks (`sdlc-task`)
+
+The generic flavor plus workflow: the repo gets an `origin` remote (a bare mock-GitHub repo the
+agent pushes to) and `github_*` MCP tools seeded with the task's issues. The deliverable is a
+pushed branch with a pull request — grading checks the PR head out of the remote, brings in the
+hidden tests, runs the test command (weight 0.8), and scores the PR itself (0.2): a structural
+title/body check by default, or `pr_rubric` judged by `LLMJudgeGrader` when provided. See the
+`sentry-fix-pr` sample in `tasks.py`:
+
+```bash
+hud eval tasks.py claude --task-ids sentry-fix-pr -y --runtime local
+```
+
+## SWE-bench Pro tasks
+
+Each of the 731 public [SWE-bench Pro](https://github.com/scaleapi/SWE-bench_Pro-os) instances
+ships a prebuilt image (`jefzda/sweap-images:<tag>`, `linux/amd64`) with the repo and toolchain
+baked in. Running `swe_tasks.py` fetches the dataset row plus the official
+`run_script.sh`/`parser.py` into `instances/<id>/` and builds `Dockerfile.hud` with the
+instance's image as `BASE`, so the image serves this env from inside. Grading replays the
+official evaluator: resolved iff every `fail_to_pass` **and** `pass_to_pass` test passes.
+
+```bash
+uv run swe_tasks.py instance_NodeBB__NodeBB-04998908ba6721d64eba79ae3b65a351dcfbc5b5-vnan
+hud eval swe_tasks.py claude --task-ids nodebb-04998908
+uv run swe_tasks.py <id>... --push registry.io/acme   # push for cloud runtimes
+```
+
+## Tests
+
+```bash
+uv run pytest tests/ -q --ignore=tests/test_integration.py   # offline + hermetic local e2e
+uv run pytest tests/test_integration.py -v                   # SWE-bench gold-patch check (Docker)
+```
+
+`test_local_rollout.py` runs the generic and SDLC flavors end to end against a fixture 3-branch
+repo (no Docker or network): the golden ref grades 1.0 and the untouched baseline 0.0. The
+integration suite is the same check for built SWE-bench Pro instances.
+
+## Caveats
+
+- Public benchmarks are public: a networked agent could fetch solutions from GitHub. Disable
+  network egress at the runtime layer if that matters for your run.
+- Non-root local runs require bubblewrap. If it is unavailable, serving fails rather than
+  exposing vaulted answer-key refs to the agent process.
+- The uid wall needs `setpriv` (util-linux) in the image; the repo path comes from `REPO_DIR`
+  (`/app` in instance images).
+- The agent runs as uid 1000, so the baked repo must belong to it. The workspace only chowns
+  its own directory at start (O(1), keeps boot fast); the tree is owned where it's staged —
+  the generic build chowns `/app` in the clone step, and task setup re-chowns after root
+  mutates the worktree (checkout, vaulting).
+
+## Documentation
+
+See the [full docs](https://docs.hud.ai) for tasks, evaluation, and scaling.

environments/coding/coding/__init__.py

@@ -0,0 +1,5 @@
+"""The coding environment's core: repo-lifecycle primitives and task flavors."""
+
+from . import repo, swe_bench_pro
+
+__all__ = ["repo", "swe_bench_pro"]

environments/coding/coding/github.py

@@ -0,0 +1,158 @@
+"""A minimal mock GitHub for SDLC tasks.
+
+One in-process store holds issues (seeded per task) and the pull requests and
+comments the agent creates; :func:`serve` publishes it as ``github_*`` MCP
+tools over an ``mcp`` capability. The remote itself is a bare git repo on
+disk (see :func:`coding.repo.create_remote`) — the agent pushes branches with
+plain git and references them from pull requests here.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import socket
+from dataclasses import asdict, dataclass, field
+from typing import Any
+
+from fastmcp import FastMCP
+from hud.capabilities import Capability
+
+
+@dataclass
+class Issue:
+    number: int
+    title: str
+    body: str
+    labels: list[str] = field(default_factory=list)
+    state: str = "open"
+    comments: list[str] = field(default_factory=list)
+
+
+@dataclass
+class PullRequest:
+    number: int
+    title: str
+    body: str
+    head: str
+    base: str
+    state: str = "open"
+    comments: list[str] = field(default_factory=list)
+
+
+class MockGitHub:
+    """Issue/PR store with an action log (for rubric grading)."""
+
+    def __init__(self) -> None:
+        self.issues: dict[int, Issue] = {}
+        self.pull_requests: dict[int, PullRequest] = {}
+        self.actions: list[dict[str, Any]] = []
+
+    def seed(self, issues: list[dict[str, Any]]) -> None:
+        """Reset the store to the task's fixtures."""
+        self.issues = {
+            int(entry["number"]): Issue(
+                number=int(entry["number"]),
+                title=str(entry["title"]),
+                body=str(entry.get("body", "")),
+                labels=[str(label) for label in entry.get("labels", [])],
+            )
+            for entry in issues
+        }
+        self.pull_requests = {}
+        self.actions = []
+
+    def _record(self, action: str, **details: Any) -> None:
+        self.actions.append({"action": action, **details})
+
+    # ─── operations (the tool surface) ────────────────────────────────
+
+    def list_issues(self) -> list[dict[str, Any]]:
+        return [asdict(issue) for issue in self.issues.values()]
+
+    def get_issue(self, number: int) -> dict[str, Any]:
+        return asdict(self._issue(number))
+
+    def close_issue(self, number: int) -> dict[str, Any]:
+        issue = self._issue(number)
+        issue.state = "closed"
+        self._record("close_issue", number=number)
+        return asdict(issue)
+
+    def comment_on_issue(self, number: int, body: str) -> dict[str, Any]:
+        issue = self._issue(number)
+        issue.comments.append(body)
+        self._record("comment_on_issue", number=number, body=body)
+        return asdict(issue)
+
+    def create_pull_request(self, title: str, body: str, head: str, base: str) -> dict[str, Any]:
+        number = len(self.pull_requests) + 1
+        pr = PullRequest(number=number, title=title, body=body, head=head, base=base)
+        self.pull_requests[number] = pr
+        self._record("create_pull_request", number=number, title=title, head=head)
+        return asdict(pr)
+
+    def list_pull_requests(self) -> list[dict[str, Any]]:
+        return [asdict(pr) for pr in self.pull_requests.values()]
+
+    def _issue(self, number: int) -> Issue:
+        if number not in self.issues:
+            raise ValueError(f"no such issue: #{number}")
+        return self.issues[number]
+
+    # ─── grading views ────────────────────────────────────────────────
+
+    def latest_pull_request(self) -> PullRequest | None:
+        return next(reversed(self.pull_requests.values()), None)
+
+    def transcript(self) -> str:
+        """Everything the agent did here, for rubric judging."""
+        return json.dumps(
+            {
+                "issues": [asdict(issue) for issue in self.issues.values()],
+                "pull_requests": [asdict(pr) for pr in self.pull_requests.values()],
+                "actions": self.actions,
+            },
+            indent=2,
+        )
+
+
+def serve(github: MockGitHub) -> tuple[asyncio.Task[None], Capability]:
+    """Serve *github* as ``github_*`` MCP tools; returns the server task + capability."""
+    server: FastMCP = FastMCP(name="github")
+
+    @server.tool
+    def github_list_issues() -> list[dict[str, Any]]:
+        """List all issues in the repository."""
+        return github.list_issues()
+
+    @server.tool
+    def github_get_issue(number: int) -> dict[str, Any]:
+        """Get one issue by number."""
+        return github.get_issue(number)
+
+    @server.tool
+    def github_close_issue(number: int) -> dict[str, Any]:
+        """Close an issue."""
+        return github.close_issue(number)
+
+    @server.tool
+    def github_comment_on_issue(number: int, body: str) -> dict[str, Any]:
+        """Add a comment to an issue."""
+        return github.comment_on_issue(number, body)
+
+    @server.tool
+    def github_create_pull_request(title: str, body: str, head: str, base: str = "main") -> dict[str, Any]:
+        """Open a pull request from branch *head* (already pushed to the remote) into *base*."""
+        return github.create_pull_request(title, body, head, base)
+
+    @server.tool
+    def github_list_pull_requests() -> list[dict[str, Any]]:
+        """List all pull requests."""
+        return github.list_pull_requests()
+
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+        sock.bind(("127.0.0.1", 0))
+        port = int(sock.getsockname()[1])
+    task = asyncio.create_task(server.run_async(transport="http", host="127.0.0.1", port=port, show_banner=False))
+    return task, Capability.mcp(name="github", url=f"http://127.0.0.1:{port}/mcp")