feat(eval): Terminal-Bench 2.0 integration via a Harbor installed-agent adapter

docs/terminal-bench.md

@@ -0,0 +1,80 @@
+# Running hi on Terminal-Bench 2.0
+
+[Terminal-Bench](https://github.com/harbor-framework/terminal-bench) is the
+standard benchmark for terminal coding agents; [Harbor](https://github.com/harbor-framework/harbor)
+is its official harness. Each task runs in its own Docker environment; Harbor
+installs the agent into the container, hands it the task instruction, and
+scores the resulting container state with the task's own tests — outcome-only,
+so intermediate behavior earns no credit. This is the setting studied in
+"Failure as a Process" (arXiv:2607.09510), whose 21 model–scaffold baselines
+span 19–45% pass rate.
+
+## One-time setup
+
+1. **Harbor** (needs Python ≥3.12 and Docker running):
+
+   ```bash
+   uv tool install harbor
+   ```
+
+2. **A Linux build of `hi`** matching the task-image architecture. The
+   published Terminal-Bench 2.0 task images are **linux/amd64 only** — even on
+   Apple Silicon (Docker runs them under Rosetta), the binary must be x86_64.
+   The workspace's GPU crates don't build in the container, but the binary
+   crate builds alone; the reproducible path is a containerized build:
+
+   ```bash
+   docker run --rm --platform linux/amd64 -v "$PWD":/src:ro \
+     -v "$PWD/target-linux-amd64":/out \
+     rust:1-bookworm cargo build --release -p hi \
+     --manifest-path /src/Cargo.toml --target-dir /out
+   export HI_AGENT_BINARY="$PWD/target-linux-amd64/release/hi"
+   ```
+
+   (An emulated build on Apple Silicon takes several minutes; it only needs
+   to happen once per hi revision.)
+
+## Running
+
+```bash
+export PIPENETWORK_API_KEY=...        # or the provider your model needs
+scripts/terminal_bench.sh sample      # ~10 tasks, plumbing + cost check
+scripts/terminal_bench.sh full        # full dataset — mind the spend
+scripts/terminal_bench.sh task 'git*' # tasks matching a glob
+```
+
+Model selection: `HI_TB_MODEL=<provider>/<model>` (default
+`pipenetwork/ipop/coder-balanced`). Known providers: `pipenetwork`,
+`anthropic`, `openrouter`, `openai` — the adapter maps them to `hi --provider`
+and forwards the matching API key from your environment into the container.
+
+Results land under `jobs/terminal-bench/<job-name>/` — per-trial directories
+with the agent transcript (`hi.txt`), hi's own `--report` telemetry
+(`hi-report.json`, including `repeated_verify_failures` and the trajectory
+timeline), and Harbor's verifier verdicts. Harbor prints the aggregate pass
+rate at the end of the job.
+
+## How the adapter works
+
+`integrations/terminal_bench/hi_agent.py` implements Harbor's
+`BaseInstalledAgent`:
+
+- `install()` uploads `$HI_AGENT_BINARY` into the container at `/opt/hi/hi`
+  and symlinks it onto `PATH`.
+- `run()` executes one `hi --plain --no-save --allow-unverified` turn with the
+  task instruction, `--report` pointed at Harbor's log directory, and
+  `XDG_CONFIG_HOME` isolated so a config file baked into a task image can
+  never shadow the model routing. The command never fails the trial on hi's
+  exit code — scoring is the verifier's job.
+- `populate_context_post_run()` reads the downloaded report and fills Harbor's
+  token accounting plus a `hi_telemetry` metadata block.
+
+## Caveats
+
+- **Architecture**: the uploaded binary must match the task images. Harbor on
+  Apple Silicon uses aarch64 images by default; build accordingly.
+- **Timeouts**: Terminal-Bench tasks carry their own generous agent timeouts;
+  hi's dynamic step caps apply within them. Use `--timeout-multiplier` on
+  `harbor run` for slow models.
+- **Cost**: the full dataset is hundreds of agent turns. Always run `sample`
+  first and extrapolate spend before `full`.

integrations/terminal_bench/hi_agent.py

@@ -0,0 +1,132 @@
+"""Harbor (Terminal-Bench 2.0) installed-agent adapter for `hi`.
+
+Harbor spins up each task's Docker environment, calls ``install()`` to place
+the agent inside it, runs the task instruction through ``run()``, then scores
+the resulting container state with the task's own tests (outcome-only, like
+the Terminal-Bench paper protocol). This adapter uploads a Linux build of
+``hi`` (see docs/terminal-bench.md for producing one), runs it one-shot with
+``--report``, and surfaces hi's token usage and trajectory telemetry back to
+Harbor.
+
+Usage::
+
+    export HI_AGENT_BINARY=/path/to/linux/hi
+    export PIPENETWORK_API_KEY=...
+    harbor run -d [email protected] \
+        --agent integrations.terminal_bench.hi_agent:HiAgent \
+        -m pipenetwork/ipop/coder-balanced -n 4
+"""
+
+import json
+import os
+import shlex
+from pathlib import Path
+from typing import override
+
+from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
+from harbor.environments.base import BaseEnvironment
+from harbor.models.agent.context import AgentContext
+
+_REPORT = "/logs/agent/hi-report.json"
+_OUTPUT = "/logs/agent/hi.txt"
+
+# Harbor model prefix (-m "<provider>/<model>") → (hi --provider, key sources).
+_PROVIDERS = {
+    "pipenetwork": ("pipenetwork", ["PIPENETWORK_API_KEY", "HI_API_KEY"]),
+    "anthropic": ("anthropic", ["ANTHROPIC_API_KEY", "HI_API_KEY"]),
+    "openrouter": ("openai", ["OPENROUTER_API_KEY", "HI_API_KEY"]),
+    "openai": ("openai", ["OPENAI_API_KEY", "HI_API_KEY"]),
+}
+
+
+class HiAgent(BaseInstalledAgent):
+    """Container-installed `hi` driven one-shot per task instruction."""
+
+    @staticmethod
+    @override
+    def name() -> str:
+        return "hi"
+
+    @override
+    def get_version_command(self) -> str | None:
+        return "hi --version"
+
+    @override
+    async def install(self, environment: BaseEnvironment) -> None:
+        binary = os.environ.get("HI_AGENT_BINARY", "")
+        if not binary or not Path(binary).is_file():
+            raise ValueError(
+                "HI_AGENT_BINARY must point at a Linux build of `hi` matching "
+                "the task image architecture — see docs/terminal-bench.md"
+            )
+        await self.exec_as_root(environment, command="mkdir -p /opt/hi")
+        await environment.upload_file(binary, "/opt/hi/hi")
+        await self.exec_as_root(
+            environment,
+            command=(
+                "chmod 755 /opt/hi/hi && ln -sf /opt/hi/hi /usr/local/bin/hi "
+                "&& hi --version"
+            ),
+        )
+
+    @with_prompt_template
+    async def run(
+        self,
+        instruction: str,
+        environment: BaseEnvironment,
+        context: AgentContext,
+    ) -> None:
+        if not self.model_name or "/" not in self.model_name:
+            raise ValueError(
+                "pass -m <provider>/<model>, e.g. pipenetwork/ipop/coder-balanced"
+            )
+        provider, model = self.model_name.split("/", 1)
+        if provider not in _PROVIDERS:
+            raise ValueError(
+                f"unknown provider {provider!r}; known: {sorted(_PROVIDERS)}"
+            )
+        hi_provider, key_sources = _PROVIDERS[provider]
+
+        env: dict[str, str] = {}
+        for key in key_sources:
+            value = os.environ.get(key)
+            if value:
+                env["HI_API_KEY"] = value
+                break
+        # Model routing comes entirely from flags/env — never let a config
+        # file baked into the task image shadow it.
+        env["XDG_CONFIG_HOME"] = "/tmp/hi-xdg"
+
+        # `|| true`: Terminal-Bench scoring is outcome-only. An unverified or
+        # incomplete hi turn exits nonzero, but the task tests — not the exit
+        # code — decide the trial, so the verifier must always get to run.
+        command = (
+            "{ hi --plain --no-save --allow-unverified "
+            f"--provider {hi_provider} -m {shlex.quote(model)} "
+            f"--report {_REPORT} "
+            f"{shlex.quote(instruction)} || true; }} "
+            f"2>&1 </dev/null | tee {_OUTPUT}"
+        )
+        await self.exec_as_agent(environment, command=command, env=env)
+
+    @override
+    def populate_context_post_run(self, context: AgentContext) -> None:
+        report_path = self.logs_dir / "hi-report.json"
+        if not report_path.exists():
+            return
+        try:
+            report = json.loads(report_path.read_text())
+        except (OSError, json.JSONDecodeError):
+            return
+        # Schema v2 nests usage as {"session": {...}, "turn": {...}}; older
+        # reports carried flat top-level token fields.
+        usage = report.get("usage") or {}
+        session = usage.get("session") or usage
+        context.n_input_tokens = session.get("input_tokens") or report.get("input_tokens")
+        context.n_cache_tokens = session.get("cache_read_tokens")
+        context.n_output_tokens = session.get("output_tokens")
+        context.metadata = {
+            "hi_outcome": report.get("outcome"),
+            "hi_telemetry": report.get("telemetry"),
+            "hi_verification": report.get("verification"),
+        }

scripts/terminal_bench.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+# Run hi on Terminal-Bench 2.0 via Harbor.
+#
+# Usage:
+#   scripts/terminal_bench.sh sample          # 10-task health-check sample
+#   scripts/terminal_bench.sh full            # the full dataset (mind cost!)
+#   scripts/terminal_bench.sh task <glob>     # tasks matching a name glob
+#
+# Requires: docker running, `uv tool install harbor`, a Linux build of hi at
+# $HI_AGENT_BINARY (see docs/terminal-bench.md), and PIPENETWORK_API_KEY (or
+# override HI_TB_MODEL, e.g. anthropic/claude-sonnet-5 + ANTHROPIC_API_KEY).
+set -euo pipefail
+cd "$(dirname "$0")/.."
+# harbor runs in its own uv tool environment; the adapter module lives in this
+# repo, so put the repo root on PYTHONPATH for the import-path --agent.
+export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}$PWD"
+
+MODE="${1:-sample}"
+MODEL="${HI_TB_MODEL:-pipenetwork/ipop/coder-balanced}"
+DATASET="${HI_TB_DATASET:[email protected]}"
+JOBS_DIR="${HI_TB_JOBS_DIR:-jobs/terminal-bench}"
+CONCURRENT="${HI_TB_CONCURRENT:-4}"
+
+: "${HI_AGENT_BINARY:?set HI_AGENT_BINARY to a Linux build of hi (docs/terminal-bench.md)}"
+[ -f "$HI_AGENT_BINARY" ] || { echo "HI_AGENT_BINARY not found: $HI_AGENT_BINARY"; exit 2; }
+docker ps >/dev/null || { echo "docker daemon not running"; exit 2; }
+
+ARGS=(
+  run
+  --dataset "$DATASET"
+  --agent integrations.terminal_bench.hi_agent:HiAgent
+  --model "$MODEL"
+  --n-concurrent "$CONCURRENT"
+  --jobs-dir "$JOBS_DIR"
+  --agent-include-logs 'hi-report.json'
+  --agent-include-logs 'hi.txt'
+)
+
+case "$MODE" in
+  sample)
+    # A fixed, arbitrary-but-stable spread of ten real TB-2.0 tasks for
+    # plumbing + cost checks before a full run.
+    for t in adaptive-rejection-sampler build-cython-ext configure-git-webserver \
+             db-wal-recovery chess-best-move cobol-modernization \
+             count-dataset-tokens circuit-fibsqrt cancel-async-tasks \
+             break-filter-js-from-html; do
+      ARGS+=(--include-task-name "$t")
+    done
+    ;;
+  full) ;;
+  task)
+    ARGS+=(--include-task-name "${2:?usage: terminal_bench.sh task <glob>}")
+    ;;
+  *)
+    echo "unknown mode: $MODE (sample|full|task <glob>)"; exit 2 ;;
+esac
+
+exec harbor "${ARGS[@]}"