feat(eval): Terminal-Bench 2.0 integration via a Harbor installed-agent adapter

[davidrhodus]

Runs hi on Terminal-Bench 2.0 (the arXiv:2607.09510 setting) through Harbor's container-installation mode:

• integrations/terminal_bench/hi_agent.py: uploads a Linux hi build into each task container, runs one --plain --no-save --allow-unverified turn (exit code never fails the trial — TB scoring is outcome-only via the task verifier), and feeds hi's --report usage + trajectory telemetry into Harbor's per-trial accounting.
• scripts/terminal_bench.sh (sample/full/task-glob modes) and docs/terminal-bench.md (amd64-only task images — containerized --platform linux/amd64 build; cost guidance).

First sample run (10 tasks, ipop/coder-balanced): 5/10 pass, zero harness exceptions, 6.4M input tokens — against the paper's 19–45% range across 21 systems (n=10, so indicative only). repeated_verify_failures from #88 fired on real TB tasks. Full 89-task run extrapolates to ~55–60M input tokens.

🤖 Generated with Claude Code

[davidrhodus]

Full 89-task baseline run ([email protected], ipop/coder-balanced, 1 attempt/task)

Pass rate: 42/89 = 47.2% · 6h13m wall (concurrency 3) · 66.4M input / 536k output tokens · 20 trials hit task timeout budgets (scored as attempted, mostly failures) · repair loop entered in 7 runs, repeated_verify_failures fired 8 times.

Context: the "Failure as a Process" study (arXiv:2607.09510) reports 19–45% across 21 model–scaffold systems on this same 89-task subset, with GPT-5+OpenHands at 45%. Different model generations (their data is May-2025 models), so this is not an apples-to-apples scaffold comparison — but it places hi + a 2026 mid-tier coding model above every configuration in that table.

PASSED (42): bn-fit-modify, break-filter-js-from-html, build-cython-ext, cancel-async-tasks, circuit-fibsqrt, cobol-modernization, custom-memory-heap-crash, db-wal-recovery, distribution-search, extract-elf, feal-differential-cryptanalysis, feal-linear-cryptanalysis, financial-document-processor, fix-code-vulnerability, fix-git, fix-ocaml-gc, git-leak-recovery, git-multibranch, largest-eigenval, log-summary-date-ranges, mailman, mcmc-sampling-stan, merge-diff-arc-agi-task, model-extraction-relu-logits, mteb-retrieve, multi-source-data-merger, nginx-request-logging, openssl-selfsigned-cert, password-recovery, path-tracing-reverse, polyglot-c-py, portfolio-optimization, protein-assembly, prove-plus-comm, pytorch-model-cli, query-optimize, regex-log, reshard-c4-data, sparql-university, sqlite-db-truncate, sqlite-with-gcov, tune-mjcf

FAILED (28): adaptive-rejection-sampler, build-pmars, build-pov-ray, configure-git-webserver, constraints-scheduling, count-dataset-tokens, dna-assembly, dna-insert, extract-moves-from-video, gcode-to-text, hf-model-inference, kv-store-grpc, large-scale-text-editing, modernize-scientific-stack, mteb-leaderboard, overfull-hbox, polyglot-rust-c, pypi-server, pytorch-model-recovery, raman-fitting, sam-cell-seg, sanitize-git-repo, schemelike-metacircular-eval, torch-pipeline-parallelism, torch-tensor-parallelism, video-processing, vulnerable-secret, winning-avg-corewars

ERRORED-scored-as-fail (19): caffe-cifar-10 (AgentTimeoutError), chess-best-move (AgentTimeoutError), code-from-image (AgentTimeoutError), compile-compcert (AgentTimeoutError), crack-7z-hash (AgentTimeoutError), filter-js-from-html (VerifierTimeoutError), gpt2-codegolf (AgentTimeoutError), headless-terminal (AgentTimeoutError), install-windows-3.11 (AgentTimeoutError), llm-inference-batching-scheduler (AgentTimeoutError), make-doom-for-mips (AgentTimeoutError), make-mips-interpreter (AgentTimeoutError), path-tracing (AgentTimeoutError), qemu-alpine-ssh (NonZeroAgentExitCodeError), qemu-startup (NonZeroAgentExitCodeError), regex-chess (AgentTimeoutError), rstan-to-pystan (AgentTimeoutError), train-fasttext (AgentTimeoutError), write-compressor (AgentTimeoutError)

🤖 Generated with Claude Code