feat(evals): add attested local and E2B Harbor runners

[kfallah]

Scope

This branch explored interchangeable Pi runner backends for Harbor evaluation, with local execution as the default and E2B as an explicit option.

Disposition

The implementation includes broader lifecycle machinery than the reusable backend seam requires. It is closed without merge, and the backend selection will be rebuilt against current main.

[greptile-apps]

Greptile Summary

This PR routes Harbor ground-truth evaluation through a new PiRunnerBackendSpec abstraction that supports both a local Docker backend (the default) and an explicit E2B accelerated backend. It adds full lifecycle attestation for every runner and task environment, a unified spend authority for E2B sandboxes and template builds, and crash-safe reconciliation via durable lease ledgers.

• Runner backend abstraction: LocalPiRunnerSpec and E2BPiRunnerSpec are Pydantic-discriminated variants that carry a canonical config_digest and a PiRunnerAttestation verified before any scored cell is accepted. The lease ledger (RunnerLeaseLedger) persists creating/active/retired state atomically using os.replace + fsync so crash recovery can prove or reap every resource.
• Budget integration: A single TimedResourceBudgetAccount governs E2B runner and task-environment sandboxes; reservations are bound by the same policy digest and ledger path as the provider hard cap, with fail-closed forfeit paths for ambiguous create or cleanup outcomes.
• E2B environment and build registry: ExactE2BEnvironment in e2b_environment.py (2 300+ lines) implements immutable build IDs, content-keyed build records, spend-limit attestation, and storage capacity validation; the new wmh harness register-e2b-build and prepare-e2b-build CLI commands expose this to operators.

Confidence Score: 4/5

Safe to merge; all findings are P2 quality/consistency observations with no correctness or data-integrity impact on the happy path.

The PR is a substantial, well-engineered addition with atomic lease ledgers, attested lifecycle policies, budget hard caps, and schema-versioned attestation. Two P2 issues were found: a missing capacity-error retry in _BudgetedProjectSandboxFactory (inconsistent with E2BOneShotRunnerFactory) and a five-branch combinatorial if/elif in default_sandbox_factory. Neither causes incorrect behavior today. Score is 4 rather than 5 because the retry gap is a meaningful consistency gap that could surface under E2B load.

wmh/agents/project.py (_BudgetedProjectSandboxFactory.call) and wmh/harness/e2b_sandbox.py (default_sandbox_factory) warrant a second look before the E2B path goes into high-volume use.

Important Files Changed

Filename	Overview
wmh/harness/pi_runner_backend.py	New file (1 102 lines) implementing LocalContainerRunnerFactory and E2BOneShotRunnerFactory with durable lease ledgers, multi-step attestation, and budget integration; lifecycle and resource isolation checks are thorough.
wmh/harness/e2b_sandbox.py	Adds SandboxLifecyclePolicy TypedDict, E2B constants, and reap_e2b_runner_lease; expands default_sandbox_factory with a 5-branch if/elif chain to avoid passing allow_internet_access=True explicitly — correct but more complex than necessary.
wmh/evals/harbor/e2b_environment.py	New 2 307-line file implementing ExactE2BEnvironment with content-keyed build registry, spend-limit attestation, storage capacity validation, and schema_version 3 attestation; well-tested.
wmh/evals/harbor/evaluator.py	Migrates from runner_image string to PiRunnerBackendSpec; adds E2B preflight, resource budget bindings, and runner environment digest tracking in run identity; logic is correct.
wmh/tracking/budget.py	Adds TimedResourceClass, TimedResourceRole, BudgetTerminalProvenance, and reconcile helpers; uses _budget_policy_dump/_budget_event_dump to strip new optional fields from serialized digests for backward-compatible ledger reads.
wmh/evals/harbor/results.py	Adds runner environment attestation and lease receipt validation with schema_version 2 (Docker) / 3 (E2B) discrimination; cross-trial lease replay detection added via _validate_unique_resource_leases.
wmh/agents/project.py	Adds _BudgetedProjectSandboxFactory with attestation and budget tracking for proposer project sandboxes; direct factory() call skips capacity-error retry present in E2BOneShotRunnerFactory.
wmh/evals/harbor/agent.py	Migrates WmhPiAgent from runner_image to PiRunnerBackendSpec; builds runner_factory per run() invocation so the lease receipt written to context metadata is always post-retirement.
wmh/evals/harbor/config.py	Wires EXACT_E2B_ENVIRONMENT_IMPORT_PATH for E2B backend, validates resource_budget_bindings in kwargs, adds allow_preexisting_e2b_builds flag.
wmh/evals/harbor/scorer.py	Removes earlier E2B block, migrates to PiRunnerBackendSpec, bumps configuration_id schema_version to 2, adds resource budget accounts.
wmh/cli/harness_eval.py	Adds register-e2b-build and prepare-e2b-build CLI commands; exposes runner backend selection through existing eval command.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Eval as HarborEvaluator
    participant Cfg as PiRunnerBackendSpec
    participant Fac as build_pi_runner_factory()
    participant LRF as LocalContainerRunnerFactory
    participant ERF as E2BOneShotRunnerFactory
    participant Ledger as RunnerLeaseLedger
    participant SB as E2B Sandbox
    participant Attest as PiRunnerAttestation

    Eval->>Cfg: "coerce_runner_spec(runner_image | spec)"
    Eval->>Fac: build_pi_runner_factory(spec, lease_ledger_dir)
    alt "backend == local"
        Fac->>LRF: LocalContainerRunnerFactory(spec)
    else "backend == e2b"
        Fac->>ERF: E2BOneShotRunnerFactory(spec, ledger)
    end

    Eval->>ERF: __enter__() / open()
    ERF->>Ledger: begin(lease_id, provider_expiry_horizon_s)
    ERF->>SB: create_sandbox(factory) with retries
    SB-->>ERF: sandbox handle
    ERF->>Ledger: activate(sandbox_id)
    ERF->>Attest: _attest(sandbox, get_info())
    Attest-->>ERF: "attestation schema_version=3"

    Eval->>ERF: run_pi_turn(runner_factory)
    ERF->>SB: commands.run(pi_node_cmd)
    SB-->>ERF: turn output

    ERF->>ERF: __exit__() / close()
    ERF->>SB: kill_sandbox() with retries
    alt kill success
        ERF->>Ledger: retire(lease_id)
    else kill failed
        ERF->>Ledger: cleanup_failed(lease_id)
        ERF->>ERF: reap_e2b_runner_lease(lease_id)
    end

    ERF-->>Eval: "lease_receipt state=retired or cleanup_failed"
    Eval->>Eval: write attestation + receipt to task metadata

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Eval as HarborEvaluator
    participant Cfg as PiRunnerBackendSpec
    participant Fac as build_pi_runner_factory()
    participant LRF as LocalContainerRunnerFactory
    participant ERF as E2BOneShotRunnerFactory
    participant Ledger as RunnerLeaseLedger
    participant SB as E2B Sandbox
    participant Attest as PiRunnerAttestation

    Eval->>Cfg: "coerce_runner_spec(runner_image | spec)"
    Eval->>Fac: build_pi_runner_factory(spec, lease_ledger_dir)
    alt "backend == local"
        Fac->>LRF: LocalContainerRunnerFactory(spec)
    else "backend == e2b"
        Fac->>ERF: E2BOneShotRunnerFactory(spec, ledger)
    end

    Eval->>ERF: __enter__() / open()
    ERF->>Ledger: begin(lease_id, provider_expiry_horizon_s)
    ERF->>SB: create_sandbox(factory) with retries
    SB-->>ERF: sandbox handle
    ERF->>Ledger: activate(sandbox_id)
    ERF->>Attest: _attest(sandbox, get_info())
    Attest-->>ERF: "attestation schema_version=3"

    Eval->>ERF: run_pi_turn(runner_factory)
    ERF->>SB: commands.run(pi_node_cmd)
    SB-->>ERF: turn output

    ERF->>ERF: __exit__() / close()
    ERF->>SB: kill_sandbox() with retries
    alt kill success
        ERF->>Ledger: retire(lease_id)
    else kill failed
        ERF->>Ledger: cleanup_failed(lease_id)
        ERF->>ERF: reap_e2b_runner_lease(lease_id)
    end

    ERF-->>Eval: "lease_receipt state=retired or cleanup_failed"
    Eval->>Eval: write attestation + receipt to task metadata

Loading

_{Reviews (3): Last reviewed commit: "fix(e2b): honor persisted orphan expiry" | Re-trigger Greptile}

[kfallah]

Addressed the fresh Greptile review at da8fc4b. Runner lease records now persist their own conservative provider expiry before create. Reconciliation prefers the prior sandbox observed end time plus clock skew, then that persisted bound, and uses the current spec horizon only for legacy records lacking both. This prevents shorter replacement specs from skipping cleanup and larger replacement specs from reaping a resource already guaranteed dead. The regression suite covers creating, active, cleanup-failed, retired, legacy, shrink, growth, and skew cases; 204 focused tests plus whole-project Ruff, format, and ty pass.

The storage-unit question is also verified: E2B documents diskTotal as bytes, matching the pinned Python SDK model and the existing byte-to-MiB conversion: https://e2b.dev/docs/sandbox/metrics

[kfallah]

Closing because this branch is superseded and is not intended for merge. Exact head preserved: da8fc4b. No branch was deleted.