Skip to content

Deepen fresh-execution release gate (deferred from #6) #30

Description

@ElbertePlinio

Follow-ups deferred from the PR for #6 (clean-replay grading). None block the v1.0 rigor bar; #6 already makes clean-replay THE grading path and the gate fail-closed on missing/mismatched provenance. These harden it further.

Deferred hardening (from the #6 review panel: Opus + GLM + codex-sol)

  • gradingMode independent cross-check. The gate trusts the self-reported gradingMode: 'clean_replay' string. Now mitigated by enforcing agentWorkspaceIsolation.restrictedPathsAbsent, but there is no fully independent post-hoc proof a separate grading dir was used. Consider recording + asserting the grading path differs from the agent-workspace path.
  • TOCTOU in hidden-fixture leak check. _hiddenFixtureIsolation probes the agent workspace after the run; an agent could in principle read + delete a leaked fixture before the probe. Primary defense (_shouldExcludeWorkspacePath never assembles _hidden into the agent workspace) is solid, so this is defense-in-depth. Fix: snapshot the workspace file manifest before the agent runs and diff post-run.
  • Full-bundle digest drift. The gate cross-checks the hidden-verifier digest subset against live bundles, but not the full taskBundleDigest. Evaluator-visible (non-hidden) fixture drift goes undetected. Lower risk than hidden-verifier drift.
  • Minor dedup. release_report.dart _stringMapEquals vs leaderboard_exporter.dart _mapEquals are near-identical.

Pre-existing flaky test (unrelated to #6, found while validating)

  • test/agent/agent_harness_test.dartMinimalAgentHarness bounds command output expects AgentRunStatus.failure on output flood but races to success. Fails deterministically in isolation on main (3/3), passes in the full parallel suite. Make the flood deterministic (sustained output that always trips the bound before completion).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions