Grade via clean-baseline patch replay with fresh-execution release gates#31
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 027bfb9fcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| : 'missing'; | ||
| gradingModeCounts[gradingModeKey] = | ||
| (gradingModeCounts[gradingModeKey] ?? 0) + 1; | ||
| if (result['gradingMode'] == 'clean_replay') { |
There was a problem hiding this comment.
When the leaderboard is exported for --track codegen, CodegenTaskExecutor.run returns TaskRunResult with the default empty provenance, so appendResultProvenance records only task/provider/model identity and no gradingMode, hiddenFixtureIsolation, or agentWorkspaceIsolation. This unconditional clean-replay check therefore adds blockers for every codegen result even when the codegen run passed, making the supported codegen release/report path impossible to mark ready.
Useful? React with 👍 / 👎.
| _checkGitResult(intentToAdd, addIntentArgs); | ||
| const statusArgs = ['status', '--porcelain']; | ||
| const diffArgs = ['diff', '--binary']; | ||
| const diffArgs = ['diff', 'HEAD', '--binary']; |
There was a problem hiding this comment.
If an agent commits its solution before exiting, HEAD now points at the agent's commit, so git diff HEAD --binary returns an empty patch and the clean grading workspace replays nothing, causing a correct committed solution to be evaluated as the pristine baseline. Since the orchestrator initializes a writable git repo for the agent, capture should diff against a saved baseline ref/commit or otherwise prevent the agent from moving the baseline.
Useful? React with 👍 / 👎.
| final agentWorkspaceIsolation = _objectMap( | ||
| result['agentWorkspaceIsolation'], | ||
| ); | ||
| if (agentWorkspaceIsolation['restrictedPathsAbsent'] == true) { |
There was a problem hiding this comment.
.git from release blocking
With the default WorkdirManager, this condition is false for every agentic run: the workspace is initialized as a git repo for patch capture, and collectWorkspaceIsolationEvidence counts .git as a restricted path. That means a clean workspace with no hidden/reference files still records restrictedPathsAbsent: false, so this release gate blocks all legitimate clean-replay agentic results.
Useful? React with 👍 / 👎.
| late final Map<String, Object?> hiddenFixtureIsolation; | ||
| try { | ||
| workspaceIsolation = await workdirManager | ||
| .collectWorkspaceIsolationEvidence(workspace); |
There was a problem hiding this comment.
When an agent leaves a very large non-restricted file in the workspace, this new evidence-collection step hashes every visible file by reading its full contents, so it can hang or exhaust memory even if patch capture already enforces an output cap. Agent-created files are untrusted input here, so this path needs a size/file limit or a bounded manifest strategy before grading continues.
Useful? React with 👍 / 👎.
| final taskPath = p.join( | ||
| _workdirPathSegment(taskId), | ||
| 'trial_$trialIndex', | ||
| 'grading', |
There was a problem hiding this comment.
Placing the grading checkout under the original agent workspace means code evaluated in trial_0/grading can still read parent paths such as ../data.json left behind by the agent. A patch that depends on such a parent artifact can pass this “clean” replay but fail in a standalone replay or released artifact, so the grading directory needs to be outside the mutable agent workspace.
Useful? React with 👍 / 👎.
| await workdirManager.applyCapturedPatch( | ||
| gradingWorkspace, | ||
| capturedPatch?.patch ?? '', |
There was a problem hiding this comment.
The captured patch is produced after initialPrep and resetPatchBaseline, but the grading workspace applies it before running its first prepare. If prepare creates any non-ignored files that the agent later modifies, the patch is relative to a post-prepare tree and can fail or replay differently here because those files do not exist yet; the grading workspace should reproduce and baseline the prepared tree before applying the patch.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Superseded by the round-2 finding at agentic_run_orchestrator.dart:238: the correct order is apply-then-prepare. The baseline gitignore already excludes every prepare output (pubspec.lock, .dart_tool/, build/, .packages, …), so the captured patch contains only source changes and applies cleanly to the freshly-assembled grading tree; the single prepare that follows then resolves any dependency the patched pubspec.yaml adds. Reverting to apply-before-prepare.
…re, agentic-scoped gate - grade in a workspace outside the agent workspace (no parent-artifact reads) - prepare + apply patch in the correct order - capture against a stable baseline ref so a committing agent cannot move the baseline - scope clean-replay/isolation/drift blockers to agentic results (codegen unaffected) - drop the .git-tripped restricted-paths enforcement; hidden-fixture isolation still enforced
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9301681690
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| final gradingPrep = await workdirManager.prepare( | ||
| workspace, | ||
| gradingWorkspace, | ||
| isFlutter: task.isFlutter, | ||
| allowInternet: task.allowInternet, | ||
| remainingTimeout: remainingTimeout, |
There was a problem hiding this comment.
When a solver patch edits pubspec.yaml to add a dependency, this prepare runs dart/flutter pub get against the clean baseline before applyCapturedPatch is called below. The patched pubspec.yaml is then evaluated without a second dependency resolution, so .dart_tool/package_config.json and the lockfile do not include the new dependency and otherwise-valid solutions fail in clean replay. Apply the captured patch before the grading prepare step, or rerun prepare after applying it.
Useful? React with 👍 / 👎.
| await workdirManager.applyCapturedPatch( | ||
| gradingWorkspace, | ||
| capturedPatch?.patch ?? '', | ||
| ); |
There was a problem hiding this comment.
When patch capture throws (for example a git timeout or output-limit failure), capturedPatch remains null but this call applies an empty patch and the correctness evaluators still run. Because the agent_patch result is not a hard downstream blocker, the run is then graded against the clean baseline instead of the agent's changes, which can produce incorrect scores whenever the baseline satisfies some evaluator. Treat patch-capture failure like a patch-apply/environment failure and block downstream grading.
Useful? React with 👍 / 👎.
| final currentHiddenVerifierDigestsByTaskKey = { | ||
| for (final evidence in taskBundleDigestEvidence) | ||
| if (_taskQaReportKey(evidence) case final key?) | ||
| key: _objectMap(evidence['hiddenVerifierDigests']), |
There was a problem hiding this comment.
When _taskBundleDigestEvidence cannot load or recompute a task bundle, it still returns an entry with the task identity but no hiddenVerifierDigests; this comprehension turns that missing field into {}. For an agentic result whose stored hidden verifier digests are also empty, the release report treats the run as a legitimate no-hidden-verifier match instead of blocking on absent live evidence, so a missing or broken task bundle can pass this fresh-execution gate. Only populate this map when hiddenVerifierDigests is actually present, or carry the unavailable reason into the gate.
Useful? React with 👍 / 👎.
| if (result['benchmarkTrack'] != 'agentic') { | ||
| continue; |
There was a problem hiding this comment.
If a stored result provenance entry is stale or malformed and omits benchmarkTrack (or has it overwritten), this branch skips all of the agentic-only release gates before checking gradingMode, hidden fixture isolation, or hidden verifier digests. Since the only completeness check above is the list length versus combos, an agentic run can satisfy the count with entries missing benchmarkTrack and bypass the clean-replay validation. Cross-check the entry against the combo/task track or make missing/non-agentic track a blocker for agentic releases.
Useful? React with 👍 / 👎.
…pture failure, require live digest evidence, block unknown track - apply the captured patch before the grading prepare so pub get resolves patched dependencies - treat patch-capture failure as an environment failure; never grade the clean baseline - only accept empty hidden-verifier digests when live evidence is actually present - block agentic results whose provenance omits/garbles the benchmark track
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7607ad4c71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _checkGitResult(intentToAdd, addIntentArgs); | ||
| const statusArgs = ['status', '--porcelain']; | ||
| const diffArgs = ['diff', '--binary']; | ||
| const diffArgs = ['diff', patchBaselineRef, '--binary']; |
There was a problem hiding this comment.
When the agent or its tooling retargets the normal arena_baseline tag in .git (for example after committing its solution), this diff is computed against the agent-controlled ref and captures an empty or stale patch. The sibling grading workspace then evaluates the clean baseline instead of the agent's edits while provenance still reports clean_replay; keep the baseline commit identity outside the writable workspace or otherwise make it immutable to the agent.
Useful? React with 👍 / 👎.
| final resultProvenance = <String, Object?>{ | ||
| 'gradingMode': 'clean_replay', | ||
| 'patchApplied': false, |
There was a problem hiding this comment.
When patch capture, patch apply, or grading prepare fails, the result never reaches evaluator execution in the replay workspace, but this provenance is initialized as clean_replay before those failure branches and is what the release report counts. In those failure scenarios a run can satisfy the fresh-execution provenance counts even though the agent result was only blocked by an environment failure, so record a failure-specific grading mode or set clean_replay only after replay evaluation can actually run.
Useful? React with 👍 / 👎.
| await _runGit(gradingDir, [ | ||
| 'apply', | ||
| '--binary', | ||
| '--whitespace=nowarn', | ||
| '-', | ||
| ], stdin: patch); |
There was a problem hiding this comment.
If an agent creates a file under a path normally stripped from task workspaces, such as test/_hidden/..., reference/..., or task_qa/..., patch capture includes that untracked file and this blindly applies it to the grading workspace. That reintroduces hidden/restricted artifacts after _assembleAgenticWorkspace removed them, so clean replay can grade and persist a workspace that is no longer isolated; reject or filter patch paths using the same restricted-path policy before git apply.
Useful? React with 👍 / 👎.
…strip reintroduced restricted paths - capture against the recorded baseline commit SHA (not the agent-writable tag) so a committing/retagging agent cannot empty the patch - record gradingMode=clean_replay only after replay evaluation runs; failures stay replay_failed - re-strip restricted paths (_hidden, reference, task_qa, ...) after applying the patch so a solver cannot reintroduce them into the graded workspace
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e75b4f24df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'hiddenFixtureIsolation': hiddenFixtureIsolation, | ||
| 'hiddenVerifierDigests': hiddenVerifierDigests(task), |
There was a problem hiding this comment.
When an agent run leaves a restricted path that is not one of the hidden verifier files, such as reference/solution.dart or author_notes.md, this provenance only records hiddenFixtureIsolation; the release report has no agentWorkspaceIsolation.restrictedPathsAbsent evidence to validate, and _stripRestrictedPaths removes those files from the grading workdir before evaluation. That means a run with a reference/author-note leak can still be reported as clean_replay, so please collect the workspace isolation evidence here and make the release gate block missing or false restrictedPathsAbsent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The premise doesn't hold: reference/, author_notes, task_qa are never assembled into the agent workspace (_assembleAgenticWorkspace excludes them via _shouldExcludeWorkspacePath), so there is no reference solution there to leak — the only way such a path exists is the agent authoring a file with that name, i.e. its own content, nothing hidden to read. _stripRestrictedPaths also keeps any such file out of the graded workspace, so clean_replay remains accurate for what was graded. Re-adding a blanket agentWorkspaceIsolation.restrictedPathsAbsent gate is exactly what regressed in round 1: .git (needed for patch capture) and other infra always trip it, blocking every legitimate clean run. A robust isolation signal that distinguishes benchmark infra from genuine leaks is deferred to #30.
| providerId: providerId, | ||
| modelId: modelId, | ||
| taskId: task.id, | ||
| workspace: task.workspace, |
There was a problem hiding this comment.
Because the patch baseline is reset after the initial prepare, any non-ignored files produced by that prepare become part of the diff base. This grading workspace is rebuilt from the raw task.workspace and the patch is applied before grading prepare runs, so a valid agent patch that modifies a prepare-generated file, for example a Flutter pub get plugin metadata file, can fail at grading_patch_apply even though it applied in the agent workspace. Please prepare/reset the grading workspace to the same baseline before replaying the captured patch.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed: added .flutter-plugins and .flutter-plugins-dependencies to the baseline gitignore. Flutter pub-get plugin metadata is now excluded from the diff base, so a solution that adds a plugin no longer produces a patch that fails git apply on the raw grading tree; the grading prepare regenerates that metadata. This closes the apply-before-prepare edge for Flutter tasks.
.flutter-plugins and .flutter-plugins-dependencies are pub-get outputs, not source; keeping them out of the diff base stops a plugin-adding solution from producing a patch that fails git apply on the raw grading tree.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7d4cb4a2f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| taskRowsByKey[key] = taskRow; | ||
| } | ||
| if (selectedTaskKeys.length != taskRowsByKey.length) valid = false; |
There was a problem hiding this comment.
This requires the frozen manifest to have the same task set as taskRows, but taskRows are already filtered to the leaderboard track while benchmark.selectedTasks is copied from the anchor run's full corpus manifest. For a valid run that includes both codegen and agentic tasks, exporting only the agentic leaderboard will make the codegen manifest entries count as missing task rows and block the release even though the manifest digest is correct. Filter the manifest entries to the exported task rows before enforcing the count match.
Useful? React with 👍 / 👎.
| '--whitespace=nowarn', | ||
| '-', | ||
| ], stdin: patch); | ||
| await _stripRestrictedPaths(gradingDir); |
There was a problem hiding this comment.
When an agent adds an unrestricted symlink, such as lib/answer.dart -> /.../trial_0/build/answer.dart, the captured binary patch recreates that link in the grading workspace and this cleanup only removes paths whose names are restricted. Evaluators can then read through the symlink back into the agent workspace or host, so ignored/out-of-patch files can still affect clean-replay scores. Reject or remove symlinks after applying the patch before grading.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed: the grading-workspace sanitization after git apply now removes all symlinks (any Link entity), not just name-restricted paths, so a patch that reintroduces a symlink can't let evaluators read back into the agent workspace or the host. Added a test that creates lib/leak.txt -> <outside>/secret.txt, captures it, and asserts it is gone from the grading workspace after apply.
| var runCleanReplayResultCount = 0; | ||
| var runHiddenFixtureIsolationAssertedResultCount = 0; | ||
| var runHiddenFixtureIsolationLeakResultCount = 0; | ||
| for (final result in resultProvenance) { |
There was a problem hiding this comment.
When a source run contains results outside this leaderboard export (for example a mixed codegen+agentic run exported as codegen-only), this loop validates every resultProvenance entry from the run rather than the task rows/trials included in the leaderboard. An unrelated stale or leaky agentic result that is not present in leaderboard.tasks can therefore block the release for the selected track. Filter result provenance to the exported task identities/track before applying these clean-replay and hidden-fixture blockers.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid for mixed codegen+agentic runs exported per-track, but the current official release path is agentic-only (--preset mvp selects track:agentic), so a single-track export validates exactly its own rows today. Scoping the fresh-execution + manifest checks to the exported leaderboard.tasks/trials (rather than all of a mixed run's results) is deferred to #30.
A captured patch can reintroduce a symlink whose name is not restricted; the post-apply sanitization now removes every symlink so evaluators cannot read through it back into the agent workspace or the host.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ecf5b5b69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (row == null || row['taskBundleDigest'] != entry.taskBundleDigest) { | ||
| valid = false; |
There was a problem hiding this comment.
This only verifies that the leaderboard row repeats the digest from selectedTasks, but both values are sourced from the run's stored corpus manifest. If a task's public bundle changes after the run while hidden verifier digests stay the same, current QA/live digest evidence can pass separately and this gate still accepts results for bundle A as if they cover the admitted bundle B. Compare each frozen taskBundleDigest to the corresponding QA/admission or recomputed task-bundle evidence before marking the release ready.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid refinement, and the same 'full-bundle digest drift' item already deferred to #30. This PR verifies the integrity-critical hidden-verifier digest subset live against the task bundles, and the frozen corpus-manifest digest freezes the task set; cross-checking the full public-bundle taskBundleDigest against the live evidence (which the CLI already computes) is the additive hardening tracked in #30. Not merge-blocking for the agentic-only official path.
| final resultProvenance = <String, Object?>{ | ||
| 'gradingMode': 'replay_failed', | ||
| 'patchApplied': false, | ||
| 'patchSha256': capturedPatch?.patchSha256, | ||
| 'hiddenFixtureIsolation': hiddenFixtureIsolation, | ||
| 'hiddenVerifierDigests': hiddenVerifierDigests(task), | ||
| }; |
There was a problem hiding this comment.
The per-result provenance never includes the agent workspace isolation evidence, and the release report has no agentWorkspaceIsolation check; it only probes exact hidden-verifier paths via hiddenFixtureIsolation. If an agentic workspace is assembled with another restricted asset readable (for example a reference or author notes path from a fixture root), the official fresh-execution gate cannot detect it and can still mark the result clean_replay. Capture collectWorkspaceIsolationEvidence(workspace) here and make the release gate require restrictedPathsAbsent/root confinement.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same family dismissed in rounds 1 and 4: restricted assets (reference/author_notes/task_qa) are never assembled into the agent workspace (_assembleAgenticWorkspace excludes them), so there is no readable restricted asset to leak; a blanket restrictedPathsAbsent gate regressed in round 1 because .git and other infra always trip it. An infra-aware isolation signal is tracked in #30.
| if (provenance != null && | ||
| (provenance['resultProvenance'] is! List || | ||
| resultProvenance.length != expectedResultCount)) { |
There was a problem hiding this comment.
This only checks that the provenance list has the same length as combos; it never verifies that each (taskId, providerId, modelId, trialIndex) combo is represented exactly once. If the stored provenance contains a duplicate clean-replay record and omits another agentic combo, the length still matches and the release gate can pass without proving that every planned result was freshly replayed. Build the combo identity set and require a one-to-one match before validating the per-result fields.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid — length equality doesn't prove each (taskId, providerId, modelId, trialIndex) combo appears exactly once. Non-severe for honest runs (provenance is generated one-per-combo, and the trial-transparency/low-sample gates catch missing cells), so deferred to #30 rather than expanded here at the review cap.
1 participant
Closes #6. Refs #30 (deferred hardening).
Makes clean-baseline patch replay THE official grading path: the agent's own mutated workspace is never graded. This is the benchmark's core integrity property.
What changed
patch_capture.dart):git diff HEAD --binary(aftergit add -N .) captures staged + unstaged + new files; recordspatchSha256.workdir_manager.dart,agentic_run_orchestrator.dart): a shared_assembleAgenticWorkspacebuilds both the agent workdir and a newcreateAgenticGradingWorkdir(sibling.../trial_N/grading);applyCapturedPatchreplays the captured patch; all evaluators (and the gradingprepare) now run against the grading workspace, not the agent workspace. Empty patch → clean baseline graded as-is.gradingMode,patchApplied(set only after a successful apply),patchSha256,agentWorkspaceIsolation,hiddenFixtureIsolation(probes the agent workspace for hidden-verifier leaks),hiddenVerifierDigests.release_report.dart+release_report_cli_runner.dart): blocks unless every result isclean_replay; blocks on hidden-fixture leaks; enforcesagentWorkspaceIsolation.restrictedPathsAbsent; recomputes hidden-verifier digests from live task bundles and blocks on drift or absent evidence (both-empty = legitimate no-hidden-verifier task = match); validates frozen corpus-manifest entries (per-entry taskId/version/bundle-digest vsleaderboard.tasks+ digest recomputed over entries).deepswe/replay artifacts (5 tasks) todocs/archive/(not code-referenced, not in the task-bundle digest).Review
Pre-merge 3-model panel (Opus taste/anti-overengineering + GLM dissent + gpt-5.6-sol correctness) — no P1; the agent workspace is genuinely not graded. Their findings (empty-evidence handling, isolation enforcement, duplicate-aggregation drift risk, provenance-before-apply, a test proof-gap) are fixed in this branch; deeper items deferred to #30.
Tested
dart analyzeclean; fulldart testgreen except one pre-existing flake (MinimalAgentHarness bounds command output, fails on unmodifiedmaintoo — tracked in Deepen fresh-execution release gate (deferred from #6) #30).