Skip to content

feat: add preset selection and corpus manifest #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ElbertePlinio merged 5 commits into from
Jul 10, 2026
Merged

feat: add preset selection and corpus manifest #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cceaa37
feat: add preset selection and corpus manifest
ElbertePlinio Jul 10, 2026
4df9019
fix: clean errors for empty and unknown presets
ElbertePlinio Jul 10, 2026
1efdc8f
fix: forward-compatible schema readers, preset override and admitted-…
ElbertePlinio Jul 10, 2026
7353f65
fix: verify preset QA identity, propagate leaderboard schema version,…
ElbertePlinio Jul 10, 2026
1ef1aff
fix: put official runs on preset path, verify admitted bundle digest,…
ElbertePlinio Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions app/bin/dart_arena_headless.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ Map<String, Object?> _helpJson() {
'dart run --verbosity=error dart_arena:dart_arena_headless --config run.json',
'options': [
{'name': '--config', 'value': 'path', 'required': true},
{'name': '--preset', 'value': 'mvp', 'required': false},
{'name': '--help', 'required': false},
],
'configFormat': 'json',
Expand Down
41 changes: 41 additions & 0 deletions app/lib/core/task_bundle_digest.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,47 @@ const taskBundleDigestFileRoots = [
'negative_cases',
];

class CorpusManifestEntry {
const CorpusManifestEntry({
required this.taskId,
required this.taskVersion,
required this.taskBundleDigest,
});

final String taskId;
final int taskVersion;
final String taskBundleDigest;

Map<String, Object?> toJson() => {
'taskId': taskId,
'taskVersion': taskVersion,
'taskBundleDigest': taskBundleDigest,
};
}

String corpusManifestDigestSha256(Iterable<CorpusManifestEntry> entries) {
final sorted = entries.toList()
..sort((a, b) {
final id = a.taskId.compareTo(b.taskId);
if (id != 0) return id;
final version = a.taskVersion.compareTo(b.taskVersion);
if (version != 0) return version;
return a.taskBundleDigest.compareTo(b.taskBundleDigest);
});
return sha256
.convert(
utf8.encode(
sorted
.map(
(entry) =>
'${entry.taskId}\u0000${entry.taskVersion}\u0000${entry.taskBundleDigest}',
)
.join('\n'),
),
)
.toString();
}

Future<String> taskBundleDigestSha256(Directory bundleDirectory) async {
final files = await _taskBundleDigestFiles(bundleDirectory);
final bytesBuilder = BytesBuilder(copy: false);
Expand Down
37 changes: 36 additions & 1 deletion app/lib/export/leaderboard_exporter.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Future<Map<String, Object?>> buildLeaderboardExport(
for (final runId in sortedRunIds)
if (runsById[runId] != null) runsById[runId]!,
];
final corpusManifest = _corpusManifestForRun(runsById[selected.anchorRunId]);

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate corpus manifest before aggregating runs

For aggregate-compatible exports this takes the corpus manifest only from the anchor run, while selected.taskRuns may include other runs whose compatibility was checked only by task id/version/track, harness, scoring, and weights. If a second selected run has the same task versions but a different config.corpusManifest.digestSha256, the leaderboard advertises the anchor digest and selectedTasks while mixing results from another corpus; include the manifest digest in compatibility or omit/flag the manifest when selected runs disagree.

Useful? React with 👍 / 👎.

@ElbertePlinio ElbertePlinio Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already covered: _RunCompatibilitySignature includes corpusManifestDigest, and both isCompatibleWith and == require corpusManifestDigest == other.corpusManifestDigest, so selected runs with different corpus digests are not aggregated. The advertised anchor digest therefore always matches every combined run.

final taskKeys = selected.taskRuns.map(_taskKey).toSet();
final modelKeys = selected.taskRuns
.map((taskRun) => '${taskRun.providerId}:${taskRun.modelId}')
Expand Down Expand Up @@ -132,7 +133,7 @@ Future<Map<String, Object?>> buildLeaderboardExport(
);

return <String, Object?>{
'schemaVersion': 1,
'schemaVersion': 2,

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep leaderboard schema compatible with the web parser

Publishing a freshly exported leaderboard now writes schemaVersion: 2, but the Svelte loader still rejects anything where value.schemaVersion !== 1 in web/src/lib/data/leaderboard.ts. The publish workflow writes this JSON to web/static/data/leaderboard.v1.json, so the public site will fall back to malformed/empty leaderboard data until the web parser is updated or the exporter keeps emitting version 1.

Useful? React with 👍 / 👎.

@ElbertePlinio ElbertePlinio Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already handled in this diff: the Svelte loader at web/src/lib/data/leaderboard.ts:302 now reads if (schemaVersion == null || schemaVersion < 1) return null; — it accepts >= 1, so a v2 leaderboard parses. (Separately fixing the hard-coded returned schemaVersion: 1 per the P3 note below.)

'generatedAt': generatedAt.toIso8601String(),
'benchmark': <String, Object?>{
'name': 'PickArena',
Expand All @@ -143,6 +144,11 @@ Future<Map<String, Object?>> buildLeaderboardExport(
'evaluatorSchemaVersion': dartArenaEvaluatorSchemaVersion,
'track': options.track,
'dataPolicy': options.strategy.kebabName,
if (corpusManifest != null) ...{
'preset': corpusManifest['preset'],
'selectedTasks': corpusManifest['tasks'],
'corpusManifestDigestSha256': corpusManifest['digestSha256'],
},
},
'source': <String, Object?>{
'anchorRunId': selected.anchorRunId,
Expand All @@ -167,6 +173,21 @@ Future<Map<String, Object?>> buildLeaderboardExport(
};
}

Map<String, Object?>? _corpusManifestForRun(Run? run) {
if (run == null) return null;
final provenance = _decodeRunProvenance(run.provenanceJson);
if (provenance == null) return null;
final manifest = _objectMap(
_objectMap(provenance['config'])['corpusManifest'],
);
if (manifest['preset'] is! String ||
manifest['tasks'] is! List ||
manifest['digestSha256'] is! String) {
return null;
}
return manifest;
}

_SelectedTaskRuns _selectTaskRuns({
required LeaderboardExportOptions options,
required List<Run> completedRuns,
Expand All @@ -188,6 +209,7 @@ _SelectedTaskRuns _selectTaskRuns({
final anchorSignature = _RunCompatibilitySignature.fromTaskRuns(
taskRunsByRunId[anchorRun.id] ?? const <TaskRun>[],
scoringSchemaVersion: anchorWeights.scoringSchemaVersion,
corpusManifestDigest: _corpusManifestDigestForRun(anchorRun),
harnessKinds: _harnessKinds(
anchorRun,
taskRunsByRunId[anchorRun.id] ?? const <TaskRun>[],
Expand All @@ -202,6 +224,7 @@ _SelectedTaskRuns _selectTaskRuns({
final candidateSignature = _RunCompatibilitySignature.fromTaskRuns(
entry.value,
scoringSchemaVersion: candidateWeights.scoringSchemaVersion,
corpusManifestDigest: _corpusManifestDigestForRun(candidateRun),
harnessKinds: _harnessKinds(candidateRun, entry.value),
);
if (!anchorSignature.isCompatibleWith(candidateSignature)) continue;
Expand Down Expand Up @@ -1260,11 +1283,13 @@ class _RunCompatibilitySignature {
required this.harnessIds,
required this.harnessKinds,
required this.scoringSchemaVersion,
required this.corpusManifestDigest,
});

factory _RunCompatibilitySignature.fromTaskRuns(
List<TaskRun> taskRuns, {
required int? scoringSchemaVersion,
required String? corpusManifestDigest,
required Map<String, String> harnessKinds,
}) {
return _RunCompatibilitySignature(
Expand All @@ -1283,18 +1308,21 @@ class _RunCompatibilitySignature {
.toList()
..sort()),
scoringSchemaVersion: scoringSchemaVersion,
corpusManifestDigest: corpusManifestDigest,
);
}

final List<String> taskKeys;
final List<String> harnessIds;
final List<String> harnessKinds;
final int? scoringSchemaVersion;
final String? corpusManifestDigest;

bool isCompatibleWith(_RunCompatibilitySignature other) {
return _listEquals(taskKeys, other.taskKeys) &&
_listEquals(harnessIds, other.harnessIds) &&
_listEquals(harnessKinds, other.harnessKinds) &&
corpusManifestDigest == other.corpusManifestDigest &&
(scoringSchemaVersion == null ||
other.scoringSchemaVersion == null ||
scoringSchemaVersion == other.scoringSchemaVersion);
Expand All @@ -1306,6 +1334,7 @@ class _RunCompatibilitySignature {
_listEquals(taskKeys, other.taskKeys) &&
_listEquals(harnessIds, other.harnessIds) &&
_listEquals(harnessKinds, other.harnessKinds) &&
corpusManifestDigest == other.corpusManifestDigest &&
scoringSchemaVersion == other.scoringSchemaVersion;
}

Expand All @@ -1315,9 +1344,15 @@ class _RunCompatibilitySignature {
Object.hashAll(harnessIds),
Object.hashAll(harnessKinds),
scoringSchemaVersion,
corpusManifestDigest,
);
}

String? _corpusManifestDigestForRun(Run run) {
final digest = _corpusManifestForRun(run)?['digestSha256'];
return digest is String && digest.isNotEmpty ? digest : null;
}

Map<String, String> _harnessKinds(Run run, List<TaskRun> taskRuns) {
final provenanceJson = run.provenanceJson;
if (provenanceJson == null || provenanceJson.trim().isEmpty) return const {};
Expand Down
54 changes: 50 additions & 4 deletions app/lib/export/release_report.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ const _requiredArtifactBundleChecksumPaths = [
..._standardBundleFilePaths,
];

const _artifactBundleManifestSchemaVersion = 1;
const _artifactBundleManifestSchemaVersions = {1, 2};
const _artifactBundleChecksumsSchemaVersion = 1;
const _artifactRunResultsSchemaVersion = 1;
const _taskQaSummarySchemaVersion = 1;
Expand Down Expand Up @@ -202,6 +202,23 @@ Map<String, Object?> buildReleaseReport({
if (taskSetId == null) {
blockers.add('Leaderboard benchmark task set id metadata is missing.');
}
final preset = _nonEmptyString(benchmark['preset']);
final corpusManifestDigest = _nonEmptyString(
benchmark['corpusManifestDigestSha256'],
);
final selectedTasks = benchmark['selectedTasks'];
final digestIsSha256 =
corpusManifestDigest != null &&
RegExp(r'^[0-9a-f]{64}$').hasMatch(corpusManifestDigest);
if (preset == null ||
!digestIsSha256 ||
selectedTasks is! List ||
selectedTasks.isEmpty) {
Comment on lines +213 to +216

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate the frozen corpus manifest before releasing

For a release-report input with a malformed or hand-edited leaderboard, any non-empty selectedTasks list and any non-empty corpusManifestDigestSha256 string passes this new gate, even if the digest is not a SHA-256 or the selected task ids/versions do not match the leaderboard task rows. Since this report is the release gate that certifies the frozen corpus, recompute/validate the manifest entries against leaderboard.tasks and require a 64-hex digest before allowing the report to become ready.

Useful? React with 👍 / 👎.

@ElbertePlinio ElbertePlinio Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a 64-hex sha256 format guard on the digest here so a non-sha256 no longer passes. The deeper cross-validation (recompute manifest entries against leaderboard.tasks) belongs to the fresh-execution release gate in #6 (verify current hidden-test digests + admission cleanliness + sandbox provenance); tracked there rather than expanding this PR.

Comment on lines +213 to +216

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate corpus manifest entries before release

When a leaderboard file has benchmark.selectedTasks as any non-empty list (for example ['task.a'], or maps missing taskVersion/taskBundleDigest), this condition clears the new frozen-corpus blocker as long as the top-level digest is 64 hex. In that malformed-input path the release report can still become ready while it lacks the exact task/version/bundle snapshot this gate is meant to require, because the rest of the QA checks compare loaded QA reports to leaderboard.tasks rather than to selectedTasks. Please validate each selected task entry (and ideally the digest over those entries) before treating the corpus manifest as present.

Useful? React with 👍 / 👎.

@ElbertePlinio ElbertePlinio Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and deferred to #6 (the fresh-execution release gate) rather than expanding this PR. This PR's gate now requires preset + a 64-hex top-level digest + non-empty selectedTasks; the deeper per-entry validation (each selected taskId/taskVersion/taskBundleDigest matches leaderboard.tasks, plus a digest recomputed over those entries) is exactly #6's scope. In practice this only bites a hand-edited leaderboard — our own exporter emits the correct snapshot. Tracked in #6.

blockers.add(
'Leaderboard is missing the frozen corpus manifest; run with --preset to '
'snapshot the corpus before an official release.',
);
}
if (evaluatorSchemaVersion <= 0) {
blockers.add(
'Leaderboard benchmark evaluator schema version metadata is missing.',
Expand Down Expand Up @@ -344,7 +361,7 @@ Map<String, Object?> buildReleaseReport({
);

return {
'schemaVersion': 1,
'schemaVersion': 2,
'releaseId': options.releaseId,
'generatedAt': generatedAt.toIso8601String(),
'status': blockers.isEmpty ? 'ready' : 'blocked',
Expand All @@ -362,6 +379,9 @@ Map<String, Object?> buildReleaseReport({
'evaluatorSchemaVersion': evaluatorSchemaVersion,
'track': benchmark['track'],
'dataPolicy': benchmark['dataPolicy'],
'preset': benchmark['preset'],
'selectedTasks': benchmark['selectedTasks'],
'corpusManifestDigestSha256': benchmark['corpusManifestDigestSha256'],
Comment on lines +382 to +384

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Block releases missing the corpus manifest

When a leaderboard was not produced from --preset mvp (or its provenance is missing/corrupt), benchmark['preset'], selectedTasks, and corpusManifestDigestSha256 are copied here as null but no blocker is added before status is computed, so an official release can still be marked ready without the frozen corpus snapshot this schema is supposed to guarantee. Validate these fields and their task-set match before allowing a ready report.

Useful? React with 👍 / 👎.

},
'source': {
'anchorRunId': source['anchorRunId'],
Expand Down Expand Up @@ -1749,7 +1769,7 @@ Map<String, Object?> _artifactBundleSummary({

if (schemaVersion <= 0) {
blockers.add('Run artifact bundle manifest schema version is missing.');
} else if (schemaVersion != _artifactBundleManifestSchemaVersion) {
} else if (!_artifactBundleManifestSchemaVersions.contains(schemaVersion)) {
blockers.add(
'Run artifact bundle manifest schema version $schemaVersion is unsupported.',
);
Expand Down Expand Up @@ -2016,7 +2036,7 @@ Map<String, Object?> _artifactBundleSummary({

final complete =
schemaVersion > 0 &&
schemaVersion == _artifactBundleManifestSchemaVersion &&
_artifactBundleManifestSchemaVersions.contains(schemaVersion) &&
manifestRunId != null &&
runIdInLeaderboardSource &&
manifestTaskRunCount > 0 &&
Expand Down Expand Up @@ -6344,11 +6364,37 @@ Map<String, Object?> _taskQaSummary(
'track': entry['track'],
'status': entry['status'],
'failureCount': _intValue(entry['failureCount']),
..._f2pP2pForTaskQaReport(_taskQaReportForEntry(reports, entry)),
},
],
};
}

Map<String, Object?>? _taskQaReportForEntry(
List<Map<String, Object?>> reports,
Map<String, Object?> entry,
) {
final key = _taskQaReportKey(entry);
if (key == null) return null;
for (final report in reports) {
if (_taskQaReportKey(report) == key) return report;
}
return null;
}

Map<String, Object?> _f2pP2pForTaskQaReport(Map<String, Object?>? report) {
final checks = _objectMap(report?['checks']);
final f2p = checks['baselineHiddenFailed'];
final publicPassed = checks['referencePublicPassed'];
final hiddenPassed = checks['referenceHiddenPassed'];
return {
'f2p': f2p is bool ? f2p : null,
'p2p': publicPassed is bool && hiddenPassed is bool
? publicPassed && hiddenPassed
: null,
};
}

Map<String, Object?> _taskQaSummaryIntegrityAudit({
required Map<String, Object?> summary,
required Object? rawReports,
Expand Down
2 changes: 1 addition & 1 deletion app/lib/export/run_manifest.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ class RunManifestV1 {
final Object? provenance;

Map<String, Object?> toJson() => {
'schemaVersion': 1,
'schemaVersion': 2,

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep exported manifest schema accepted by release reports

Newly exported bundles now write manifest.json with schemaVersion 2, but buildReleaseReport still validates artifact manifests against _artifactBundleManifestSchemaVersion = 1 and adds an unsupported-schema blocker otherwise. Any release report built from a freshly exported bundle will therefore be marked blocked even though it came from the current exporter; update the release-report validator/schema constant in the same change or keep this at 1 until it is supported.

Useful? React with 👍 / 👎.

@ElbertePlinio ElbertePlinio Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already handled in this same diff: release_report.dart:17 now defines _artifactBundleManifestSchemaVersions = {1, 2} (a set), and the validator at :1755/:2022 accepts any member — a freshly exported v2 bundle is not blocked. The old scalar _artifactBundleManifestSchemaVersion = 1 no longer exists.

'generatedAt': generatedAt,
'run': run,
'appVersion': appVersion,
Expand Down
6 changes: 6 additions & 0 deletions app/lib/headless/headless_benchmark_runner.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ class HeadlessBenchmarkConfig {
required this.modelsByProvider,
this.agentHarnesses = const [],
this.agentHarnessProvenance = const {},
this.preset,
this.corpusManifest,
required this.evaluatorConfig,
required this.evaluatorWeights,
required this.workdirManager,
Expand Down Expand Up @@ -60,6 +62,8 @@ class HeadlessBenchmarkConfig {
final Map<String, List<String>> modelsByProvider;
final List<AgentHarness> agentHarnesses;
final Map<String, Map<String, Object?>> agentHarnessProvenance;
final String? preset;
final Map<String, Object?>? corpusManifest;
final EvaluatorConfig evaluatorConfig;
final Map<String, double> evaluatorWeights;
final WorkdirManager workdirManager;
Expand Down Expand Up @@ -223,6 +227,8 @@ class HeadlessBenchmarkRunner {
generatedCodeSandboxEnforced: config.generatedCodeSandboxEnforced,
generatedCodeSandboxBackend: config.generatedCodeSandboxBackend,
agentHarnessProvenance: config.agentHarnessProvenance,
preset: config.preset,
corpusManifest: config.corpusManifest,
);
final provenanceJson = await buildRunProvenanceJson(
runId: config.runId,
Expand Down
Loading
Loading