Skip to content

Preset flag + self-describing run manifest in release report/leaderboard export #7

Description

@ElbertePlinio

MVP-completion slice: stop selecting official tasks by hand and make every published run self-describing.

  • --preset flag in the headless CLI selecting active admitted tasks by metadata (single flag over task.yaml metadata — no preset registry; there is one real preset today)
  • Release report + leaderboard export name the preset, exact task ids, task versions, and a corpus manifest digest (pin published numbers to a frozen corpus snapshot, not a drifting HEAD)
  • Export raw trial rows and first-class f2p/p2p fields (baselineHiddenFailed / referencePublicPassed / referenceHiddenPassed) — fields only; no dashboards until the corpus is bigger
  • Record per run: harness version, prompt template, tool permissions, timeouts, retry policy, model/provider config

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions