Skip to content

Harness provenance aggregation refinements (legacy DBs, direct-runner callers, custom-template args) #28

Description

@ElbertePlinio

Deferred from PR #26 (review cap). Three aggregation-correctness refinements to the harness-kind compatibility signature — none are crash/data-loss/security, and none affect the official v1.0 path (fresh DB, minimal-only parity runs):

  1. Legacy DB compat (leaderboard_exporter.dart ~:1330): agentic runs created before the agentHarnesses provenance field have no config.agentHarnesses; the empty harness-kind signature makes them incompatible with new minimal runs sharing the same harnessId. Treat missing provenance as 'minimal/legacy' or otherwise compatible with minimal.
  2. Direct HeadlessBenchmarkRunner callers (headless_benchmark_runner.dart ~:32): callers bypassing runHeadlessCli get empty agentHarnessProvenance, so two runs with the same provider/harness id but different harness impls could aggregate. Derive harness provenance in the runner from the actual harnesses.
  3. Custom-template args in provenance (command_template_agent_harness.dart ~:113): custom command-template provenance records only kind/agent/agentVersion, not the executable+argument template — two custom runs with the same name/version but different args (e.g. permission flags) would aggregate despite being different scaffolds. Include a template/args hash in provenance.

Also folds in #27 (scope compatibility to selected results). All are exporter/provenance refinements to do together in one focused pass. Refs #12 #10 #27.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions