Roadmap: reliable mobile benchmark v1.0 — post-audit master plan

[ElbertePlinio]

Master plan from the 2026-07-08 status audit (two code scouts + gpt-5.5/opus-4.8/glm-5.2 review panel). Full audit context lives in the session notes; this issue tracks execution order.

Where we are: runner/QA machinery is solid (hidden split, negative cases, repeated flake checks, sandboxed official runs, honest unknown-telemetry). But the published board is 1 model × 5 tasks from June 6, official grading still happens in the agent's own workspace, all 10 admission reports are stale (gitDirty=true), and V2 froze behind unanswered owner packets — now answered (defer; see #9).

Ordered phases:

• Phase 0 — QA integrity fixes: QA integrity fixes: prompt-safety, diff_size, reference audits, clean admissions (Phase 0) #5 (also closes Replace fixed-width breakpoints in ui.action_bar_overflow reference solution with measurement-based overflow #3 and QA/sandbox hardening: npm-installed Droid binds + sandboxed admission validation in tests #4)
• Housekeeping (parallel, not gating): Housekeeping: record V2 owner decisions (defer all three) + archive June audit docs #9
• Phase 1 — Preset + self-describing run manifest: Preset flag + self-describing run manifest in release report/leaderboard export #7 (merged PR feat: add preset selection and corpus manifest #29)
• Phase 2 — Clean-baseline patch replay grading + fresh-execution gates: Clean-baseline patch replay as the official grading path + fresh-execution release gates #6 (merged PR Grade via clean-baseline patch replay with fresh-execution release gates #31)
• Phase 3 — Official multi-model run v1.0 (shakedown → published): Official multi-model benchmark run v1.0 (harness parity, CIs, denominator policy) #8
• Simplification epic (owner-approved 2026-07-09): CLI + web only, desktop UI deleted, minimal uniform agent: Simplify to CLI + web (DeepSWE shape): delete desktop UI, pure-Dart engine, minimal uniform agent #12
• Web landing + results site: Web: landing + results site (leaderboard with CI + cost, methodology, run page) #13
• Phase 4 — Corpus growth toward 15–20 tasks; category slices only after that
• Later — cross-framework tracks (RN Android → native Android → iOS) and security category, sequenced after Flutter v1.0 is credible

v1.0 bar (all must hold before calling it done):

• Corpus pinned + clean (frozen manifest digest, clean admissions, gates reject dirty/mismatched evidence)
• Grading ≠ agent workspace (clean-replay path asserted by release gates) — done (Clean-baseline patch replay as the official grading path + fresh-execution release gates #6)
• Prompt-safety scans the real resolved prompt; no pass/fail diff-size gate
• Reference solutions satisfy their own contracts (all 10 audited)
• ≥5 materially different models × 10 tasks × ≥3 trials under one harness-parity contract
• Published with CIs, explicit n, raw trial rows, denominator policy, cost/latency, negative-case results

Corpus milestones (epic ladder):

• v1.0 — 10 tasks, first published multi-model board (rigor proven: clean-replay grading, honest CIs)
• v1.1 — 15–20 tasks (category slices unlock; second-wave: undoable cart, checkout result pop, localized empty state)
• v1.5 — 40–50 Flutter tasks (deep flagship track, agent-assisted authoring via task-multiplication workflow behind the admission gates)
• v2.0 — 100+ tasks across Flutter + React Native Android + native Android + iOS, security category, cross-framework parity families

Scale note: authoring scales via the task-multiplication workflow with admission gates as the quality firewall; at 100 tasks × 10 models × 3 trials (~3,000 sandboxed runs per release) trial concurrency and provider budgets become the binding constraint.