CreatorBench v1.4: adaptive routing, sealed evidence, and review UI

[HomenShum]

Outcome

CreatorBench v1.4 is the first frozen NodeVideo benchmark release to meet the declared corpus scale and workflow-coverage targets while preserving creator/source-disjoint splits, sealed private evaluation, public proof isolation, and honest claim generation.

This PR deliberately does not claim universal editing performance. It measures what the current orchestration layer can defend: rights and policy checks, route eligibility, review/abstention behavior, canonical workflow execution evidence, deterministic export/reopen integrity, and the remaining need for blinded human editing-quality review.

What changed

• Universal creator-request, EditPlan, executor-capability, RouteReceipt, and outcome contracts.
• Adaptive executor router constrained by budget, privacy, egress, rights, policy, latency, and requested workflow.
• 8 workflow families across 16 domains, including:
  • talking-head cleanup;
  • golden-quote variants;
  • permitted reference/template editing;
  • dance choreography;
  • subject-aware reframe;
  • captioned multi-format repurposing;
  • founder/product launch;
  • tracking overlays.
• Canonical deterministic workflow execution evidence for talking-head cleanup and multi-format center-crop/export pilots.
• Rights ledger, creator/source-disjoint split generation, duplicate detection, freeze receipts, sealed evaluation, leakage checks, and measured-claim generation.
• Freeze-time validation now rejects benchmark-version drift and instance-to-source referential drift.
• Durable Convex review lifecycle:
  • pseudonymous reviewer identity;
  • blinded assignment claims;
  • duplicate-case prevention;
  • explicit opt-in;
  • verified writes, history, export, and deletion;
  • fail-closed behavior when the backend is unavailable.
• Public CreatorBench UI with:
  • coverage, subgroup weaknesses, route evidence, freeze receipt, and blind Review Lab;
  • prominent zero-human-review limitation;
  • JSON/CSV downloads;
  • responsive desktop, tablet, and mobile layouts;
  • light/dark themes and a Cursor-like NodeAgent evidence sidebar on desktop.
• Public claim generation now refuses automatic/assisted usability and silent-failure claims until human review coverage is complete.
• Build receipt binds benchmark version, freeze ID, source commit, and public-report SHA.

Frozen v1.4 corpus

• 250 rights-governed clips
• 205 creator/source collections
• 16 domains
• 8 workflow families
• 8 benchmark scenarios
• 6,128 workflow instances
• split sources:
  • 125 development
  • 50 public test
  • 50 private held-out
  • 25 adversarial
• split instances:
  • 2,576 development
  • 1,472 public test
  • 1,392 private held-out
  • 688 adversarial

Speech metadata covers 92/134 eligible sources with 3,422 transcribed words, 717 quote segments, and 286 detected silence regions.

Freeze and sealed evidence

• benchmark version: creatorbench-v1.4
• freeze ID: creatorbench-freeze:f3fb72fb06287374e5206fee
• frozen source commit: 95850b605cab74946a5cb422fef94ff6858fe5fd
• config hash: sha256:f2b9b95d371ab1b27605842dc473b59b1df92116fd9f050ad80a73dbf3b53d3e
• benchmark manifest hash: sha256:f9c4a4e4b5e3b2447914066686125a0892c2a73da18b44edc2a31f4aec07f00f

Sealed private routing

• 1,392 private held-out instances
• 50 creator-disjoint private sources
• 14 represented private domains
• 6 represented private workflows
• human review required: 688/1,392 (49.4%; Wilson 46.8%–52.0%)
• safely abstained: 704/1,392 (50.6%; Wilson 48.0%–53.2%)
• automatic/assisted usability: not measured
• editing-quality silent-failure incidence: not measured
• completed human reviews: 0

Canonical public claim:

On creatorbench-v1.4, covering 1392 private held-out workflow instances from 50 creator-disjoint sources across 6 workflows, NodeVideo routed 49.4% to human review and safely abstained in 50.6%. No automatic or assisted usability claim is made because only 0/1392 results have human review. Editing-quality silent-failure incidence remains unverified.

Deterministic workflow evidence

Talking-head public pilot:

• 71/71 rendered and reopened
• 0 word truncations
• mean speech retention 1.0
• all 71 remain review-required
• 21/21 eligible sealed private cases rendered and reopened
• 16 private cases skipped because no official SRT was available

Public multi-format center-crop pilot:

• 99 sources
• 297 outputs across 16:9, 9:16, and 1:1
• 297/297 decode, geometry, export, and reopen checks passed
• local execution cost: $0

These are technical workflow checks, not human editing-quality claims.

Verification

• npm run check: pass
• unit: 67 files / 278 tests pass
• CreatorBench proof suite: 16/16 pass
• public/private leakage verification: pass
  • private locator leakage: 0
  • hidden target leakage: 0
• CreatorBench Playwright + Axe, local: 40/40 across five viewport projects
• CreatorBench Playwright + Axe, production: 40/40 across five viewport projects
• production pixel proof: six desktop/tablet/mobile × light/dark captures inspected
• UTF-8/mojibake: pass
• console errors: 0
• page-level horizontal overflow: 0
• Vercel build receipt and downloaded public-report SHA: exact match
• GitHub checks: quality and node-platform conformance pass

Production proof

• public surface: https://nodevideo-pi.vercel.app/creatorbench
• deployed source commit: 804ec170d9ad44c928a628e5c46518bc38e7682e
• production deployment: https://nodevideo-bgeppi8pp-hshum2018-gmailcoms-projects.vercel.app
• Convex deployment: spotted-cat-868
• public report SHA-256: d3d69511e45ba9197b230616d9d820f774d8caab7a9b96fdbd6b64793735ae0a

The frozen source commit fixes the benchmark population and evaluator boundary. Later commits publish generated evidence, bind the UI tests to v1.4, and stamp the deployment without changing the frozen private population.

Remaining hard gate

The benchmark infrastructure, corpus, workflow coverage, sealed execution, UI, and production proof are complete. The release still needs real blinded human judgments and correction-time measurements before it can report:

• automatic usable rate;
• assisted usable rate;
• editing-quality silent-failure incidence;
• median/p95 creator correction time;
• inter-rater agreement;
• subgroup usability thresholds.

No synthetic or model-generated labels are substituted for that human work.

Claim boundary

Defensible now:

NodeVideo has a governed universal creator-request and adaptive-executor contract measured on frozen CreatorBench v1.4 across 250 clips, 205 creator/source collections, 16 domains, 8 workflows, and 6,128 instances, with sealed routing, deterministic workflow evidence, public proof isolation, and explicit review/abstention behavior.

Not defensible yet:

NodeVideo performs universally on arbitrary unseen creator footage.

That stronger statement requires completing the blinded human-review phase and passing prespecified workflow and subgroup thresholds.

[HomenShum]

Final preflight preview (deployment-protected):

• CreatorBench UI: https://nodevideo-e8iux4wds-hshum2018-gmailcoms-projects.vercel.app/creatorbench
• Machine claim: https://nodevideo-e8iux4wds-hshum2018-gmailcoms-projects.vercel.app/benchmarks/creatorbench-v1/results/public-claim.json
• deployed source commit: 477c1775cd80e7b1fd52dd5ca3be4c3f18a74165

Signed-in browser verification: real report rendered, no console errors, UTF-8, no horizontal overflow, null correction/cost metrics fail closed as “Not reported.” All CI gates are green.

The hard benchmark-size gate intentionally remains red (5/250 clips, 5/75 creators, 4/15 domains, 40/2,000 instances); no blinded human-reviewed renders exist, so usable output remains 0 and no arbitrary-footage performance claim is made.