Skip to content

Blueprint ingestion & eval correctness bugs surfaced by architecture audit #56

Description

@nojibe

A code audit of the eval pipeline and ingestion paths (following the July architecture review with the founding engineer) surfaced a cluster of correctness bugs. Tracking them together since they ship as one branch.

Findings

  1. Push-to-main webhook diverges from every other ingestion pathsrc/app/api/webhooks/github-push/route.ts used a raw single-document yaml.load and required an embedded id field, while the PR webhook and scheduled evals use the canonical parseAndNormalizeBlueprint with path-derived IDs. Consequences: multi-document blueprints that pass PR staging silently never run after merge; the deprecated embedded-id policy is contradicted; blueprints without models fail downstream; dedupe hashes never match the cron path (hashed unresolved vs resolved model lists), so the same blueprint could double-run.
  2. Schema validation is warn-only on submission pathsvalidateBlueprintSchema (Ajv) results were logged and discarded, so structurally invalid blueprints proceeded into paid staging runs and failed late with no actionable feedback to the contributor.
  3. Dead judge-config knobsFORCE_EXPERIMENTAL = true in src/cli/evaluators/llm-coverage-evaluator.ts silently overrode the per-blueprint useExperimentalScale flag; judgeMode was threaded through but never read (scoring is always consensus/mean); custom judge panels silently disable the backup judge with no documentation.
  4. "Models Automatically Excluded" alert dead on artifact-format runsbuildCoreData (src/lib/storageService.ts) deleted excludedModels from core.json claiming it would be "recalculated client-side", but core.json ships null placeholder response matrices, so the client scan (undefined/empty-string check in useComparisonData) could never fire.
  5. PR staging validates only the first 10 promptsapplyPREvalLimits (src/lib/pr-eval-limiter.ts) truncated with slice(0, 10), so pre-merge review of a several-hundred-prompt blueprint (e.g. a multilingual suite organised in language blocks) only ever exercised the opening block.
  6. 404 links in experimentsexperiments/redlines/page.tsx and experiments/ndeltas/page.tsx linked to nonexistent top-level /redlines/{configId} and /ndeltas/{modelId} instead of their /experiments/... detail routes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions