A code audit of the eval pipeline and ingestion paths (following the July architecture review with the founding engineer) surfaced a cluster of correctness bugs. Tracking them together since they ship as one branch.
Findings
- Push-to-main webhook diverges from every other ingestion path —
src/app/api/webhooks/github-push/route.ts used a raw single-document yaml.load and required an embedded id field, while the PR webhook and scheduled evals use the canonical parseAndNormalizeBlueprint with path-derived IDs. Consequences: multi-document blueprints that pass PR staging silently never run after merge; the deprecated embedded-id policy is contradicted; blueprints without models fail downstream; dedupe hashes never match the cron path (hashed unresolved vs resolved model lists), so the same blueprint could double-run.
- Schema validation is warn-only on submission paths —
validateBlueprintSchema (Ajv) results were logged and discarded, so structurally invalid blueprints proceeded into paid staging runs and failed late with no actionable feedback to the contributor.
- Dead judge-config knobs —
FORCE_EXPERIMENTAL = true in src/cli/evaluators/llm-coverage-evaluator.ts silently overrode the per-blueprint useExperimentalScale flag; judgeMode was threaded through but never read (scoring is always consensus/mean); custom judge panels silently disable the backup judge with no documentation.
- "Models Automatically Excluded" alert dead on artifact-format runs —
buildCoreData (src/lib/storageService.ts) deleted excludedModels from core.json claiming it would be "recalculated client-side", but core.json ships null placeholder response matrices, so the client scan (undefined/empty-string check in useComparisonData) could never fire.
- PR staging validates only the first 10 prompts —
applyPREvalLimits (src/lib/pr-eval-limiter.ts) truncated with slice(0, 10), so pre-merge review of a several-hundred-prompt blueprint (e.g. a multilingual suite organised in language blocks) only ever exercised the opening block.
- 404 links in experiments —
experiments/redlines/page.tsx and experiments/ndeltas/page.tsx linked to nonexistent top-level /redlines/{configId} and /ndeltas/{modelId} instead of their /experiments/... detail routes.
A code audit of the eval pipeline and ingestion paths (following the July architecture review with the founding engineer) surfaced a cluster of correctness bugs. Tracking them together since they ship as one branch.
Findings
src/app/api/webhooks/github-push/route.tsused a raw single-documentyaml.loadand required an embeddedidfield, while the PR webhook and scheduled evals use the canonicalparseAndNormalizeBlueprintwith path-derived IDs. Consequences: multi-document blueprints that pass PR staging silently never run after merge; the deprecated embedded-idpolicy is contradicted; blueprints withoutmodelsfail downstream; dedupe hashes never match the cron path (hashed unresolved vs resolved model lists), so the same blueprint could double-run.validateBlueprintSchema(Ajv) results were logged and discarded, so structurally invalid blueprints proceeded into paid staging runs and failed late with no actionable feedback to the contributor.FORCE_EXPERIMENTAL = trueinsrc/cli/evaluators/llm-coverage-evaluator.tssilently overrode the per-blueprintuseExperimentalScaleflag;judgeModewas threaded through but never read (scoring is always consensus/mean); custom judge panels silently disable the backup judge with no documentation.buildCoreData(src/lib/storageService.ts) deletedexcludedModelsfromcore.jsonclaiming it would be "recalculated client-side", but core.json ships null placeholder response matrices, so the client scan (undefined/empty-string check inuseComparisonData) could never fire.applyPREvalLimits(src/lib/pr-eval-limiter.ts) truncated withslice(0, 10), so pre-merge review of a several-hundred-prompt blueprint (e.g. a multilingual suite organised in language blocks) only ever exercised the opening block.experiments/redlines/page.tsxandexperiments/ndeltas/page.tsxlinked to nonexistent top-level/redlines/{configId}and/ndeltas/{modelId}instead of their/experiments/...detail routes.