audit: report narrative layer (headline/insights/recommendations) is ungrounded — no semantic validation against check evidence

[AlexanderMakarov]

Effort: ~2–3 days

Profit: high — the headline/insights/recommendations blocks are the part of the report the decision-maker actually reads, on every run of every real project. Errors here are the hardest of all to check: the prose sounds like derived logic, so only a reader who knows the repo well can catch it (a user already labeled one such passage a "well-written hallucination"). Grounding this layer protects the credibility of the whole product.

Problem

The report narrative (headline, insights[], recommendations[]) is authored as free-form LLM prose in SKILL.md Step 5 and applied via patch-report, and nothing enforces that its claims match the check evidence:

• patchReportBlocks() (audit_patch.ts:370-437) validates structure only (known keys, types). No semantic checks.
• The check_id cited by each recommendation is not verified against the dimension artifacts — the "verify the id" instruction in SKILL.md:190 is advisory.
• report-context deliberately contains only check verdicts + evidence strings (no repo content), so the authoring model cannot even validate a causal claim it is about to make.
• Grounding rules ("transcribe verbatim", "never invent numbers") cover headline numbers; insights[].theme/so_what/improves have no grounding requirement at all.

Real-world results from the Slack feedback thread:

• Insight title "Spec-driven development is documented but not practiced" — "documented" was invented; no such evidence exists.
• "so every verification loop depends on a human running Maven/npm by hand" — invented causal argumentation appended to a correct finding.
• "New engineers and agents have to read DECISIONS.md and source line-by-line to reconstruct contracts…" — called out by the user as "well-written hallucination. Basically making taste choice to sound like logical argument."

Fix

Layered, cheapest first:

1. Mechanical validation in patch-report: reject a recommendation whose check_id does not exist in the dimension artifacts or whose cited check is not FAIL/WARN; reject weak_areas entries naming checks whose status is SKIP.
2. Authoring instructions (SKILL.md Step 5): require every so_what/improves claim to be traceable to a quoted evidence string or check value from report-context; explicitly forbid invented causal clauses ("so …", "which means …") that go beyond the evidence; insight titles must not assert facts (e.g. "documented") absent from evidence.
3. Verification sub-step: after authoring and before patch-report, the orchestrator re-reads its blocks against the report-context output and drops/rewrites any sentence it cannot anchor to a specific check.

Definition of done

• patch-report fails (with a clear error listing the offending entries) on recommendations citing unknown or non-FAIL/WARN check_ids, and on insights listing SKIPped checks as weak areas; engine tests cover both rejections.
• SKILL.md Step 5 contains the grounding rules for insight prose (traceability requirement, no invented causal clauses, no unevidenced facts in titles) and the pre-patch verification sub-step.
• output-format.md "Authoring integrity" section updated to state the enforced (not advisory) contract.
• npm run build:audit-engine re-run, dist/ committed, plugin version bumped.
• QA harness run (tools/ai-readiness-audit/qa) on one known repo: authored blocks pass the new validation and contain no claims that cannot be pointed at a check evidence line.