Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema by Copilot · Pull Request #2496 · Hack23/riksdagsmonitor

Copilot · 2026-05-14T23:24:19Z

This issue tightens analysis templates so recurring structural gaps cannot slip through (partial Family-E discipline, incomplete Devil’s-Advocate KJ coverage, missing posterior confidence rows, inconsistent re-run metadata, and missing legal/implementation tracking).
This PR updates templates and gate enforcement so these requirements are explicit, auditable, and blocking when absent.

Methodology reflection contract hardened (9 required sections)
- Expanded analysis/templates/methodology-reflection.md with a gate-enforced required-section contract covering:
  - ICD 203 full checklist
  - Devil’s-Advocate KJ coverage matrix (100% coverage target)
  - confidence distribution with explicit posterior
  - Lagrådet/Statskontoret/SKR tracking
  - sibling-folder ingestion record
  - unified re-run log schema
  - banned-phrase zero-count grid
  - Pass 1 → Pass 2 delta table
  - improvement opportunities linked to PIR roll-forward
Related template alignment
- analysis/templates/devils-advocate.md: added required KJ-coverage matrix preamble mapping all KJs from intelligence assessment.
- analysis/templates/intelligence-assessment.md: added confidence distribution table with explicit Posterior column.
- Added reusable fragment analysis/templates/lagradet-tracking.md for Lagrådet/Statskontoret/SKR tracking fields.
Gate enforcement + prompt contract updates
- scripts/agentic/analysis-gate.ts: replaced loose methodology-reflection checks with required-section presence validation and unified re-run schema column checks.
- .github/prompts/05-analysis-gate.md: updated structural gate spec and inline gate logic to match the new required sections.
- .github/prompts/04-analysis-pipeline.md: updated artifact catalogue expectations for methodology-reflection.md.
Test coverage
- tests/agentic-analysis-gate.test.ts: added/updated Family C checks to require all nine methodology-reflection sections and fail when any required block is missing.

// analysis-gate.ts (methodology-reflection contract)
const requiredSectionChecks = [
  { label: 'ICD 203 audit checklist', present: /ICD\\s+203/i.test(content) },
  { label: "Devil's-Advocate KJ coverage matrix", present: /Devil'?s[-\\s]Advocate.*(KJ|Key\\s+Judgment).*Coverage/i.test(content) },
  { label: 'Confidence distribution with posterior', present: /Confidence\\s+Distribution/i.test(content) && /Posterior/i.test(content) },
  // ...remaining required sections...
];

github-actions · 2026-05-14T23:25:16Z

🏷️ Automatic Labeling Summary

This PR has been automatically labeled based on the files changed and PR metadata.

Applied Labels: size-xs

Label Categories

🗳️ Content: news, dashboard, visualization, intelligence
💻 Technology: html-css, javascript, workflow, security
📊 Data: cia-data, riksdag-data, data-pipeline, schema
🌍 I18n: i18n, translation, rtl
🔒 ISMS: isms, iso-27001, nist-csf, cis-controls
🏗️ Infrastructure: ci-cd, deployment, performance, monitoring
🔄 Quality: testing, accessibility, documentation, refactor
🤖 AI: agent, skill, agentic-workflow

For more information, see .github/labeler.yml.

github-actions · 2026-05-14T23:26:07Z

🔍 Lighthouse Performance Audit

Category	Score	Status
Performance	85/100	🟡
Accessibility	95/100	🟢
Best Practices	90/100	🟢
SEO	95/100	🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/490ae0e9-9598-4271-aabd-69926c634300 Co-authored-by: pethers <[email protected]>

github-actions · 2026-05-15T00:18:57Z

🔍 Lighthouse Performance Audit

Category	Score	Status
Performance	85/100	🟡
Accessibility	95/100	🟢
Best Practices	90/100	🟢
SEO	95/100	🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

github-actions · 2026-05-15T06:14:30Z

🔍 Lighthouse Performance Audit

Category	Score	Status
Performance	85/100	🟡
Accessibility	95/100	🟢
Best Practices	90/100	🟢
SEO	95/100	🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Copilot

Pull request overview

This PR hardens the agentic analysis gate and templates around methodology-reflection.md, making previously informal run-audit requirements explicit across templates, prompt documentation, gate logic, and tests.

Changes:

Expands methodology-reflection requirements to nine required sections.
Adds related template guidance for KJ coverage, posterior confidence, and Lagrådet/Statskontoret/SKR tracking.
Updates TypeScript and prompt-level gate checks plus unit tests for the new methodology-reflection contract.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`tests/agentic-analysis-gate.test.ts`	Adds shared valid methodology-reflection fixture and updates Family C tests.
`scripts/agentic/analysis-gate.ts`	Replaces loose methodology-reflection check with nine required-section predicates.
`analysis/templates/methodology-reflection.md`	Adds gate-enforced required-section contract and supporting tables.
`analysis/templates/lagradet-tracking.md`	Adds reusable tracking fragment for legal/implementation signals.
`analysis/templates/intelligence-assessment.md`	Adds posterior confidence distribution table guidance.
`analysis/templates/devils-advocate.md`	Adds required KJ coverage matrix guidance.
`.github/prompts/05-analysis-gate.md`	Updates analysis gate prompt and inline shell checks for the new contract.
`.github/prompts/04-analysis-pipeline.md`	Updates artifact catalogue expectations for methodology-reflection.


+## 🎯 Key Judgment Coverage Matrix (Required)
+
+> **[REQUIRED]** Before writing hypotheses, list **every** KJ from `intelligence-assessment.md` and map it to at least one challenge row. Gate expectation: **100% KJ coverage**.


+  const requiredSectionChecks: Array<{ label: string; present: boolean }> = [
+    { label: 'ICD 203 audit checklist', present: /ICD\s+203/i.test(content) },
+    {
+      label: "Devil's-Advocate KJ coverage matrix",
+      present: /Devil'?s[-\s]Advocate.*(KJ|Key\s+Judgment).*Coverage/i.test(content),
+    },
+    {
+      label: 'Confidence distribution with posterior',
+      present: /Confidence\s+Distribution/i.test(content) && /Posterior/i.test(content),
+    },
+    {
+      label: 'Lagrådet / Statskontoret / SKR tracking',
+      present: lower.includes('lagrådet') &&
+        lower.includes('statskontoret') &&
+        (lower.includes('skr') || lower.includes('sveriges kommuner')),
+    },
+    {
+      label: 'Sibling-folder ingestion record',
+      present: /Sibling[-\s]?Folder\s+Ingestion/i.test(content),
+    },
+    {
+      label: 'Re-run log unified schema',
+      present: hasReRunColumns,
+    },
+    {
+      label: 'Banned-phrase audit grid',
+      present: /Banned[-\s]?Phrase\s+Audit/i.test(content),
+    },
+    {
+      label: 'Pass 1 to Pass 2 delta table',
+      present: lower.includes('pass 1') && lower.includes('pass 2') && lower.includes('delta'),
+    },
+    {
+      label: 'Improvement opportunities linked to PIR roll-forward',
+      present: lower.includes('improvement opportunities') &&
+        lower.includes('pir') &&
+        lower.includes('roll-forward'),
+    },
+  ];


+  grep -qiE 'Banned[-[:space:]]?Phrase[[:space:]]+Audit' "$REF" \
+    || { echo "❌ methodology-reflection.md: missing banned-phrase audit grid"; FAIL=1; }
+  grep -qiE 'Pass[[:space:]]*1.*Pass[[:space:]]*2.*Delta' "$REF" \
+    || { echo "❌ methodology-reflection.md: missing Pass 1→Pass 2 delta table"; FAIL=1; }
+  grep -qiE 'Improvement[[:space:]]+Opportunities.*PIR[[:space:]]+Roll[-[:space:]]?Forward' "$REF" \
+    || { echo "❌ methodology-reflection.md: missing improvement opportunities to PIR roll-forward section"; FAIL=1; }


+
+---
+
+## 🎯 Devil's-Advocate Key Judgment Coverage Matrix


+| Track | Applicability | Status | Latest evidence (URL + date) | Next check |
+|-------|:-------------:|--------|-------------------------------|------------|
+| Lagrådet yttrande | Y/N | `not applicable / referral pending / yttrande published` | `[REQUIRED]` | `[REQUIRED]` |
+| Statskontoret implementation signal | Y/N | `found / none found / not triggered` | `[REQUIRED]` | `[REQUIRED]` |
+| SKR operational impact signal | Y/N | `found / none found / not triggered` | `[REQUIRED]` | `[REQUIRED]` |


+  const requiredSectionChecks: Array<{ label: string; present: boolean }> = [
+    { label: 'ICD 203 audit checklist', present: /ICD\s+203/i.test(content) },
+    {
+      label: "Devil's-Advocate KJ coverage matrix",
+      present: /Devil'?s[-\s]Advocate.*(KJ|Key\s+Judgment).*Coverage/i.test(content),


+    {
+      label: 'Confidence distribution with posterior',
+      present: /Confidence\s+Distribution/i.test(content) && /Posterior/i.test(content),
+    },


+  grep -qiE "Devil'?s[-[:space:]]Advocate.*(KJ|Key[[:space:]]+Judgment).*Coverage" "$REF" \
+    || { echo "❌ methodology-reflection.md: missing Devil's-Advocate KJ-coverage matrix"; FAIL=1; }
+  grep -qiE 'Confidence[[:space:]]+Distribution' "$REF" || { echo "❌ methodology-reflection.md: missing confidence distribution section"; FAIL=1; }
+  grep -qiE 'Posterior' "$REF" || { echo "❌ methodology-reflection.md: missing Posterior column/marker"; FAIL=1; }


Initial plan

f524c95

Copilot AI assigned Copilot and pethers May 14, 2026

github-actions Bot added the size-xs Extra small change (< 10 lines) label May 14, 2026

Copilot started work on behalf of pethers May 14, 2026 23:52 View session

Copilot AI linked an issue May 14, 2026 that may be closed by this pull request

[Analysis Templates] Enforce 100% KJ coverage, complete Family-E per-document, and unify re-run / confidence templates #2493

Open

8 tasks

Enforce methodology reflection section contract and gate checks

a933a38

Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/490ae0e9-9598-4271-aabd-69926c634300 Co-authored-by: pethers <[email protected]>

Copilot AI changed the title ~~[WIP] Enforce 100% KJ coverage and unify templates~~ Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema May 15, 2026

github-actions Bot added documentation Documentation updates testing Test coverage refactor Code refactoring size-l Large change (250-1000 lines) labels May 15, 2026

Copilot finished work on behalf of pethers May 15, 2026 00:18

Copilot AI requested a review from pethers May 15, 2026 00:18

Merge branch 'main' into copilot/improve-analysis-templates-coverage

60d9b0b

pethers marked this pull request as ready for review May 15, 2026 06:12

Copilot AI review requested due to automatic review settings May 15, 2026 06:12

Copilot started reviewing on behalf of pethers May 15, 2026 06:13 View session

Copilot AI reviewed May 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema#2496

Enforce methodology-reflection contract: 9 required sections + gate/test validation for KJ coverage, posterior confidence, and re-run schema#2496
Copilot wants to merge 3 commits into
mainfrom
copilot/improve-analysis-templates-coverage

Copilot AI commented May 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		## 🎯 Key Judgment Coverage Matrix (Required)

		> [REQUIRED] Before writing hypotheses, list every KJ from `intelligence-assessment.md` and map it to at least one challenge row. Gate expectation: 100% KJ coverage.

Conversation

Copilot AI commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 14, 2026

🏷️ Automatic Labeling Summary

Label Categories

Uh oh!

github-actions Bot commented May 14, 2026

🔍 Lighthouse Performance Audit

Uh oh!

github-actions Bot commented May 15, 2026

🔍 Lighthouse Performance Audit

Uh oh!

github-actions Bot commented May 15, 2026

🔍 Lighthouse Performance Audit

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 14, 2026 •

edited

Loading