Enhance triage skills with updated scoring (#6294)

* Update Generate-FeatureAreaReport.ps1 to only search for area-* feature areas

* Add Validate-FeatureAreaReport.ps1 to issue-triage-report skill

* Enhance triage skills with updated scoring, confidence, and P-scale labels

Scoring System:
- New weights: reactions=30%, age=30%, comments=30%, severity=10%
- P-scale severity labels: P0=critical, P1=high, P2=medium, P3=low
- Confidence scoring with grep-friendly [confidence:XX] format (0-100)

Label Consolidation:
- Merged Hot + Popular into Popular (>=5 reactions threshold)

Contact Schema:
- Simplified from {primary, secondary} to single {contact} field
- Removed legacy schema backward compatibility

Area Suggestions:
- Get-IssueDetails.ps1 dynamically fetches area labels via Get-RepositoryLabels.ps1
- area-Notifications covers all notification types (toast, badge, push, wns)
- Fixed area-PowerManagement naming

Documentation:
- Added PowerShell examples alongside Bash for confidence filtering
- Updated all SKILL.md files with new configuration details

* Revert trending criteria to include recency as a scoring factor

* Remove copy of ScoringConfig.json in favor of the filepath

* ReportLib.ps1: Remove defaults in Get-ScoringConfig and rely on ScoringConfig.json

* ReportLib.ps1: Remove defaults in Get-IssueScore and rely on ScoringConfig.json for severity labels

* ReportLib.ps1: Update Get-AreaContacts to rely on area-contacts.json, fails fast on errors

* GetHighlightScore.ps1: Use Get-Issue score for calculation. Let this script handle human-readable formatting

* ReportLib.ps1: Remove unused thresholds assignment from Get-IssueScore

* Skills: updates the label fetching code so Get-RepositoryLabels.ps1 is the source of truth

* Create area-keywords.json

* Update Get-IssueDetails.ps1 to reference area labels json

* Enforce trending_days recency check for Trending highlight label

- Add RawUpdateAgeDays to Get-IssueScore (computed from updatedAt)
- Require comments >= trending_comments AND updated within trending_days
  in both Get-HighlightLabels (ReportLib) and Get-DetailedIssueScore
- Fix scoring-algorithm.md: comments >= 5 -> >= 10, add trending_days
  to highlight table and thresholds reference
- Add missing trending_days to SKILL.md config example

* Update scoring-algorithm.md to walk through scripts and config files

* Remove sample json with hardcoded weight values

* Correct Validate-FeatureAreaReport.ps1 to use PSScriptRoot instead of System.Management.Automation.InvocationInfo.MyCommand.Path

* Remove security label from issue-triage-report skill

* Remove keyword classification on area labels and replace with agent instructions

* Clean up block word matching in ReportLib.ps1

* Convert script-based confidence scoring in issue-triage-report skill to LLM based confidence scoring (what we originally intended)

* Update Get-AreaContacts.ps1 and ReportLib.ps1 to use the contacts/notes schema

* Move severity and blocker determination to agent SKILL.md instead of PowerShell script

* Add agent assessments for severity and blockers

* Remove bucket scoring

* Scale weights to percentages and use double loading for point values

* Replace outdated severity labels with severity multipliers

* Update docs to reflect recent scoring changes

* Have human-edited IssueAssesments.json take scoring priority

* Fix indentation typo in SKILL.md for issue-triage-report

* Minor corrections in SKILL.md and scoring-algorithm.md

* Remove labelPriority

* Bringing back scores between 0 to 100

* Condense ReadAgentAssessments and ReadIssueAssessments into one ReadAssessments file

* Condense Get-AgentAssessmentsPath and Get-IssueAssessmentsPath into one Get-AssessmentsPath file

* Remove unused Test-HasLabelMatching

* Add Get-IssueScore docs

* Move Get-DetailedIssueScore into ReportLib.ps1

* Update Scoring Results section of scoring-algorithm.md

* Actually consolidate Read-IssueAssessments and Read-AgentAssessments into ReadAssessments in ReportLib.ps1

* Add instructions to persistent reasoning to files in SKILL.md

Author: lauren-ciha

.github/skills/issue-triage-report/SKILL.md

@@ -14,7 +14,7 @@ This skill generates comprehensive GitHub Feature Area Status reports that help
 
 1. **Engineering triage meetings** — Create status reports showing open issues, needs-triage counts, and highlighted issues per feature area.
 
-2. **Priority analysis requests** — Identify which issues should get engineering focus based on multi-factor scoring (reactions, age, severity, blockers).
+2. **Priority analysis requests** — Identify which issues should get engineering focus based on deterministic scoring (reactions, age, comments) plus agent content assessment.
 
 3. **Feature area health checks** — Assess the health of specific areas by analyzing issue distribution, triage backlog, and proposal counts.
 
@@ -46,25 +46,7 @@ The area contacts file maps feature areas to team members. This file is **requir
    cp .github/skills/issue-triage-report/references/area-contacts.json .user/issue-triage-report/area-contacts.json
    ```
 
-3. Edit `.user/issue-triage-report/area-contacts.json` with your team's actual contacts:
-   ```json
-   {
-     "areaContacts": {
-       "area-FeatureName": {
-         "primary": "Primary Contact Name",
-         "secondary": "Secondary Contact or null",
-         "notes": "Optional notes"
-       }
-     },
-     "specialAreas": {
-       "triageOnly": ["area-TriageOnlyExample"],
-       "crossFunctional": {
-         "area-CrossTeam": "org/related-repo"
-       }
-     },
-     "lastUpdated": "YYYY-MM"
-   }
-   ```
+3. Edit `.user/issue-triage-report/area-contacts.json` with your team's actual contacts
 
 > **Note**: The `.user/` folder is gitignored, so your team contacts remain private.
 
@@ -146,16 +128,42 @@ Retrieve or update the area-to-contact mapping configuration.
    ./scripts/Generate-FeatureAreaReport.ps1 -OutputFormat markdown -HighlightCount 3
    ```
 
-3. Review highlighted issues and their scores
+  Before running the command above, have the agent save runtime assessments to:
+  - `./references/AgentAssessments.json`
+
+   This file is loaded automatically at script start and applied as per-run overrides.
+   Every assessed issue entry must include agent reasoning in the `reasoning` field.
+
+   Required per-issue schema in `assessments`:
+   - `severityTier`: `critical|high|medium|low|none`
+   - `isBlocker`: `true|false`
+   - `reasoning`: short rationale explaining severity/blocker judgment
+
+   Example:
+   ```json
+   {
+      "assessments": {
+         "2894": {
+            "severityTier": "high",
+            "isBlocker": false,
+            "reasoning": "Frequent user impact in notifications flow; clear repro in comments; no confirmed workaround."
+         }
+      }
+   }
+   ```
+
+3. Run an agent content review for each highlighted issue using title, body, labels, and comments.
 
-4. Copy output to team communication channel (Teams, email, wiki)
+4. Add annotation tags in report narrative: `[severity:critical|high|medium|low|none]`, `[blocker:yes|no]`, and optional `[confidence:XX]`.
+
+5. Copy output to team communication channel (Teams, email, wiki)
 
 ### Workflow 2: Analyze a Specific Feature Area
 
 1. Get area-specific issues:
-   ```powershell
-   gh issue list --repo microsoft/WindowsAppSDK --label "area-Notification" --state open --json number,title,labels,reactionGroups,createdAt,comments
-   ```
+    ```powershell
+    gh issue list --repo microsoft/WindowsAppSDK --label "area-Notifications" --state open --json number,title,body,labels,reactionGroups,createdAt,comments
+    ```
 
 2. Check individual issue scores:
    ```powershell
@@ -173,35 +181,93 @@ Retrieve or update the area-to-contact mapping configuration.
 
 2. Update specific area contact:
    ```powershell
-   ./scripts/Get-AreaContacts.ps1 -Area "area-Notification" -Update
+   ./scripts/Get-AreaContacts.ps1 -Area "area-Notifications" -Update
    ```
 
 ## Highlight Scoring Algorithm
 
-Issues are scored (0-100) based on multiple factors. See [scoring-algorithm.md](./references/scoring-algorithm.md) for complete details.
+Issues are scored with a normalized composite on a `0..100` scale. See [scoring-algorithm.md](./references/scoring-algorithm.md) for complete details.
+
+PowerShell computes deterministic score factors and highlight labels. Severity and blocker are assessment-driven inputs.
+
+Scripts load baseline assessments from:
+
+- `./references/IssueAssessments.json`
+
+They also load runtime agent assessments from:
+
+- `./references/AgentAssessments.json`
+
+If both files include the same issue number, `IssueAssessments.json` takes precedence.
+
+If either file is missing or malformed, scripts emit status/warning output and continue with fallback behavior.
+
+Implementation note: scripts load these files through a single `Read-Assessments` function in `./scripts/ReportLib.ps1` using `-AssessmentType Issue` or `-AssessmentType Agent`.
+
+## Agent Content Review
+
+After generating scores, review each highlighted issue and assign these annotations based on title, body, labels, and comments:
+
+- `[severity:critical|high|medium|low|none]`
+- `[blocker:yes|no]`
+- `[confidence:XX]`
+
+After assigning annotations, persist the same decision to `./references/AgentAssessments.json` and include `reasoning` for every updated issue entry.
+
+Use this severity rubric:
+
+| Tier | Meaning |
+|------|---------|
+| `critical` | Crash, data loss, severe regression, or broad user/system impact |
+| `high` | Serious functional break affecting key workflows |
+| `medium` | User-visible defect or limitation with practical workaround |
+| `low` | Minor issue, edge case, docs/polish impact |
+| `none` | No clear severity signal from issue content |
+
+Mark `[blocker:yes]` only when issue content explicitly indicates dependency blocking (for example: "blocked by", "blocking release", "must be fixed before") and no workaround has been provided. Otherwise use `[blocker:no]`.
+
+Use this confidence rubric:
+
+| Score | Level | Meaning |
+|-------|-------|---------|
+| **80-100** | High | Multiple strong signals agree: issue content, labels, reactions, age, comments, and recent discussion all support highlighting it |
+| **60-79** | Medium-High | Strong support from the score breakdown and issue details, with only minor ambiguity |
+| **40-59** | Medium | Reasonable highlight candidate, but some evidence is weak, stale, or mixed |
+| **20-39** | Low | The numeric score is carrying most of the case; supporting context is limited or ambiguous |
+| **0-19** | Very Low | The issue surfaced mechanically, but the agent cannot defend highlighting it with the available evidence |
+
+Confidence should consider:
+
+- whether the issue body and comments clearly support the highlight label
+- whether the numeric score is driven by multiple meaningful factors instead of one outlier
+- whether the issue still appears relevant after reading recent discussion
+- whether the highlight reason would be easy to defend in a triage meeting
 
 ### Quick Reference: Score Factors
 
-| Factor | Weight | Description |
+| Factor | Weight (between 0 and 1) | Description |
 |--------|--------|-------------|
-| **Reactions** | 30 | Total reactions (👍, ❤️, 🚀, etc.) indicate community interest |
-| **Age** | 25 | Older untriaged issues get higher priority |
-| **Comments** | 20 | Active discussion indicates importance |
-| **Severity** | 15 | Labels like `bug`, `regression`, `crash` increase score |
-| **Blockers** | 10 | Issues blocking other work get prioritized |
+| **Reactions** | `weights.reactions` | `round(min(1, totalReactions / rankingCeilings.reactions) * weight * 100, 1)` |
+| **Age** | `weights.age` | `round(min(1, issueAgeDays / rankingCeilings.ageDays) * weight * 100, 1)` |
+| **Comments** | `weights.comments` | `round(min(1, commentCount / rankingCeilings.comments) * weight * 100, 1)` |
+| **Severity** | `weights.severity` | `round(weight * severityMultipliers[tier] * 100, 1)` from assessments (`critical/high/medium/low/none`) |
+| **Blockers** | `weights.blockers` | Adds `round(weight * 100, 1)` when assessed `isBlocker=true` |
+
+`Total = min(100, round(sum of factor points, 1))`.
+
+`rankingCeilings` are independent per-factor scales and do not need to sum to any target.
+
+`recommendationBands` are fixed score-ratio boundaries (`high`, `medium`, `normal`) used to classify recommendations. They are ratio cutoffs, not statistical percentiles.
 
 ### Highlight Labels (Output)
 
 The report adds reason labels to highlighted issues:
 
 | Label | Meaning |
 |-------|---------|
-| `🔥 Hot` | High reaction count (community demand) |
+| `🌟 Popular` | High reaction count (≥5 reactions) |
 | `⏰ Aging` | Open > 90 days without triage |
-| `🐛 Regression` | Marked as regression or recent breakage |
-| `🚧 Blocker` | Blocking other issues or teams |
-| `📈 Trending` | High comment activity recently |
-| ` Popular` | Feature proposal with significant support |
+| `📈 Trending` | High comment activity (≥10 comments) |
 
 ## Report Output Format
 
@@ -210,44 +276,39 @@ The report adds reason labels to highlighted issues:
 ```markdown
 | Feature Area | Area Contact | Open | Triage | Proposals | Closed | Highlights |
 |--------------|--------------|------|--------|-----------|--------|------------|
-| area-Notification | Notifications Owner | 34 | 8 | 11 | 0 | 🔥 [#2894](link) Hot, ⏰ [#3001](link) Aging |
-| area-Widgets | Widgets Owner | 21 | 10 | 4 | 0 | 📈 [#3958](link) Trending |
+| area-Notification | Contact Name | 34 | 8 | 11 | 0 | 🌟 [#2894](link), ⏰ [#3001](link) |
+| area-Widgets | Contact Name | 21 | 10 | 4 | 0 | 📈 [#3958](link) |
+```
+
+### Agent-Reviewed Output
+
+When producing the final narrative report, the agent should append content-review annotations:
+
+```markdown
+| Feature Area | Area Contact | Open | Triage | Proposals | Closed | Highlights |
+|--------------|--------------|------|--------|-----------|--------|------------|
+| area-Notification | Contact Name | 34 | 8 | 11 | 0 | 🌟 [#2894](link) [severity:high] [blocker:no] [confidence:85], ⏰ [#3001](link) [severity:medium] [blocker:no] [confidence:72] |
+| area-Widgets | Contact Name | 21 | 10 | 4 | 0 | 📈 [#3958](link) [severity:low] [blocker:no] [confidence:68] |
 ```
 
 ### Special Status Indicators
 
 | Indicator | Meaning |
 |-----------|---------|
 | `0️⃣🐛🥳` | Zero bugs — celebrate! |
-| `🆕` | New area (no historical data) |
 | `-` | Data not applicable or unavailable |
 
 ## Configuration
 
 ### Area Contacts
 
-Contact mappings are stored in [area-contacts.md](./references/area-contacts.md). Update this file when team assignments change.
+Contact mappings are stored in [area-contacts.json](./references/area-contacts.json). Update this file when team assignments change.
 
 ### Custom Scoring Weights
 
-Modify scoring weights in `./scripts/ScoringConfig.json`:
-
-```json
-{
-  "weights": {
-    "reactions": 30,
-    "age": 25,
-    "comments": 20,
-    "severity": 15,
-    "blockers": 10
-  },
-  "thresholds": {
-    "hot_reactions": 10,
-    "aging_days": 90,
-    "trending_comments": 5
-  }
-}
-```
+Modify scoring weights in `./scripts/ScoringConfig.json`.
+
+`severityMultipliers` maps assessment tiers to the percentage of `weights.severity` that is applied.
 
 ## Troubleshooting
 
@@ -257,13 +318,13 @@ Modify scoring weights in `./scripts/ScoringConfig.json`:
 | `authentication required` | Run `gh auth login` and follow prompts |
 | Rate limit exceeded | Wait or use `--limit` to reduce API calls |
 | Missing area label | Issue may use non-standard label; check label list |
-| Contact not found | Update [area-contacts.md](./references/area-contacts.md) |
+| Contact not found | Update [area-contacts.json](./references/area-contacts.json) |
 
 ## Common Commands Reference
 
 ```powershell
-# List all area labels
-gh label list --repo microsoft/WindowsAppSDK --search "area-" --json name
+# List all area labels (uses Get-RepositoryLabels.ps1 as the single source of truth)
+./.github/skills/triage-meeting-prep/scripts/Get-RepositoryLabels.ps1 -Filter "area-*" -OutputFormat table
 
 # Get issue details with reactions
 gh issue view 4651 --repo microsoft/WindowsAppSDK --json number,title,labels,reactionGroups,createdAt,comments,author
@@ -278,5 +339,5 @@ gh issue list --repo microsoft/WindowsAppSDK --state open --limit 1000 --json nu
 ## References
 
 - [Scoring Algorithm Details](./references/scoring-algorithm.md) — Complete scoring methodology
-- [Area Contacts](./references/area-contacts.md) — Feature area ownership mapping
+- [Area Contacts](./references/area-contacts.json) — Feature area ownership mapping
 - [Report Template](./templates/report-template.md) — Customizable output template

.github/skills/issue-triage-report/references/AgentAssessments.json

@@ -0,0 +1,5 @@
+{
+  "assessments": {},
+  "generatedAt": "2026-04-14T00:00:00Z",
+  "assessedBy": "Copilot"
+}

.github/skills/issue-triage-report/references/IssueAssessments.json

@@ -0,0 +1,11 @@
+{
+  "assessments": {
+    "1234": {
+      "severityTier": "high",
+      "isBlocker": false,
+      "reasoning": "Example entry. Update for your triage run."
+    }
+  },
+  "generatedAt": "2026-04-14T00:00:00Z",
+  "assessedBy": "Copilot"
+}

.github/skills/issue-triage-report/references/area-contacts.json

@@ -2,9 +2,9 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$comment": "Area contacts configuration. Create your own copy at .user/issue-triage-report/area-contacts.json with actual team contacts.",
   "areaContacts": {
-    "area-ExampleFeature": { "primary": "Primary Contact Name", "secondary": null },
-    "area-AnotherFeature": { "primary": "Primary Contact", "secondary": "Secondary Contact" },
-    "area-FeatureWithNotes": { "primary": "Contact Name", "secondary": null, "notes": "Special handling notes" }
+    "area-ExampleFeature": { "contact": "Contact Name" },
+    "area-AnotherFeature": { "contact": "Contact Name", "notes": "Optional notes about this area" },
+    "area-FeatureWithNotes": { "contact": "Contact Name", "notes": "Special handling notes" }
   },
   "specialAreas": {
     "triageOnly": ["area-TriageOnlyExample"],

.github/skills/issue-triage-report/references/issue-assessments-schema.json

@@ -0,0 +1,40 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "IssueAssessments",
+  "type": "object",
+  "additionalProperties": false,
+  "properties": {
+    "assessments": {
+      "type": "object",
+      "description": "Issue-number keyed assessments. Keys are issue numbers as strings.",
+      "additionalProperties": {
+        "type": "object",
+        "additionalProperties": false,
+        "properties": {
+          "severityTier": {
+            "type": "string",
+            "enum": ["critical", "high", "medium", "low", "none"],
+            "description": "Optional. Empty or missing value falls back to default behavior."
+          },
+          "isBlocker": {
+            "type": "boolean",
+            "description": "Optional. Empty or missing value falls back to default behavior."
+          },
+          "reasoning": {
+            "type": "string",
+            "description": "Optional human or agent notes."
+          }
+        }
+      }
+    },
+    "generatedAt": {
+      "type": "string",
+      "description": "Optional ISO-8601 timestamp."
+    },
+    "assessedBy": {
+      "type": "string",
+      "description": "Optional source marker, e.g. Copilot or human alias."
+    }
+  },
+  "required": ["assessments"]
+}