Skip to content

Commit 6ebd230

Browse files
Copilothanniavalera
Copilot
and
hanniavalera
committed
Add Copilot-driven test coverage agent (4-file system)
Agent-Logs-Url: https://github.com/microsoft/vscode-cmake-tools/sessions/2ef2460f-de0e-4cf1-bca3-cc67940ee648 Co-authored-by: hanniavalera <[email protected]>
1 parent 24f6112 commit 6ebd230

4 files changed

Lines changed: 327 additions & 0 deletions

File tree

‎.github/PULL_REQUEST_TEMPLATE/coverage.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
name: Coverage improvement
3+
about: PR opened by the Copilot coverage agent
4+
---
5+
6+
## Coverage improvement
7+
8+
### What this PR covers
9+
<!-- List which files' coverage improved and by how much -->
10+
| File | Before | After | Δ |
11+
|------|--------|-------|---|
12+
| | | | |
13+
14+
### Closes
15+
<!-- Link the coverage issue: Closes #NNN -->
16+
17+
### Self-audit results
18+
19+
- [ ] `yarn backendTests` passes
20+
- [ ] `yarn unitTests` passes
21+
- [ ] Every file listed in the issue improved by ≥ 10 percentage points OR reached ≥ 60% line coverage
22+
- [ ] No test uses `assert.ok(true)` or is an empty stub
23+
- [ ] Test names describe behavior, not implementation
24+
- [ ] No test depends on another test's side effects
25+
- [ ] Presets-mode and kits/variants mode both exercised where relevant
26+
- [ ] Single-config and multi-config generator paths both tested where relevant
27+
- [ ] `yarn lint` passes
28+
- [ ] `CHANGELOG.md` has an entry under `Improvements:`
29+
30+
### Coverage delta (paste `nyc` text report here)
31+
```
32+
(paste output of: npx nyc report --reporter=text)
33+
```

‎.github/copilot-test-coverage.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
---
2+
description: "Instructions for the Copilot coding agent when working on test-coverage issues."
3+
applyTo: "test/unit-tests/**/*.ts"
4+
---
5+
6+
# Test Coverage Agent — Self-Audit Protocol
7+
8+
You are improving test coverage for `microsoft/vscode-cmake-tools`.
9+
This file contains mandatory protocol. Read all of it before writing code.
10+
11+
## Step 1 — Orient before writing
12+
13+
- Read every source file you will test in full
14+
- Read existing tests for that module (if any) in `test/unit-tests/`
15+
- Identify every exported function, class, and branch condition
16+
- Do **not** write tests for private implementation details — test the public API
17+
18+
## Step 2 — Write real tests, not stubs
19+
20+
Every test must:
21+
- Have a descriptive name: `'expandString handles undefined variable in preset context'`
22+
- Assert exactly one logical behavior per `test()` block
23+
- Not depend on side effects from another test
24+
- Use `assert.strictEqual` / `assert.deepStrictEqual` over loose equality
25+
- For async code: `await` and assert the resolved value — never `.then()`
26+
27+
## Step 3 — Run the full self-audit before opening the PR
28+
29+
```bash
30+
# 1. Type-check test files
31+
npx tsc -p test.tsconfig.json --noEmit
32+
33+
# 2. Lint
34+
yarn lint
35+
36+
# 3. Run backend tests (fast — no xvfb needed)
37+
yarn backendTests
38+
39+
# 4. Run unit tests
40+
yarn unitTests
41+
42+
# 5. Confirm coverage improved for the specific file
43+
npx nyc --reporter=text \
44+
node ./node_modules/mocha/bin/_mocha \
45+
-u tdd --timeout 999999 --colors \
46+
-r ts-node/register \
47+
-r tsconfig-paths/register \
48+
"./test/unit-tests/backend/**/*.test.ts"
49+
```
50+
51+
All five steps must pass. If any fail, fix the failures before opening the PR.
52+
53+
## Step 4 — Coverage bar
54+
55+
Do **not** open the PR unless every file listed in the issue has either:
56+
- improved by ≥ 10 percentage points from the baseline in the issue, **OR**
57+
- reached ≥ 60% line coverage
58+
59+
## Test quality rules specific to this repo
60+
61+
| Rule | Why |
62+
|------|-----|
63+
| Test both `useCMakePresets` branches where the source branches on it | Most bugs affect only one mode |
64+
| Test both single-config and multi-config generator paths where relevant | `CMAKE_BUILD_TYPE` vs `--config` are frequent bug sources |
65+
| For `src/expand.ts` changes: test every macro type | `copilot-instructions.md` explicitly mandates this |
66+
| For `src/diagnostics/`: test each compiler family's parser | `diagnostics.test.ts` is the largest test file — keep it comprehensive |
67+
| Use `@cmt/` path alias for imports from `src/` | Never use relative paths from outside `src/` |
68+
| Never use `console.log` in test files | Use the module logger or plain `assert` |
69+
70+
## PR requirements
71+
72+
- Branch name: `coverage/<module-name>-tests`
73+
- Open as **ready for review** only after the self-audit checklist in the issue is fully checked
74+
- PR description must use `.github/PULL_REQUEST_TEMPLATE/coverage.md`
75+
- `CHANGELOG.md` must have one entry under `Improvements:`

‎.github/scripts/check-coverage-and-open-issue.mjs‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
#!/usr/bin/env node
2+
// @ts-check
3+
import { readFileSync } from 'fs';
4+
import { execSync } from 'child_process';
5+
6+
const THRESHOLD = Number(process.env.THRESHOLD ?? 60);
7+
const SUMMARY_PATH = 'coverage/coverage-summary.json';
8+
const REPO = process.env.GITHUB_REPOSITORY; // e.g. "microsoft/vscode-cmake-tools"
9+
const RUN_URL = `https://github.com/${REPO}/actions/runs/${process.env.GITHUB_RUN_ID}`;
10+
11+
// ── 1. Read Istanbul JSON summary ────────────────────────────────────────────
12+
let summary;
13+
try {
14+
summary = JSON.parse(readFileSync(SUMMARY_PATH, 'utf8'));
15+
} catch {
16+
console.log('No coverage summary found — skipping issue creation.');
17+
process.exit(0);
18+
}
19+
20+
// ── 2. Find files below threshold ────────────────────────────────────────────
21+
const belowThreshold = [];
22+
23+
for (const [file, metrics] of Object.entries(summary)) {
24+
if (file === 'total') continue;
25+
if (!file.startsWith('src/')) continue;
26+
27+
const linePct = metrics.lines.pct ?? 0;
28+
const branchPct = metrics.branches.pct ?? 0;
29+
const fnPct = metrics.functions.pct ?? 0;
30+
31+
if (linePct < THRESHOLD || fnPct < THRESHOLD) {
32+
belowThreshold.push({ file, linePct, branchPct, fnPct });
33+
}
34+
}
35+
36+
const total = summary.total ?? {};
37+
const totalLines = total.lines?.pct ?? 0;
38+
39+
console.log(`\nTotal line coverage: ${totalLines}% (threshold: ${THRESHOLD}%)`);
40+
console.log(`Files below threshold: ${belowThreshold.length}`);
41+
42+
if (belowThreshold.length === 0 && totalLines >= THRESHOLD) {
43+
console.log('✅ Coverage is above threshold — no issue needed.');
44+
process.exit(0);
45+
}
46+
47+
// ── 3. Build the issue body (this is the Copilot agent's instruction set) ────
48+
belowThreshold.sort((a, b) => a.linePct - b.linePct); // worst files first
49+
50+
const tableRows = belowThreshold
51+
.slice(0, 20) // cap at 20 files per issue to keep it focused
52+
.map(f => `| \`${f.file}\` | ${f.linePct}% | ${f.branchPct}% | ${f.fnPct}% |`)
53+
.join('\n');
54+
55+
const fileList = belowThreshold
56+
.slice(0, 20)
57+
.map(f => `- \`${f.file}\` — ${f.linePct}% line coverage`)
58+
.join('\n');
59+
60+
const issueBody = `\
61+
## Coverage below ${THRESHOLD}% threshold
62+
63+
> **This issue is the instruction set for the GitHub Copilot coding agent.**
64+
> Copilot: read this entire body before writing a single line of code.
65+
66+
**Coverage run:** ${RUN_URL}
67+
**Total line coverage:** ${totalLines}% — threshold is ${THRESHOLD}%
68+
69+
### Files requiring new tests
70+
71+
| File | Lines | Branches | Functions |
72+
|------|-------|----------|-----------|
73+
${tableRows}
74+
75+
---
76+
77+
### Agent instructions
78+
79+
You are improving test coverage in \`microsoft/vscode-cmake-tools\`.
80+
Read \`.github/copilot-test-coverage.md\` before starting — it contains the
81+
mandatory self-audit protocol and test quality rules for this repo.
82+
83+
**Files to cover (worst first):**
84+
${fileList}
85+
86+
For each file:
87+
1. Read the source file fully before writing any test
88+
2. Identify the module's exported API surface
89+
3. Write tests in \`test/unit-tests/\` that cover the uncovered branches
90+
4. Run the self-audit steps from \`copilot-test-coverage.md\`
91+
5. Only open the PR after every self-audit step passes
92+
93+
### Self-audit checklist (must be checked before opening PR)
94+
95+
- [ ] \`yarn backendTests\` passes with no new failures
96+
- [ ] \`yarn unitTests\` passes with no new failures
97+
- [ ] Each file listed above improved by ≥ 10 percentage points OR reached ≥ ${THRESHOLD}% line coverage
98+
- [ ] No test uses \`assert.ok(true)\` or is an empty stub
99+
- [ ] Test names describe behavior: \`'expandString handles undefined variable'\` not \`'test 1'\`
100+
- [ ] No test depends on another test's side effects
101+
- [ ] Presets-mode and kits/variants mode both exercised where the source branches on \`useCMakePresets\`
102+
- [ ] Single-config and multi-config generator paths both tested where relevant
103+
- [ ] \`yarn lint\` passes
104+
- [ ] \`CHANGELOG.md\` has an entry under \`Improvements:\`
105+
106+
### Constraints
107+
108+
- Tests go in \`test/unit-tests/\` — use Mocha \`suite\`/\`test\` with \`assert\`
109+
- Import source under test via the \`@cmt/\` path alias
110+
- Do **not** open the PR as a draft if the self-audit fails — fix it first
111+
- Do **not** touch source files outside \`test/\`
112+
`;
113+
114+
// ── 4. Open the issue via gh CLI ──────────────────────────────────────────────
115+
const title = `chore: improve test coverage -- ${belowThreshold.length} files below ${THRESHOLD}% (run ${new Date().toISOString().slice(0, 10)})`;
116+
117+
const cmd = [
118+
'gh', 'issue', 'create',
119+
'--repo', REPO,
120+
'--title', JSON.stringify(title),
121+
'--body', JSON.stringify(issueBody),
122+
'--label', 'test-coverage',
123+
].join(' ');
124+
125+
console.log(`\nOpening issue: ${title}`);
126+
execSync(cmd, { stdio: 'inherit' });

‎.github/workflows/coverage-agent.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: Coverage Agent
2+
3+
on:
4+
push:
5+
branches: [ main, 'release/**' ]
6+
pull_request:
7+
branches: [ main, 'release/**' ]
8+
schedule:
9+
- cron: '0 3 * * 1' # weekly Monday 03:00 UTC — keeps noise low
10+
workflow_dispatch:
11+
inputs:
12+
threshold:
13+
description: 'Line-coverage threshold (0-100)'
14+
required: false
15+
default: '60'
16+
17+
env:
18+
THRESHOLD: ${{ github.event.inputs.threshold || '60' }}
19+
20+
jobs:
21+
coverage:
22+
runs-on: ubuntu-latest
23+
permissions:
24+
contents: write
25+
issues: write
26+
27+
steps:
28+
- uses: actions/checkout@v3
29+
30+
- uses: actions/setup-node@v3
31+
with:
32+
node-version: 20
33+
34+
- name: Install Yarn
35+
run: npm install -g yarn
36+
37+
- name: Install dependencies
38+
run: yarn install
39+
40+
- name: Build (test config)
41+
run: yarn pretest # tsc -p test.tsconfig.json
42+
43+
# ── Coverage: backend tests (pure Mocha — nyc wraps directly) ──────────
44+
- name: Run backend tests with coverage
45+
run: |
46+
npx nyc \
47+
--reporter=json \
48+
--reporter=text \
49+
--include='src/**/*.ts' \
50+
--exclude='src/**/*.d.ts' \
51+
--temp-dir=.nyc_output \
52+
node ./node_modules/mocha/bin/_mocha \
53+
-u tdd --timeout 999999 --colors \
54+
-r ts-node/register \
55+
-r tsconfig-paths/register \
56+
"./test/unit-tests/backend/**/*.test.ts"
57+
58+
# ── Coverage: unit tests (VS Code extension host — via xvfb) ──────────
59+
- name: Run unit tests with coverage
60+
uses: GabrielBB/[email protected]
61+
with:
62+
run: |
63+
npx nyc \
64+
--reporter=json \
65+
--include='src/**/*.ts' \
66+
--exclude='src/**/*.d.ts' \
67+
--temp-dir=.nyc_output \
68+
node ./out/test/unit-tests/runTest.js
69+
continue-on-error: true # coverage report still matters even if some tests fail
70+
71+
# ── Merge all coverage runs and emit summary ──────────────────────────
72+
- name: Merge coverage and generate summary
73+
run: |
74+
npx nyc merge .nyc_output coverage/coverage-combined.json
75+
npx nyc report \
76+
--reporter=json-summary \
77+
--reporter=lcov \
78+
--temp-dir=coverage \
79+
-t .nyc_output
80+
81+
- name: Upload coverage artifacts
82+
uses: actions/upload-artifact@v4
83+
with:
84+
name: coverage-report
85+
path: coverage/
86+
retention-days: 14
87+
88+
# ── Evaluate coverage and open issue if below threshold ───────────────
89+
- name: Check coverage and open issue
90+
env:
91+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
92+
THRESHOLD: ${{ env.THRESHOLD }}
93+
run: node .github/scripts/check-coverage-and-open-issue.mjs

0 commit comments

Comments
 (0)