Skip to content

Commit bebd936

Browse files
mhuisiclaude
andcommitted
test: new test framework
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 20c6ed9 commit bebd936

129 files changed

Lines changed: 23537 additions & 10538 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/CLAUDE.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,57 @@ npm run lint
2121
cd vscode-lean4 && npm run package
2222
```
2323

24+
## Testing
25+
26+
The full test framework is documented in `vscode-lean4/test/README.md`. Read it once before authoring or debugging tests — it covers prompt stubbing, the helper map, and the per-file VS Code instance model. The summary below covers the day-to-day commands and how to diagnose failures.
27+
28+
### Running tests locally
29+
30+
Always use the `test:headless*` variants — they route through `scripts/headless.mjs`, which wraps in `xvfb-run -a` on Linux (avoiding focus-stealing for the duration of the suite; requires `xorg-x11-server-Xvfb` / `xvfb`) and runs the inner command directly on Windows / macOS. Plain `npm test` opens real VS Code windows on Linux.
31+
32+
```bash
33+
# Full suite: vitest unit tests + infoview tsc check + grammar + vscode-test-cli + wdio.
34+
npm run test:headless
35+
36+
# Just the wdio (UI/InfoView) subset, from vscode-lean4/.
37+
npm run test:headless:wdio --workspace=lean4
38+
39+
# Network suite (real network: elan install, lake clone, mathlib cache).
40+
npm run test:headless:network --workspace=lean4
41+
42+
# Vitest only — pure-TypeScript tests, no VS Code, no xvfb needed.
43+
npm run test:unit
44+
45+
# Single vscode-test-cli file (replace <name>):
46+
cd vscode-lean4 && node scripts/headless.mjs npx vscode-test --label cli-<name>
47+
# e.g. cli-abbreviation, cli-launch-modes, cli-project-actions
48+
# (label list comes from filenames under test/vscode-test-cli/)
49+
```
50+
51+
`npm ci` does not build workspace `dist/` directories; both type-aware lint and the `lean4-infoview` tsc check require them. Run `npm run build:dev` (or `npm run build`) first if you've just cloned or run `npm ci`.
52+
53+
### Diagnosing test failures
54+
55+
The framework writes failure artifacts to two gitignored directories under `vscode-lean4/`. Both are wiped at the start of each run.
56+
57+
- **`vscode-test-cli-output/`** — populated by every `vscode-test-cli` and `nightly` test. On failure, `helpers/teardown.ts:dumpStateIfFailed` writes `<safe-test-name>.json` with the workbench's textual state at the failure point: diagnostics for every URI, active editor (uri/cursor/text), visible/open editors, and per-channel transcripts of every write the extension made via `window.createOutputChannel`. The output channels are also mirrored to the test runner's stdout in real time as `[output:<channel name>] <text>` — so the CI transcript reads like a live session.
58+
- **`wdio-output/`** — populated by the wdio suite. Always: `wdio-junit-<cid>.xml` (junit), `vscode-logs/` (extension host + renderer logs from `--logs-path`). On failure: `<safe-test-name>.png` (workbench screenshot from `afterTest`).
59+
60+
`runCliTests.mjs` re-emits the failure detail of any failed `cli-*` label at the bottom of the run, so a single assertion is the last thing in the scrollback rather than scrolled off behind nine labels' output.
61+
62+
`scripts/runAllTests.mjs` orchestrates the full top-level suite (unit / infoview tsc / grammar / vscode-test-cli / wdio) and **continues past failures** rather than `&&`-bailing on the first one — so a vitest regression doesn't hide a wdio regression in the same CI run. Final exit code is non-zero if any suite failed; the per-suite pass/fail summary lands at the bottom of the log.
63+
64+
### Diagnosing CI failures
65+
66+
CI runs three test jobs in `.github/workflows/test.yml` (`test`, `test-nightly`, `network`), each as an OS matrix over `ubuntu-latest` × `windows-latest`, plus `lint`/`package` in `.github/workflows/on-push.yml`. All test-job artifacts (`vscode-test-cli-output/`, `wdio-output/`, plus the raw stdout) are uploaded for inspection.
67+
68+
For a failure, follow this order:
69+
70+
1. **Read the captured `[output:<channel>] …` lines** in the job log around the failing test. Lake stdout, LSP server output, and the extension's own diagnostics flow through here. Often the assertion is downstream of an earlier `error:` line that names the actual cause (e.g., a network failure or a missing toolchain step).
71+
2. **Download the artifact zip** via the GitHub Actions UI and unzip locally. The `vscode-test-cli-output/<safe-test-name>.json` for a failed test contains the post-failure workbench state in JSON form.
72+
3. **For wdio failures**, look at the saved screenshot first — it shows what the workbench looked like at failure time, including any unexpected modal, focus state, or partial render. Pair it with `vscode-logs/` for renderer / extension-host errors.
73+
4. **For "doesn't reproduce locally"**, the most common causes are: workspace `dist/` not built before the failing step (CI's fresh runner has empty `dist/`; `npm ci` does not build); `LEAN4_TEST_HOME_OVERRIDE` set to a path inside an outer git repo (CI sets it to `${{ github.workspace }}/.lean4-test-home`, which is inside the actions/checkout clone — `setupTestHome.mjs` sets `GIT_CEILING_DIRECTORIES` to defang this); GitHub-hosted runner image changes; or a Lean stable release that broke a fixture (the toolchain pin policy is documented in `test/README.md`).
74+
2475
## Architecture
2576

2677
This is a **Lerna monorepo** with npm workspaces containing five packages:

.eslintrc.js

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,16 @@ module.exports = {
3131
'./lean4-unicode-input/tsconfig.json',
3232
'./lean4-unicode-input-component/tsconfig.json',
3333
'./vscode-lean4/tsconfig.json',
34+
'./vscode-lean4/tsconfig.test.json',
3435
'./vscode-lean4/webview/tsconfig.json',
3536
'./vscode-lean4/loogleview/tsconfig.json',
3637
'./vscode-lean4/abbreviationview/tsconfig.json',
3738
],
3839
sourceType: 'module',
3940
},
40-
ignorePatterns: ['**/dist/**/*', '**/out/**/*', '*.js'],
41+
// ESLint v8 ignores files starting with `.` by default; the negation
42+
// re-includes our dotfile-prefixed configs (`.vscode-test.mjs`).
43+
ignorePatterns: ['**/dist/**/*', '**/out/**/*', '*.js', '!.vscode-test.mjs'],
4144
plugins: ['@typescript-eslint', '@typescript-eslint/eslint-plugin'],
4245
rules: {
4346
'@typescript-eslint/adjacent-overload-signatures': 'error',
@@ -140,4 +143,16 @@ module.exports = {
140143
radix: 'off',
141144
'no-shadow': 'off',
142145
},
146+
overrides: [
147+
{
148+
// .mjs config/scripts (`.vscode-test.mjs`, `wdio.conf.mjs`, `scripts/*.mjs`)
149+
// are not part of any tsconfig, so the type-aware ruleset can't run on them.
150+
// `disable-type-checked` turns off every rule from `recommended-requiring-type-checking`
151+
// (which is extended at the top level) and `parserOptions.project: null` keeps the
152+
// parser from trying to look the file up in a project.
153+
files: ['*.mjs'],
154+
parserOptions: { project: null },
155+
extends: ['plugin:@typescript-eslint/disable-type-checked'],
156+
},
157+
],
143158
}
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Cache VS Code test binaries
2+
description: >
3+
Resolve the current stable VS Code version and restore/save the VS Code
4+
test binary cache keyed on that version. Caches both `.vscode-test`
5+
(vscode-test-cli) and `.wdio-vscode-service` (wdio); jobs that don't
6+
run wdio tolerate the second path being absent harmlessly.
7+
8+
runs:
9+
using: composite
10+
steps:
11+
# Resolve `:stable` to a concrete VS Code version so the cache key rotates
12+
# automatically when stable bumps. The previous keying — bare
13+
# `vscode-test-${{ runner.os }}-stable` — never changed when stable
14+
# moved, so each runner kept reusing the version that was first cached.
15+
- name: Resolve VS Code stable version
16+
id: vscode
17+
uses: actions/github-script@v7
18+
with:
19+
result-encoding: string
20+
script: |
21+
const r = await fetch('https://update.code.visualstudio.com/api/releases/stable')
22+
const versions = await r.json()
23+
core.setOutput('version', versions[0])
24+
25+
- name: Cache VS Code test binaries
26+
uses: actions/cache@v4
27+
with:
28+
path: |
29+
vscode-lean4/.vscode-test
30+
vscode-lean4/.wdio-vscode-service
31+
# Include the resolved version so a new stable triggers a fresh
32+
# download. `restore-keys:` falls back to the most recent cache for
33+
# this OS so the new run still benefits from partial reuse (the
34+
# `.wdio-vscode-service/` extensions dir, etc.) while it pulls the
35+
# new VS Code binary.
36+
key: vscode-test-${{ runner.os }}-${{ steps.vscode.outputs.version }}
37+
restore-keys: |
38+
vscode-test-${{ runner.os }}-
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
name: Set up elan + Lean toolchain
2+
description: >
3+
Install elan into the test-home directory (if not cached), prepend its
4+
`bin/` to `$GITHUB_PATH`, and pre-install a Lean toolchain so the
5+
download surfaces as a named CI step rather than blocking the first
6+
test invocation. Optionally sets that toolchain as the elan default.
7+
The caller is responsible for restoring/saving the elan cache around
8+
this action; cache-key choice is job-specific (resolved version for
9+
`:stable`, UTC date for `:nightly`).
10+
11+
inputs:
12+
channel:
13+
description: Lean toolchain channel (e.g. leanprover/lean4:stable, leanprover/lean4:nightly).
14+
required: true
15+
default:
16+
description: If `'true'`, set the channel as elan's default toolchain after install.
17+
required: false
18+
default: 'false'
19+
20+
runs:
21+
using: composite
22+
steps:
23+
# Run the same install scripts the extension itself executes (see
24+
# `vscode-lean4/scripts/elan-install/install-{unix.sh,windows.ps1}`,
25+
# imported by `src/utils/leanInstaller.ts` via webpack `asset/source`).
26+
# Single source of truth — edit the script files and both production
27+
# and CI pick up the change.
28+
#
29+
# The scripts default-toolchain to `leanprover/lean4:stable`; callers
30+
# that need a different default (e.g. the nightly-Lean job) pass
31+
# `default: true` so the pre-install step below flips it.
32+
- name: Install elan (Linux/macOS, if not cached)
33+
if: runner.os != 'Windows'
34+
shell: bash
35+
run: |
36+
HOME_OVERRIDE="$GITHUB_WORKSPACE/.lean4-test-home"
37+
mkdir -p "$HOME_OVERRIDE"
38+
if [ ! -x "$HOME_OVERRIDE/.elan/bin/elan" ]; then
39+
HOME="$HOME_OVERRIDE" ELAN_HOME="$HOME_OVERRIDE/.elan" \
40+
bash vscode-lean4/scripts/elan-install/install-unix.sh
41+
fi
42+
echo "$HOME_OVERRIDE/.elan/bin" >> "$GITHUB_PATH"
43+
44+
- name: Install elan (Windows, if not cached)
45+
if: runner.os == 'Windows'
46+
shell: pwsh
47+
run: |
48+
$homeOverride = Join-Path $env:GITHUB_WORKSPACE '.lean4-test-home'
49+
New-Item -ItemType Directory -Force -Path $homeOverride | Out-Null
50+
$elanBin = Join-Path $homeOverride '.elan\bin\elan.exe'
51+
if (-not (Test-Path $elanBin)) {
52+
$env:USERPROFILE = $homeOverride
53+
$env:ELAN_HOME = (Join-Path $homeOverride '.elan')
54+
& "$env:GITHUB_WORKSPACE\vscode-lean4\scripts\elan-install\install-windows.ps1"
55+
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
56+
}
57+
Add-Content -Path $env:GITHUB_PATH -Value (Join-Path $homeOverride '.elan\bin')
58+
59+
# `elan-init` only installs the elan binary and records `:stable` as the
60+
# default toolchain — it does *not* fetch the toolchain itself. Without
61+
# this step, the first test that resolves the channel eats the full
62+
# toolchain download and a network blip there surfaces as a confusing
63+
# test failure rather than a named CI step. `elan toolchain install`
64+
# eagerly re-resolves the channel, so it also covers the cache-hit /
65+
# channel-rotated case (cache restored from a previous run with an older
66+
# resolution → this step downloads the new toolchain).
67+
#
68+
# `elan toolchain install` is *not* idempotent: it errors with
69+
# `'leanprover/lean4:vX.Y.Z' is already installed` when the cache
70+
# already holds the resolved version. Treat that specific message as
71+
# success — we wanted the toolchain installed and it is — while
72+
# propagating any other non-zero exit (network failure, bad channel,
73+
# etc.).
74+
#
75+
# `shell: bash` works on Windows too because GitHub-hosted Windows
76+
# runners ship Git Bash; `elan` is the same idempotency contract on
77+
# both OSes once on PATH.
78+
- name: Pre-install ${{ inputs.channel }}${{ inputs.default == 'true' && ' + set as elan default' || '' }}
79+
shell: bash
80+
env:
81+
ELAN_HOME: ${{ github.workspace }}/.lean4-test-home/.elan
82+
CHANNEL: ${{ inputs.channel }}
83+
SET_DEFAULT: ${{ inputs.default }}
84+
run: |
85+
if out=$(elan toolchain install "$CHANNEL" 2>&1); then
86+
echo "$out"
87+
else
88+
ec=$?
89+
echo "$out"
90+
echo "$out" | grep -q 'is already installed' || exit "$ec"
91+
fi
92+
if [ "$SET_DEFAULT" = "true" ]; then
93+
elan default "$CHANNEL"
94+
fi

.github/workflows/on-push.yml

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,26 @@ on:
1414
pull_request:
1515
branches:
1616
- '*'
17+
# Restrict `push` to maintainer-curated refs. Feature branches with an open
18+
# PR fire `pull_request` instead, so this avoids the duplicate run while
19+
# still gating direct merges to `master` / release branches and tag pushes
20+
# (the publish-vsce / release jobs below only run on tag pushes anyway).
1721
push:
1822
branches:
19-
- '*'
23+
- master
24+
- 'release/**'
2025
tags:
2126
- '*'
2227

28+
# Cancel in-progress runs of this workflow when a newer commit lands on the
29+
# same ref. Rapid-fire pushes (or a `push` and `pull_request` event for the
30+
# same ref) coalesce — only the latest run survives. `workflow_dispatch`
31+
# runs get their own group via `github.run_id` so a manual publish/release
32+
# never cancels another in-flight publish/release.
33+
concurrency:
34+
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'workflow_dispatch' && github.run_id || '' }}
35+
cancel-in-progress: true
36+
2337
permissions:
2438
contents: write
2539
id-token: write
@@ -98,6 +112,15 @@ jobs:
98112
- name: Install
99113
run: npm ci
100114

115+
# Type-aware lint rules (`no-redundant-type-constituents`,
116+
# `no-unnecessary-type-assertion`) need workspace packages' `dist/`
117+
# populated so cross-package imports resolve to real types instead of
118+
# implicit `any`. Without this, `T | undefined` looks like
119+
# `any | undefined` to typescript-eslint and the rule fires
120+
# spuriously. Same rationale as the build step in `test.yml`.
121+
- name: Build (for type-aware lint)
122+
run: npm run build:dev
123+
101124
- name: Lint
102125
run: npm run lint
103126

0 commit comments

Comments
 (0)