Skip to content

Commit 98c7299

Browse files
josephfuscojosephfusco
committed
Tests: Add visual regression tests for admin reskin pages.
Add 10 new visual regression snapshot tests covering admin pages most impacted by the CSS reskin that previously had no automated visual coverage. Combined with the 22 existing tests, this brings total coverage to 32 admin pages. Also adds a single-command workflow (npm run test:visual), impact summary reporter, fast re-run mode, and auto-open HTML report to streamline CSS iteration during the admin reskin effort.
1 parent eef6fb2 commit 98c7299

6 files changed

Lines changed: 308 additions & 4 deletions

File tree

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,8 @@
129129
"test:php": "node ./tools/local-env/scripts/docker.js run --rm php ./vendor/bin/phpunit",
130130
"test:coverage": "npm run test:php -- --coverage-html ./coverage/html/ --coverage-php ./coverage/php/report.php --coverage-text=./coverage/text/report.txt",
131131
"test:e2e": "wp-scripts test-playwright --config tests/e2e/playwright.config.js",
132-
"test:visual": "wp-scripts test-playwright --config tests/visual-regression/playwright.config.js",
132+
"test:visual": "bash tests/visual-regression/compare-branches.sh",
133+
"test:visual:quick": "bash tests/visual-regression/compare-branches.sh --skip-baselines",
133134
"gutenberg:checkout": "node tools/gutenberg/checkout-gutenberg.js",
134135
"gutenberg:build": "node tools/gutenberg/build-gutenberg.js",
135136
"gutenberg:copy": "node tools/gutenberg/copy-gutenberg-build.js",

‎tests/visual-regression/README.md‎

Lines changed: 73 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,80 @@
22

33
These tests make use of Playwright, with a setup very similar to that of the e2e tests.
44

5+
## Prerequisites
6+
7+
- Node.js >= 20.10.0
8+
- A running WordPress environment (`npm run env:start`)
9+
- Playwright browsers installed (`npx playwright install chromium`)
10+
511
## How to Run the Tests Locally
612

7-
1. Check out trunk.
8-
2. Run `npm run test:visual` to generate some base snapshots.
13+
From a feature branch with a clean working tree, run:
14+
15+
```bash
16+
npm run test:visual
17+
```
18+
19+
This will automatically:
20+
1. Check out trunk to generate baseline snapshots.
21+
2. Return to your feature branch.
22+
3. Compare the current branch against those baselines.
23+
4. Print a visual impact summary in the terminal.
24+
5. Open the HTML report with side-by-side visual diffs.
25+
26+
### Quick Re-run (Skip Baseline Generation)
27+
28+
After the first run, baselines from trunk are already saved. To re-compare without regenerating them:
29+
30+
```bash
31+
npm run test:visual:quick
32+
```
33+
34+
This is useful when iterating on CSS — no need to commit changes or wait for trunk baselines to regenerate each time.
35+
36+
**Typical workflow:**
37+
```bash
38+
npm run test:visual # First run: generates baselines from trunk
39+
# ... tweak CSS ...
40+
npm run test:visual:quick # Fast: reuses existing baselines
41+
# ... tweak more CSS ...
42+
npm run test:visual:quick # Fast again
43+
```
44+
45+
### Specifying a Base Branch
46+
47+
By default, baselines are generated from `trunk`. To compare against a different branch:
48+
49+
```bash
50+
npm run test:visual -- some-other-branch
51+
```
52+
53+
### Manual 2-Step Workflow
54+
55+
You can also generate baselines and compare manually:
56+
57+
1. Check out the base branch (e.g. trunk).
58+
2. Run `npx playwright test --config tests/visual-regression/playwright.config.js --update-snapshots`
959
3. Check out the feature branch to be tested.
10-
4. Run `npm run test:visual` again. If any tests fail, the diff images can be found in `artifacts/`
60+
4. Run `npx playwright test --config tests/visual-regression/playwright.config.js`
61+
62+
## Impact Summary
63+
64+
After each run, a summary is printed to the terminal showing which pages changed and which remained unchanged:
65+
66+
```
67+
────────────────────────────────────────────────
68+
Visual Impact Summary
69+
────────────────────────────────────────────────
70+
Changed: 27 of 35 pages
71+
Unchanged: 8 of 35 pages
72+
73+
Changed:
74+
Dashboard, All Posts, Categories, Tags, ...
75+
76+
Unchanged:
77+
Login, Media Settings, ...
78+
────────────────────────────────────────────────
79+
```
1180

81+
The HTML report also opens automatically with side-by-side visual diffs for detailed inspection.

‎tests/visual-regression/compare-branches.sh‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/bin/bash
2+
set -e
3+
4+
SKIP_BASELINES=false
5+
BASELINE_BRANCH="trunk"
6+
7+
# Parse arguments.
8+
for arg in "$@"; do
9+
case "$arg" in
10+
--skip-baselines)
11+
SKIP_BASELINES=true
12+
;;
13+
*)
14+
BASELINE_BRANCH="$arg"
15+
;;
16+
esac
17+
done
18+
19+
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
20+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
21+
CONFIG="$SCRIPT_DIR/playwright.config.js"
22+
23+
if [ "$CURRENT_BRANCH" = "$BASELINE_BRANCH" ]; then
24+
echo "Error: Already on $BASELINE_BRANCH. Checkout a feature branch first."
25+
exit 1
26+
fi
27+
28+
if [ "$SKIP_BASELINES" = true ]; then
29+
# Verify baselines exist before skipping regeneration.
30+
SNAPSHOT_DIR="$SCRIPT_DIR/specs/__snapshots__"
31+
if [ ! -d "$SNAPSHOT_DIR" ] || [ -z "$(ls -A "$SNAPSHOT_DIR" 2>/dev/null)" ]; then
32+
echo "Error: No baselines found. Run 'npm run test:visual' first to generate them."
33+
exit 1
34+
fi
35+
36+
echo "Skipping baseline generation (reusing existing baselines)..."
37+
echo "Ensuring Playwright browsers are installed..."
38+
npx playwright install --with-deps chromium
39+
40+
echo "Comparing against existing baselines..."
41+
npx playwright test --config "$CONFIG" || true
42+
else
43+
if ! git diff-index --quiet HEAD --; then
44+
echo "Error: Uncommitted changes detected. Please commit or stash before running."
45+
exit 1
46+
fi
47+
48+
echo "Ensuring Playwright browsers are installed..."
49+
npx playwright install --with-deps chromium
50+
51+
echo "Generating baselines from $BASELINE_BRANCH..."
52+
git checkout "$BASELINE_BRANCH"
53+
npx playwright test --config "$CONFIG" --update-snapshots
54+
55+
echo "Comparing against $CURRENT_BRANCH..."
56+
git checkout "$CURRENT_BRANCH"
57+
npx playwright test --config "$CONFIG" || true
58+
fi

‎tests/visual-regression/playwright.config.js‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,25 @@ process.env.STORAGE_STATE_PATH ??= path.join(
1515
'storage-states/admin.json'
1616
);
1717

18+
const reporter = [
19+
[ 'list' ],
20+
[
21+
'html',
22+
{
23+
open: process.env.CI ? 'never' : 'always',
24+
outputFolder: path.join(
25+
process.env.WP_ARTIFACTS_PATH,
26+
'visual-report'
27+
),
28+
},
29+
],
30+
[ './reporters/impact-summary.js' ],
31+
];
32+
1833
const config = defineConfig( {
1934
...baseConfig,
2035
globalSetup: undefined,
36+
reporter,
2137
webServer: {
2238
...baseConfig.webServer,
2339
command: 'npm run env:start',

‎tests/visual-regression/reporters/impact-summary.js‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/**
2+
* Impact Summary Reporter
3+
*
4+
* A custom Playwright reporter that prints a visual impact summary
5+
* at the end of the test run, showing which pages changed and which
6+
* remained unchanged.
7+
*/
8+
class ImpactSummaryReporter {
9+
constructor() {
10+
this.changed = [];
11+
this.unchanged = [];
12+
}
13+
14+
onTestEnd( test, result ) {
15+
const name = test.title;
16+
17+
if ( result.status === 'passed' ) {
18+
this.unchanged.push( name );
19+
} else {
20+
this.changed.push( name );
21+
}
22+
}
23+
24+
onEnd() {
25+
const total = this.changed.length + this.unchanged.length;
26+
27+
if ( total === 0 ) {
28+
return;
29+
}
30+
31+
const separator = '─'.repeat( 48 );
32+
33+
console.log( '' );
34+
console.log( separator );
35+
console.log( 'Visual Impact Summary' );
36+
console.log( separator );
37+
console.log(
38+
`Changed: ${ this.changed.length } of ${ total } pages`
39+
);
40+
console.log(
41+
`Unchanged: ${ this.unchanged.length } of ${ total } pages`
42+
);
43+
44+
if ( this.changed.length > 0 ) {
45+
console.log( '' );
46+
console.log( 'Changed:' );
47+
console.log( ` ${ this.changed.join( ', ' ) }` );
48+
}
49+
50+
if ( this.unchanged.length > 0 ) {
51+
console.log( '' );
52+
console.log( 'Unchanged:' );
53+
console.log( ` ${ this.unchanged.join( ', ' ) }` );
54+
}
55+
56+
console.log( separator );
57+
console.log( '' );
58+
}
59+
}
60+
61+
module.exports = ImpactSummaryReporter;

‎tests/visual-regression/specs/visual-snapshots.test.js‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,4 +163,102 @@ test.describe( 'Admin Visual Snapshots', () => {
163163
mask: elementsToHide.map( ( selector ) => page.locator( selector ) ),
164164
});
165165
} );
166+
167+
test( 'Dashboard', async ({ admin, page }) => {
168+
await admin.visitAdminPage( '/index.php' );
169+
await expect( page ).toHaveScreenshot( 'Dashboard.png', {
170+
mask: elementsToHide.map( ( selector ) => page.locator( selector ) ),
171+
});
172+
} );
173+
174+
test( 'Themes', async ({ admin, page }) => {
175+
await admin.visitAdminPage( '/themes.php' );
176+
await expect( page ).toHaveScreenshot( 'Themes.png', {
177+
mask: [
178+
...elementsToHide,
179+
'.theme-screenshot img',
180+
].map( ( selector ) => page.locator( selector ) ),
181+
});
182+
} );
183+
184+
test( 'General Settings', async ({ admin, page }) => {
185+
await admin.visitAdminPage( '/options-general.php' );
186+
await expect( page ).toHaveScreenshot( 'General Settings.png', {
187+
mask: elementsToHide.map( ( selector ) => page.locator( selector ) ),
188+
});
189+
} );
190+
191+
test( 'Writing Settings', async ({ admin, page }) => {
192+
await admin.visitAdminPage( '/options-writing.php' );
193+
await expect( page ).toHaveScreenshot( 'Writing Settings.png', {
194+
mask: elementsToHide.map( ( selector ) => page.locator( selector ) ),
195+
});
196+
} );
197+
198+
test( 'Permalink Settings', async ({ admin, page }) => {
199+
await admin.visitAdminPage( '/options-permalink.php' );
200+
await expect( page ).toHaveScreenshot( 'Permalink Settings.png', {
201+
mask: elementsToHide.map( ( selector ) => page.locator( selector ) ),
202+
});
203+
} );
204+
205+
test( 'Add New Post', async ({ admin, page }) => {
206+
await admin.visitAdminPage( '/post-new.php' );
207+
await expect( page ).toHaveScreenshot( 'Add New Post.png', {
208+
mask: [
209+
...elementsToHide,
210+
'#wp-content-editor-container',
211+
].map( ( selector ) => page.locator( selector ) ),
212+
});
213+
} );
214+
215+
test( 'Edit Post', async ({ admin, page, requestUtils }) => {
216+
const post = await requestUtils.rest( {
217+
method: 'POST',
218+
path: '/wp/v2/posts',
219+
data: {
220+
title: 'Visual Regression Test Post',
221+
content: 'Test content for visual regression.',
222+
status: 'publish',
223+
},
224+
} );
225+
226+
await admin.visitAdminPage( '/post.php', `post=${ post.id }&action=edit` );
227+
await expect( page ).toHaveScreenshot( 'Edit Post.png', {
228+
mask: [
229+
...elementsToHide,
230+
'#wp-content-editor-container',
231+
].map( ( selector ) => page.locator( selector ) ),
232+
});
233+
} );
234+
235+
test( 'Site Health', async ({ admin, page }) => {
236+
await admin.visitAdminPage( '/site-health.php' );
237+
await expect( page ).toHaveScreenshot( 'Site Health.png', {
238+
mask: [
239+
...elementsToHide,
240+
'.site-health-issues .health-check-accordion',
241+
].map( ( selector ) => page.locator( selector ) ),
242+
});
243+
} );
244+
245+
test( 'Updates', async ({ admin, page }) => {
246+
await admin.visitAdminPage( '/update-core.php' );
247+
await expect( page ).toHaveScreenshot( 'Updates.png', {
248+
mask: [
249+
...elementsToHide,
250+
'form.upgrade',
251+
'.last-checked',
252+
].map( ( selector ) => page.locator( selector ) ),
253+
});
254+
} );
255+
} );
256+
257+
test.describe( 'Unauthenticated Visual Snapshots', () => {
258+
test.use( { storageState: {} } );
259+
260+
test( 'Login', async ({ page }) => {
261+
await page.goto( '/wp-login.php' );
262+
await expect( page ).toHaveScreenshot( 'Login.png' );
263+
} );
166264
} );

0 commit comments

Comments
 (0)