feat(browser): introduce `toMatchScreenshot` for Visual Regression Testing #8041

macarie · 2025-05-27T19:23:03Z

Description

This PR introduces initial support for Visual Regression Testing for Vitest via a new toMatchScreenshot assertion.

Related issue: #6265

In this initial iteration:

The feature supports PNG screenshots and uses pixelmatch as the comparator.
The architecture is extensible: new comparators or codecs can be added easily, as long as they implement the expected interface.
Comparators and codecs can be asynchronous.
Screenshot comparison is executed in Node as a browser command, which allows for future adoption of native codecs or comparators.

The logic to get a stable screenshot follows Playwright's approach (with some differences):

Uses as baseline an optional reference screenshot or captures a new screenshot.
Takes a screenshot and compares it to the baseline.
If they match, the page is considered stable and the function returns.
If not, it continues with the latest screenshot as the baseline.
Repeats until stability is reached or a timeout is hit.

The command has 6 possible outcomes:

Outcome	Description	Result
`#01`	couldn't get a stable screenshot within timeout	`fail`
`#02`	stable screenshot taken, but no reference exists and update is not allowed	`fail`
`#03`	stable screenshot taken, no reference exists, but updates are allowed	`fail`
`#04`	stable screenshot taken on first try and matches reference	`pass`
`#05`	stable screenshot taken matches reference after retries	`pass`
`#06`	stable screenshot taken doesn't match reference (fallback case)	`fail`

The client matcher expects an Element or Locator along with:

An optional name for the screenshot
An optional options object that includes:
- Screenshot options, excluding conflicting ones
- Comparator name and options
- Timeout value

To-do

Fix screenshot paths and naming
Add global config for matcher, including resolveScreenshotPath and resolveDiffPath functions
Handle default config in one place
Use annotation API (feat: annotation API #7953) to show screenshot paths
Support thresholds in pixelmatch comparator
Add documentation
Write test cases

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
Ideally, include a test that fails without this PR but passes with it.
Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

Run the tests with pnpm test:ci.

Documentation

If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

netlify · 2025-05-27T19:24:49Z

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`32c1313`
🔍 Latest deploy log	https://app.netlify.com/projects/vitest-dev/deploys/6876dadf15cba60008880ad6
😎 Deploy Preview	https://deploy-preview-8041--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

sheremet-va

Looks very good overall, just some code style issues

packages/browser/screenshot.d.ts

packages/browser/src/client/tester/context.ts

packages/browser/src/client/tester/expect/toMatchScreenshot.ts

packages/browser/src/node/commands/screenshotMatcher/index.ts

macarie · 2025-05-29T23:03:18Z

I moved the types around to what starts to hopefully make some sense. I created a shared folder for the types used in both browser and Node environments.

I've also rebased and resolved the conflicts from main.

macarie · 2025-05-29T23:06:40Z

packages/browser/jest-dom.d.ts

@@ -1,6 +1,7 @@
 // Disable automatic exports.

 import { ARIARole } from './aria-role.ts'
+import { ComparatorRegistry, ScreenshotMatcherOptions } from './context.js'


I know this creates a circular dependency between this file and context.d.ts, was wondering if it's fine tho.

To fix it I would have to move ScreenshotOptions out of context.d.ts and, as it's using Locator, at least half of the other types declared in it 😅

If it works, it works 😄

packages/vitest/src/node/types/browser.ts

packages/browser/src/node/commands/screenshotMatcher/utils.ts

packages/browser/src/client/tester/expect/toMatchScreenshot.ts

packages/browser/context.d.ts

macarie · 2025-06-03T22:51:25Z

packages/vitest/src/node/config/resolveConfig.ts

@@ -56,7 +56,7 @@ function parseInspector(inspect: string | undefined | boolean | number) {
  return { host, port: Number(port) || defaultInspectPort }
 }

-export function resolveApiServerConfig<Options extends ApiConfig & UserConfig>(
+export function resolveApiServerConfig<Options extends ApiConfig & Omit<UserConfig, 'expect'>>(


This is provided with the browser object, the new browser.expect property conflicts with expect. I noticed options.expect is not used tho, so instead of removing it at runtime I removed it from the type.

macarie · 2025-06-03T22:58:12Z

packages/vitest/src/node/types/browser.ts

+    toMatchScreenshot?: {
+      [ComparatorName in keyof ToMatchScreenshotComparators]:
+      {
+        /**
+         * The name of the comparator to use for visual diffing.
+         *
+         * @defaultValue `'pixelmatch'`
+         */
+        comparatorName?: ComparatorName
+        comparatorOptions?: ToMatchScreenshotComparators[ComparatorName]
+      }
+    }[keyof ToMatchScreenshotComparators] & ToMatchScreenshotOptions


This type is a bit chaotic, but without generics the only solution is to create discriminated unions.

The interface is compatible with ScreenshotComparatorRegistry, so I used that to extend it in the providers. In the future this will allow to add more comparators and have them typed correctly in the config by augmenting ScreenshotComparatorRegistry.

sheremet-va

Code looks good to me, but we need a lot more tests for this feature

packages/browser/context.d.ts

sheremet-va

Code looks good to me, but we need a lot more tests for this feature

test/browser/fixtures/expect-dom/toMatchScreenshot.test.ts

macarie · 2025-06-08T22:28:27Z

I changed the testing approach: instead of relying on pre-generated screenshots, the tests now create them dynamically during execution.

This makes the tests more stable and avoids failures across different OSes, browser versions, or rendering environments. All artifacts are cleaned up automatically after the tests run.

~~One thing still missing is a test for watch mode. I ran into some issues with it and will take another shot at getting it to work tomorrow.~~ ➡️ Done.

docs/guide/browser/assertion-api.md

macarie · 2025-06-11T22:38:29Z

packages/browser/src/node/commands/screenshotMatcher/index.ts

+  while (signal.aborted === false) {
+    if (decodedBaseline === null) {
+      decodedBaseline = takeDecodedScreenshot(screenshotArgument)
+    }
+
+    const [image1, image2] = await Promise.all([
+      decodedBaseline,
+      takeDecodedScreenshot(screenshotArgument),
+    ])
+
+    const comparatorResult = (await comparator(
+      image1,
+      image2,
+      { ...comparatorOptions, createDiff: false },
+    )).pass
+
+    decodedBaseline = image2
+
+    if (comparatorResult) {
+      break
+    }
+
+    retries += 1
+  }


Playwright uses a backoff mechanism based on predefined timeouts ([0, 100, 250, 500], fallbacks to 1_000) when retrying.

I'm not sure if I should implement that, if each tests waits 100ms to take a screenshot again, it adds up.

Right now, it's surprisingly hard not to get a stable screenshot in a few retries, but this doesn't mean the timing is consistent across runs. When using Playwright as a provider, consistency should not be an issue because they allow disabling animations. With WebdriverIO I don't know as I have no experience with it.

macarie · 2025-07-15T22:55:46Z

Let me know if you don't want the merge commit in the PR's history, I will rebase and remove it once we're done with the review.

Force-pushing after a rebase moves comments up in the "load more" hidden section every time I update from main 😄

sheremet-va reviewed May 27, 2025

View reviewed changes

macarie force-pushed the feat/expect-locator-to-have-screenshot branch 2 times, most recently from 6a8e1ab to d5848f0 Compare May 29, 2025 23:00

macarie commented May 29, 2025

View reviewed changes

macarie commented May 31, 2025

View reviewed changes

packages/vitest/src/node/types/browser.ts Show resolved Hide resolved

macarie commented May 31, 2025

View reviewed changes

packages/browser/src/node/commands/screenshotMatcher/utils.ts Show resolved Hide resolved

sheremet-va reviewed May 31, 2025

View reviewed changes

packages/browser/src/node/commands/screenshotMatcher/utils.ts Outdated Show resolved Hide resolved

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from dbd7baa to 2412183 Compare June 1, 2025 02:03

macarie mentioned this pull request Jun 1, 2025

Support visual regression testing in the browser #6265

Open

4 tasks

macarie force-pushed the feat/expect-locator-to-have-screenshot branch 2 times, most recently from 79afa30 to addc827 Compare June 2, 2025 18:40

sheremet-va reviewed Jun 3, 2025

View reviewed changes

packages/browser/src/client/tester/expect/toMatchScreenshot.ts Outdated Show resolved Hide resolved

sheremet-va reviewed Jun 3, 2025

View reviewed changes

packages/browser/context.d.ts Outdated Show resolved Hide resolved

macarie commented Jun 3, 2025

View reviewed changes

sheremet-va reviewed Jun 4, 2025

View reviewed changes

packages/browser/context.d.ts Outdated Show resolved Hide resolved

sheremet-va requested changes Jun 4, 2025

View reviewed changes

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from ff2c9ad to 8649b09 Compare June 4, 2025 22:33

sheremet-va reviewed Jun 5, 2025

View reviewed changes

test/browser/fixtures/expect-dom/toMatchScreenshot.test.ts Outdated Show resolved Hide resolved

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from 7fab222 to 682bbd1 Compare June 8, 2025 22:17

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from ab114c5 to 246d7c0 Compare June 11, 2025 22:09

macarie commented Jun 11, 2025

View reviewed changes

docs/guide/browser/assertion-api.md Outdated Show resolved Hide resolved

macarie commented Jun 11, 2025

View reviewed changes

macarie force-pushed the feat/expect-locator-to-have-screenshot branch 2 times, most recently from c611092 to e2776e9 Compare June 15, 2025 21:06

sheremet-va added the cr-tracked label Jun 17, 2025

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from d900731 to c7c6219 Compare June 17, 2025 22:17

macarie added 24 commits July 14, 2025 23:13

docs: browser.expect.toMatchScreenshot global config first draft

6f0e009

chore: fix links to docs

6565cc0

chore: fix failing docs deployment

f0198c3

test: try improving stability of stable screenshot capture test

3df246a

test: try getting Windows paths from snapshots

9ac7cde

chore: add JSDoc to toMatchScreenshot Jest assertion

a7eab9f

fix: change default screenshot scale to "device"

c19b4cd

feat: append platform to screenshot name

d8f0539

docs: add version near new feature

211174c

fix: change "screenshot" to "image" as diffs are not screenshots

6553353

docs: create visual regression testing guide

1926a7e

docs: mark toMatchScreenshot as experimental

eb7eeb7

test: automatic screenshot names

63ff393

test: use runIf

dced99f

test: create tests in watch mode

3971cdb

test: run with CLI helper

677d818

test: fix breaking changes after rebase

05bff66

refactor: do not use access before reading the screenshot file

b532d7b

fix: screenshot path on Windows

f639a94

test: remove atime from fs stats

d6a858a

docs: add section about running on GitHub Actions

c7a271f

docs: explain Microsot Playwright Testing usage and split in tabs

5f9f473

docs: fix config example

4404b7d

chore: apply review suggestions

7476ee6

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from 0600986 to 7476ee6 Compare July 14, 2025 22:14

macarie added 2 commits July 15, 2025 00:35

chore: fix lint issues

cffe8cf

chore: apply review suggestions

71a616e

macarie force-pushed the feat/expect-locator-to-have-screenshot branch from f84095c to 71a616e Compare July 15, 2025 22:47

Merge origin/main into feat/expect-locator-to-have-screenshot

32c1313

Uh oh!

feat(browser): introduce toMatchScreenshot for Visual Regression Testing #8041

Are you sure you want to change the base?

feat(browser): introduce toMatchScreenshot for Visual Regression Testing #8041

Conversation

macarie commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

To-do

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

Tests

Documentation

Changesets

Uh oh!

netlify bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vitest-dev ready!

Uh oh!

sheremet-va left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macarie commented May 29, 2025

Uh oh!

macarie May 29, 2025

Choose a reason for hiding this comment

Uh oh!

sheremet-va Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macarie Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

macarie Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

sheremet-va left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sheremet-va left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

macarie commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

macarie Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

macarie commented Jul 15, 2025

Uh oh!

Uh oh!

feat(browser): introduce `toMatchScreenshot` for Visual Regression Testing #8041

feat(browser): introduce `toMatchScreenshot` for Visual Regression Testing #8041

macarie commented May 27, 2025 •

edited

Loading

netlify bot commented May 27, 2025 •

edited

Loading

macarie commented Jun 8, 2025 •

edited

Loading