Contract and visual regression tests

Two bugs no other test caught

Picture a travel booking site on a Friday afternoon. Two teams ship small changes. Both changes pass every unit test, every integration test, and the five end-to-end tests that run on each pull request. By Monday, both have hurt real users.

The first change is on the backend. The bookings team renames the field price to totalPrice in the GET /bookings/:id response, and updates its own tests. The web app still reads booking.price, gets undefined, and shows "NaN" on the checkout page. Its tests pass too, because they use a hand-written fake response that still has price.

The second change is in the shared design system. A developer changes the default line-height of the Text component from 1.5 to 1.4 to tighten a marketing page. Every component test passes: the text is there, the roles are right, the clicks work.

But on the checkout page, the price line now sits 3 pixels higher, and the "Pay now" button wraps its label onto two lines in German. Nobody looked at the German page.

Why the usual tests miss them

Both bugs live at an edge, where one team's output becomes another team's input: the API between backend and frontend, and the screen between your CSS and the user's eyes. Unit and integration tests fake the other side. End-to-end tests cross the edges, but they are few, and they check text and behavior, not pixels.

• A fake drifts from the real thing. A test that mocks the API checks your code against your memory of the API. When the real API changes, the fake does not.
• Assertions do not see layout. expect(button).toBeVisible() passes when the button overlaps the footer or has white text on white. DOM tests read the tree, not the rendered picture.
• Nobody checks every combination. 120 components, times 3 themes, 4 viewports, and 2 text directions, is 2,880 screens. No person looks at that on every pull request.

Two families of tests, one idea

This lesson covers two families of tests that guard those edges. They share one idea: write down what the other side expects, as a file in version control, and fail the build when a change breaks it.

• Contract and schema tests guard the API edge. A schema check compares the new API description (an OpenAPI file or a GraphQL schema) with the old one and flags breaking changes. A consumer-driven contract goes further: each client records the exact requests it sends and the fields it reads, and the provider proves it still serves them.
• Visual regression tests guard the screen edge. They screenshot a page or component, compare it with an approved baseline image, and show a person the difference to decide whether it was meant.

The [API contracts and errors](/system-design/concepts/api-contracts-and-errors) lesson covered the basics: what a contract is, a first Pact test, can-i-deploy, and runtime validation with zod. This lesson goes further: running these tests across teams and in CI, keeping them fast and quiet, reviewing their output, and paying for them. It closes the testing module, after [testing strategy](/system-design/concepts/testing-strategy), [unit and integration tests](/system-design/concepts/unit-and-integration-tests), and [end-to-end tests](/system-design/concepts/e2e-tests).

The mental model: a recorded expectation and a judge

Think of a tailor who keeps a paper pattern of a customer's suit. Every new suit is laid on the pattern before it leaves the shop, so a sleeve 2 cm short shows at once, with no customer in the room. Contract and visual tests work the same way: the pattern is a file, and the build lays every change on it.

Every test in this lesson has the same three parts, whatever the tool:

1. A recorded expectation. A file that says what is correct today: the published schema or the consumer's contract file for an API, the baseline screenshot for a screen. It lives in git or in a service, with a version and an owner.
2. A comparison. A program that checks the new output against the recorded one: oasdiff compares two OpenAPI files, a Pact verifier replays requests against the real API, a pixel diff compares two images.
3. A judge. Something that decides whether a difference is allowed. For schemas, rules decide (removing a field is breaking, adding one is not). For screenshots, a person decides, because only a person knows if a 3 pixel shift was the goal or a bug.

Rules judge contracts, people judge pixels

This is the biggest difference between the two families, and it drives how you run them. A machine can judge a schema change with near certainty: if a field the client reads is gone, the change is breaking. So contract checks can block a merge on their own.

A screenshot difference cannot be judged by a machine alone. A new brand color is a huge diff and a correct change; a 1 pixel gap between a label and its input is a tiny diff and maybe a bug. So visual tests open a review, and a person approves or rejects each change. The approved image becomes the new baseline.

Schema checks in CI

A schema check is the cheapest contract test there is. Most APIs already publish a description of themselves: an OpenAPI file for a REST API, or the schema of a GraphQL API. A schema check compares the description on the pull request with the one on the main branch, lists every change, and fails the build if any change would break an existing client.

It needs no browser, no running server, and no work from the client teams, and it runs in seconds. That makes it the first thing to add, before consumer-driven contracts.

What counts as breaking

The rule is about direction. A client sends requests and reads responses, so a change is breaking if it asks more of requests or promises less in responses. Making requests looser or responses richer is safe.

The enum row is the tricky one. Adding "refunded" to a status enum looks safe, but a client with switch (status) and no default branch will render nothing for that booking. Some tools, such as GraphQL Inspector, put this in a middle class, "dangerous", that shows up in review without blocking.

OpenAPI diff in a pull request

For REST, an open source tool such as oasdiff compares two OpenAPI files and sorts each change into errors (definitely breaking), warnings (possibly breaking), and info. A CI job takes the main branch's file, generates the new one, and runs the diff. The job fails on errors and prints the warnings into the pull request.

# CI step on every pull request to the API repo
git show origin/main:openapi.yaml > base.yaml
npm run generate:openapi        # writes openapi.yaml from the code

oasdiff breaking base.yaml openapi.yaml --fail-on ERR

# example output
# error  [response-property-removed] at openapi.yaml
#   in API GET /bookings/{id}
#   removed the required property 'price' from the response
#   with the '200' status

Two details make this work. The schema must be generated from the code (or the code from the schema): a hand-edited YAML file drifts and gives false comfort. And the frontend must consume the same file, with types generated by a tool such as openapi-typescript, so its build breaks on the same change.

GraphQL schema checks with real usage

GraphQL has an advantage: every client query lists exactly the fields it reads. So a GraphQL schema check can ask more than "is this change breaking in theory?". It can ask "does any real client query use the field I am removing?".

• Static diff. A tool such as GraphQL Inspector compares two schema files and labels each change breaking, dangerous, or safe.
• Diff against known operations. Collect every query the frontend apps ship (many teams extract them at build time as persisted queries) and check the new schema against that list. A field that no shipped query reads can be removed, even though the static diff calls it breaking.
• Diff against traffic. Hosted platforms such as Apollo GraphOS check a schema change against the operations seen in production over a recent window. This catches old mobile app versions still in use, which a list from the current code would miss.

Making a breaking change safely

Sometimes you must rename price to totalPrice. The check does not forbid that; it forces you to do it in steps, a pattern called expand and contract.

1. Expand. Add totalPrice next to price and mark price as deprecated: true in the schema. The check passes, because adding is safe.
2. Migrate. Every client switches to totalPrice and ships. Usage data or the contract broker shows when nobody reads price any more.
3. Contract. Remove price. The check flags it as breaking, so you add an explicit, reviewed exception (an ignore file or an approval step, depending on the tool) that links to the usage data.

Consumer-driven contracts across teams

A schema check knows what the API promises, but not what each client uses. A consumer-driven contract adds that: each client records the requests it makes and the fields it reads, and the provider proves on every change that it still serves them. The [API contracts lesson](/system-design/concepts/api-contracts-and-errors) showed one Pact test; this chapter runs it across five teams without it becoming a chore.

The cast

• Consumers: the web app, the iOS app, the Android app, a backend-for-frontend, each with its own repo and release schedule.
• Provider: the bookings API. One repo, deploys several times a day.
• The broker: a shared service (the open source Pact Broker, or the hosted PactFlow) that stores every contract, every verification result, and which version of each app is deployed where.

The loop, step by step

1. The consumer records. The web app's tests call a Pact mock server, and each test records one interaction: the request, a provider state ("booking 42 exists"), and the minimal response shape it needs.
2. The consumer publishes. CI uploads the contract to the broker, tagged with the git commit and branch: web-app version a1b2c3, branch main.
3. The broker triggers the provider. If the contract changed, a broker webhook starts a verification job in the provider's CI.
4. The provider verifies. The provider's CI starts the real API with a test database, replays each request, checks each response, and publishes the result: bookings-api f9e8d7 verifies web-app a1b2c3.
5. Both sides ask before deploying. Each deploy job runs can-i-deploy, which checks the verification results against the versions already in that environment. If any pair is unverified or failed, the deploy stops.
6. Both sides record deployments. After a deploy, the job tells the broker that web-app a1b2c3 is now in production, so the next can-i-deploy checks against the right versions.

# in the web app's deploy job, before deploying
pact-broker can-i-deploy \
  --pacticipant web-app --version $GIT_SHA \
  --to-environment production

# after a successful deploy
pact-broker record-deployment \
  --pacticipant web-app --version $GIT_SHA \
  --environment production

Provider states: the part that takes work

A recorded interaction says "given booking 42 exists, GET /bookings/42 returns a booking with a numeric price". The provider must be able to make that true on demand. So the provider writes a small state handler for each named state: a function that seeds the test database with booking 42.

This is where most of the cost lives: ten consumers can name 60 states between them. Agree on a short shared list ("a confirmed booking exists", "the user has no bookings") so the provider team is not writing seed functions all week.

When a consumer needs something new

Say the web app starts reading a new field, cancellationDeadline, that the API has not built yet. The new contract would fail provider verification and block the API team's builds, which is backwards: the API team did nothing wrong.

Pact brokers handle this with pending pacts, which the provider turns on in its verification settings. A contract the provider has never passed is marked pending, so its failure is reported but does not fail the provider's build. Once the provider passes it, it becomes a normal, blocking contract. Meanwhile, can-i-deploy still stops the web app from deploying code that reads a field production does not send.

Screenshot diffing: how the comparison works

A visual test does four things: render a page or component in a real browser, take a PNG screenshot, compare it with the approved baseline PNG, and fail if they differ by more than a set amount. The hard part is the comparison, because two screenshots of the same unchanged button are often not identical, pixel for pixel.

Pixel diff with a color threshold

The simplest comparison walks both images pixel by pixel and computes a color distance for each pair. A pixel whose distance is above a per-pixel threshold counts as changed and is painted red in a diff image. If more pixels changed than the pixel budget allows, the test fails.

Popular libraries such as pixelmatch, and Playwright's built-in comparison, measure the distance in the YIQ color space, which weights brightness the way eyes do, so a small change in blue counts for less than the same change in green. Playwright's toHaveScreenshot takes the two knobs directly. threshold runs from 0 (strict) to 1 (lax) and defaults to 0.2. maxDiffPixels or maxDiffPixelRatio sets the budget, which defaults to zero, so by default any changed pixel fails the test.

Anti-aliasing: the main source of noise

Anti-aliasing is how a screen draws a smooth curve or a diagonal edge on a grid of square pixels: the edge pixels get in-between colors, part text and part background. Text gets the most of it. The exact in-between colors depend on the font file, the font rendering engine, the operating system, the GPU, and the screen's pixel ratio.

So the same Button with the same CSS, captured on a Mac laptop and on a Linux CI machine, differs along the edge of every letter. For a 120 by 40 button with a 6 letter label, that can be a few hundred pixels, every one of them noise.

Diff libraries fight this in two ways. pixelmatch tries to detect anti-aliased pixels (an edge pixel with both brighter and darker neighbors) and, by default, leaves them out of the count. And the per-pixel threshold lets small color shifts pass. Both help, and both can hide real bugs: a 1 pixel border that turned a shade darker looks a lot like anti-aliasing.

Perceptual diff

A perceptual diff asks whether a person would notice. Instead of single pixels, it compares small windows of the image for brightness, contrast, and structure; a well-known measure of this kind is SSIM, the structural similarity index. A blurry edge that moved by a fraction of a pixel scores as "the same"; a gap that opened between two boxes scores as "different".

Hosted visual testing services advertise their own perceptual or AI-assisted comparisons. Their methods are not public, so judge them on your own screenshots during a trial, not on their claims.

Kill the noise at the source

Tuning thresholds is the last step, not the first. Every bit of noise you remove at the source lets you keep thresholds strict, and strict thresholds catch real bugs. In order of payoff:

1. One rendering machine. Take every screenshot, locally and in CI, inside the same Docker image, with the same browser build and the same fonts installed. Playwright publishes official images for this. Baselines made on a developer's Mac rarely match a Linux CI run, pixel for pixel.
2. Fonts loaded before capture. Wait for document.fonts.ready, and self-host the font files so a slow font CDN cannot swap in a fallback font mid-capture.
3. Nothing that moves. Turn off CSS animations and transitions, hide the text caret, and pause videos and carousels. Playwright's animations: "disabled" does much of this.
4. Nothing that changes. Fix the clock, seed random values, and serve fixed API data.
5. Mask what you cannot freeze. Paint a solid box over ads, maps, user avatars, and third-party embeds with the mask option. A masked area never counts as changed, so mask as little as you can.

import { test, expect } from "@playwright/test";

test("checkout summary", async ({ page }) => {
  await page.clock.setFixedTime(new Date("2026-03-01T10:00:00Z"));
  await page.goto("/checkout?fixture=two-items");
  await page.evaluate(() => document.fonts.ready);

  await expect(page.getByTestId("summary")).toHaveScreenshot("summary.png", {
    animations: "disabled",
    mask: [page.getByTestId("promo-banner")],
    threshold: 0.2,        // per-pixel color tolerance, 0 to 1
    maxDiffPixels: 20,     // small budget for leftover noise
  });
});

Lab: noise versus a real bug

The lab below shows a diff map of a card after a pull request. The avatar is live data and differs on every run unless masked, and the text edges carry anti-aliasing noise from the CI machine's font hinting. Two scenarios add a real bug: a price moved down 2 pixels, or a bottom border a shade lighter. Tune the settings and watch for a red build with no bug (a false positive) and a green build with a bug (a missed bug).

Storybook-driven tests

Page-level screenshots have a problem: a page mixes 30 components, live data, and layout, so when it differs, you hunt for the cause. Component-level screenshots fix that, but you need a way to render every component in every state, alone, with fixed data. Storybook is the most common tool for that.

A story is a named, rendered state of one component: Button / Primary, Button / Disabled, DatePicker / Range selected in Arabic. Teams write stories anyway, to build and document components, so a design system with 400 stories already has 400 test cases. Storybook-driven testing runs checks on every story, automatically.

Three checks on every story

• Render and interaction tests. Each story renders in a real browser; a story that throws fails. A story can also have a play function that clicks, types, and asserts, like a small integration test you can watch step by step in the Storybook UI.
• Accessibility checks. The accessibility addon runs axe on each story and reports missing labels, low contrast, wrong ARIA, and duplicate ids, as warnings or as failures.
• Visual snapshots. Each story is screenshotted after its play function has run and diffed against its baseline. That is how states like "open dropdown" or "error shown" get covered.

import type { Meta, StoryObj } from "@storybook/react-vite";
import { expect } from "storybook/test";
import { CurrencyInput } from "./CurrencyInput";

const meta = {
  component: CurrencyInput,
  args: { label: "Amount", currency: "EUR" },
} satisfies Meta<typeof CurrencyInput>;
export default meta;
type Story = StoryObj<typeof meta>;

export const TooLarge: Story = {
  play: async ({ canvas, userEvent }) => {
    const input = canvas.getByLabelText("Amount");
    await userEvent.type(input, "1000000");
    await userEvent.tab();
    await expect(input).toHaveAttribute("aria-invalid", "true");
    await expect(canvas.getByRole("alert")).toHaveTextContent(/at most/i);
  },
};

Running stories in CI

Storybook offers two ways to run stories as tests. Since Storybook 9, its docs recommend the Vitest addon for projects built with Vite and describe the older test runner as superseded.

• The Vitest addon (@storybook/addon-vitest). It turns each story into a Vitest test that runs in a real browser through Vitest's browser mode, with Playwright driving Chromium by default. Stories then run in the same command, watch mode, and coverage report as your other unit tests. It needs a Vite-based Storybook framework.
• The test runner (@storybook/test-runner). It visits every story of a running or published Storybook in a Playwright-driven browser, runs its play function, and reports failures, using Jest underneath. Hooks let you add an axe check or a screenshot after each story. It remains the option for Storybooks built with Webpack.

Testing a design system

A design system is the extreme case for both families of tests. Dozens of teams import it, so one bad change ships to every product at once. And its "API" has two faces, both of them contracts: the pixels users see, and the props, types, and CSS tokens developers use.

Three layers of checks

• Visual: every variant, every theme. A Button with 4 variants, 3 sizes, 2 themes, and 2 text directions is 48 screens; forty such components is close to 2,000. Snapshots are the only way that gets covered at all.
• Accessibility: every story. axe on every story, plus play tests for keyboard behavior: does Escape close the menu, does focus return to the trigger. Deque's 2021 study of its own audit data found that automated checks caught about 57 percent of accessibility issues by volume, so screen reader output and much else still need a person.
• API surface: every export. The list of exported components, their prop types, and the CSS custom properties (design tokens) are a contract with every consumer. Removing a prop or renaming a token is a breaking change, exactly like removing an API field.

Generate the matrix, do not hand-write it

Writing 48 Button stories by hand is slow, and they drift. Instead, write one "all variants" story per component that renders a grid of every combination, and snapshot it in each theme. One screenshot per theme covers every variant, and a diff shows at a glance which cells changed.

Add hostile cases to the grid: a label 3 times longer than normal (German text often runs about 30 percent longer than English), right-to-left text, 200 percent zoom, and forced-colors (Windows high contrast) mode. These catch the bugs real products hit and happy-path stories miss.

Guarding the API surface

The pixel tests do not notice when a prop is renamed, because the stories are updated in the same pull request. So the design system needs a schema check of its own, on its TypeScript types.

• An API report. A tool such as Microsoft's API Extractor writes the public type signatures into a text file checked into git. Any pull request that changes the public API changes that file, so the reviewer sees it and can ask "is this a major version?".
• Type tests. Small tests with Vitest's expectTypeOf (or tsd) pin the types consumers rely on: variant accepts "primary", onChange receives a string. A refactor that widens or narrows them fails the test.
• Token diffs. Tokens such as --color-text-muted are read by product CSS the library cannot see. Treat the token list like an API schema: diff it, and treat removals and renames as breaking.
• Semantic versioning with change notes. Removing a prop or token is a major version; tools such as Changesets make each pull request say whether it is a patch, minor, or major change.

Test inside real products before release

The last check borrows the idea of consumer-driven contracts. Before publishing, install the release candidate into two or three real product apps and run their visual and end-to-end suites. The products show the components under real CSS, data, and layouts, which is where a z-index change that hides a product's sticky header shows up.

Baselines, review, and cost

A visual test is only as good as the person who approves its diffs. If reviewing is slow or confusing, people click "approve all" and the test becomes a ritual, so the workflow around baselines matters as much as the diff algorithm.

Where baselines live

• In git, next to the test. Playwright writes baselines into a folder beside each test file, with the project (browser) and platform in the file name, for example summary-chromium-linux.png. Changes show up in the pull request; thousands of PNGs make the repo heavy, so teams often store them with Git LFS.
• In a hosted service. The service keeps a baseline per story and per branch, linked to commits. The repo stays small, and review happens in the service's web UI, which reports a status back to the pull request.

Baselines follow branches

Two pull requests open at the same time can each change the header. If both compare against main, the second to merge shows the first one's change as its own diff, which confuses reviewers. Good setups compare each branch with the baselines of the commit it branched from; when a pull request merges, its approved images become main's new baselines.

Reviewing a diff well

1. Read the pull request first. A diff outside the area it meant to change is the suspicious one.
2. Approve per component, not per build. "Approve all" on 140 changes hides the one that was a bug. A change that big came from a token or base style; sample it as in the design system chapter.
3. Name an owner. The design system team approves changes to shared components; each product team approves its own pages. A diff nobody owns gets approved by whoever is in a hurry.

What it costs

Visual tests cost CI minutes, money (hosted services charge per snapshot), and reviewer time. The snapshot count multiplies fast: stories times browsers times viewports times themes, times every push to a pull request.

Take a design system with 400 stories, checked in 2 browsers, 2 viewports, and 2 themes: 3,200 snapshots per run. With 25 pushes a day across all pull requests, that is 80,000 snapshots a day, or about 1.7 million in a month of 21 working days. At one second each, that is 22 hours of browser time a day. The calculator below lets you try your own numbers.

Three levers bring it down without losing much safety:

• Only what changed. Snapshot only stories whose component files, or their imports, changed in the pull request. Some hosted services do this from the build's dependency graph; with your own setup, a script can map changed files to stories. Run the full set on main every night as a backstop.
• Fewer combinations per pull request. Run one browser, one viewport, and one theme on each push, and the full matrix on main or before a release. Most visual bugs show in every combination.
• Fewer, bigger images. One "all variants" grid per component is one snapshot instead of 48, and one diff to review.

Failure modes and how to spot them

Both families fail quietly: the checks stay green, or go red so often that nobody reads them, and the bugs ship anyway. Here are the common ways that happens:

The most common contract mistake: recording too much

A consumer test that records the whole response, with exact values, turns the contract into a snapshot of the test database. Every harmless provider change, such as a new default name or an extra list item, fails verification, and after a few weeks the provider team asks to turn contracts off.

Record only the fields the consumer code reads, and match on type or format, not exact value: price is any integer, checkIn is a date string, rooms is an array with at least one item of this shape. A contract should fail only when a change would break the consumer.

How teams use them in practice

The most common pattern: product apps keep a few dozen toHaveScreenshot checks on key screens such as checkout and sign-up; design systems run snapshots and axe on every story; large GraphQL APIs check schemas against recent client operations; and teams with mobile apps use Pact with can-i-deploy, because a broken mobile release can take days to replace on users' phones.

Using it in a system design interview

Contract and visual tests rarely get a whole interview. They come up in the last ten minutes, under "how do you ship this safely?", or when the design has a shared API, several clients, or a component library. A sentence or two then shows you have run real systems with many teams.

When to bring it up

• Several clients share an API (web, iOS, Android, partners): schema checks in the API's CI and consumer-driven contracts with a deploy gate.
• A design system or shared component library is part of the design: visual snapshots per story, axe on every story, an API report, and a canary release into one product.
• The UI is the product (checkout, a pricing page, an editor toolbar): a few visual snapshots of key screens in one Docker image.

If you have a whiteboard, draw the edges, not the tools: a box per team, arrows for the API and the shared components, and a short label on each arrow saying which check guards it. That one picture answers "how do teams ship without breaking each other?" faster than a list of tool names.

How deep to go by level