completefrontend
Open lab My progress
THE WEB PERFORMANCE FIELD GUIDE

Performance in continuous integration

Check bundle sizes and Lighthouse on every pull request with size-limit and Lighthouse CI, and set thresholds from the middle of several runs so noise can't fail a good change.

By the end, you can
  • 01
    Check bytes on every pull requestRun size-limit in a GitHub Actions workflow, and compare each pull request's sizes with main's.
  • 02
    Run Lighthouse CICollect several runs, check them with assertions at the right level, and upload the reports.
  • 03
    Set thresholds that don't flakeRead the middle of several runs, leave room above the wobble, compare with main, and test the check on a known slowdown.

A check that failed, then passed

In lesson 39.1, Ravi's team set its first budgets. A size check stopped Ben's carousel before it went live. But someone had to remember to run it. This lesson makes the checks run by themselves, on every change. Then it makes them worth trusting.

First, how the team works. Each change starts on a branch: a copy of the code where one person works without disturbing anyone. Each saved step of that work is a commit. When it's ready, they open a pull request. That's a request to merge the branch into main, the code the site runs. Someone reads it before anyone presses Merge.

Ravi wants a machine to check each pull request too. That's continuous integration, or CI. Each time someone pushes a change, a service builds it on a fresh computer and runs the team's checks. Ravi's team uses GitHub Actions. A file in the code, called a workflow, lists the steps.

Ravi's first workflow has two checks. One is size-limit, from lesson 39.1. The other runs Lighthouse once on the Mint tea page. For its limits, Ravi runs Lighthouse once on main and copies the numbers: no worse than today. The photo appeared at 1,058.1 ms in that run. That's the page's LCP, Largest Contentful Paint, and a millisecond (ms) is a thousandth of a second. Rounded up, the LCP limit became 1,059 ms.

Then Ben opens a pull request. Meera wants a timer under the brewing steps: tap it, and it counts down three minutes. Ben writes it:

Ben's timer.jsJavaScript
export function startTimer(button) {  button.addEventListener("click", () => {    let left = 180;    const tick = setInterval(() => {      left -= 1;      button.textContent = `${Math.floor(left / 60)}:${String(left % 60).padStart(2, "0")}`;      if (left === 0) clearInterval(tick);    }, 1000);  }, { once: true });}
  • Line 1 makes startTimer, which gets the button.
  • Line 2 waits for a tap. Line 9 passes { once: true }, so the listener runs only once.
  • Line 3 starts with 180 seconds. Lines 4 to 8 run once a second.
  • Line 6 writes the time left on the button, like 2:59. Line 7 stops at 0.

Ben adds two lines to app.js: an import, and a call to startTimer. Nothing runs until someone taps the button.

He pushes his branch, and both checks run. The size check passes. The Lighthouse check fails. Ben is puzzled: his timer doesn't even start until a tap. He presses Re-run on the same commit, and this time it passes:

Lighthouse CI's check on the same commit of Ben's pull request, run twice. Attempt 1: ✘ largest-contentful-paint failure for maxNumericValue assertion expected: <=1059 found: 1062.0998 Assertion failed. LCP 1,062.1 ms. Re-run: Checking assertions against 1 URL(s), 1 total run(s) All results processed! LCP 1,053.8 ms.Ben's pull request, the same commit, limit 1,059 msAttempt 1✘ largest-contentful-paintfailure for maxNumericValueassertionexpected: <=1059found: 1062.0998Assertion failed.failed: LCP 1,062.1 msRe-runChecking assertions against1 URL(s), 1 total run(s)All results processed!passed: LCP 1,053.8 ms
Real output from the course's recording. The same commit failed, then passed, with nothing changed in between.
Two attempts at the same commitPop out in the code editor (opens in a new tab)JavaScript
// Lighthouse's LCP on Ben's branch: one run, then a re-run of the same commit. In ms.const limit = 1059;const firstAttempt = 1062.1;const reRun = 1053.8; const gap = (lcp) => (lcp > limit ? `over by ${(lcp - limit).toFixed(1)} ms` : `under by ${(limit - lcp).toFixed(1)} ms`);console.log(`First attempt: ${gap(firstAttempt)}`);console.log(`Re-run: ${gap(reRun)}`);console.log(`Between the two runs: ${(firstAttempt - reRun).toFixed(1)} ms`);
  • Line 2 is the limit Ravi copied. Lines 3 and 4 are the LCPs Lighthouse measured, in ms.
  • Line 6 says how far a number is from the limit, and on which side.
  • It prints First attempt: over by 3.1 ms, Re-run: under by 5.2 ms and Between the two runs: 8.3 ms.

Nothing changed between the two attempts. The same code gave two answers, 8.3 ms apart, and the limit sat between them. So which run was right? Did the timer slow the page down? And what is a check worth if it can say both?

Why it matters: a check that fails at random teaches the team to press Re-run without reading it. Then a real slowdown comes, and nobody looks.

Real-life analogyThe pass in a restaurant kitchen

In a busy restaurant, every plate stops at the pass. That's the counter between the kitchen and the tables. The head chef checks each plate there: the right dish, hot, the right size. Only then does it go out. The checks are quick and the same every time. And a chef who sends back good plates by mistake soon stops being listened to.

In real life: The pass, where every plate stops
In web performance: The workflow, which runs on every pull request
In real life: One plate
In web performance: One pull request
In real life: Quick checks, the same every time
In web performance: size-limit and Lighthouse CI
In real life: Sending back a good plate
In web performance: A false alarm: a failed check on a change that's fine

Here is the whole workflow Ravi ends up with. The rest of the lesson builds it, one step at a time:

The workflow on every pull request, left to right: pull request (Ben pushes his branch), then build (dist/), then size-limit (bytes, exact, one run), then Lighthouse CI (five runs, the middle one), then status (pass or fail on the commit).One workflow, on every pull requestpull requestBen pusheshis branchbuilddist/size-limitbytes, exact,one runLighthouse CIfive runs,the middle onestatuspass or failon the commit
Every push to Ben's branch goes through the same steps on a fresh machine. The bytes check needs one run. The Lighthouse check needs several.
The main idea

Performance in CI means a workflow checks every pull request against the team's budgets before it's merged. Bytes are checked exactly, with one run. Lab timings wobble, so they're checked by the middle of several runs, against a limit with room above the wobble. Then a red check means a real change, and the team can trust it.

For the bigger picture, Performance budgets and monitoring has a lab where a pull request adds a library, and you see which CI checks fail. This lesson goes one question at a time:

  • How do you check bytes on every pull request?
  • How does Lighthouse CI run Lighthouse and check the results?
  • What is a synthetic test, and where does it run?
  • Where do you draw a line that noise can't cross?

How the course recorded it

  • The branches: Ravi's project with three branches. main is the Mint tea page from lesson 39.1, with the carousel loaded later. Then Ben's timer, and Ben's carousel from lesson 39.1, replayed. Each was built with esbuild 0.28.2 and checked with size-limit 14.3.0.
  • Lighthouse CI: version 0.16.1, with Lighthouse 13.6.0 in Chrome 157. The course ran the commands the workflow runs, on one laptop: 4 jobs on each branch, 5 runs in each, the branches taking turns.
  • The checks: Lighthouse CI's own lhci assert on the saved reports. Each run alone, as a one-run job checks it, and each job's five runs together.
  • The comment: size-limit-action's own code for its comment, run on size-limit's numbers for main and each branch.

Bundle size checks on every pull request

Start with the easy check. In the workflow, size-limit runs right after the build, as it did in lesson 39.1. Here is what it printed on Ben's pull request:

size-limit on Ben's pull requestText
$ npx size-limitJavaScript at loadSize limit: 4 kBSize:       3.16 kB brotlied CSS at loadSize limit: 1 kBSize:       436 B brotlied
  • Line 1 is the command the workflow runs.
  • Lines 2 to 4 are the JavaScript check: app.js sends 3.16 kB with Brotli, under its 4 kB limit.
  • Lines 6 to 8 are the CSS check: app.css sends 436 bytes, under 1 kB.

size-limit ended with exit code 0, so the step passed. An exit code is the number a program ends with: 0 means all went well. GitHub's docs say a step passes or fails by its exit code. A step that fails fails the job, and the steps after it don't run.

Notice what Ben didn't need: more runs. Bytes don't wobble. Lighthouse CI shows it well, because it counts bytes in each run too:

Same bytes, different timingsPop out in the code editor (opens in a new tab)JavaScript
// Ben's branch, 20 Lighthouse runs: the JavaScript bytes each counted, and each LCP in ms.const scriptBytes = [  3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920,  3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920, 3920,];const lcps = [  1057.3, 1055.7, 1054.4, 1054.7, 1053.9, 1056.1, 1055.3, 1054.6, 1053.2, 1054.5,  1056.3, 1062.1, 1053.8, 1053.3, 1052.8, 1057.2, 1054.8, 1054.5, 1053.9, 1054.1,]; console.log(`Different byte counts: ${new Set(scriptBytes).size}`);console.log(`Different LCPs: ${new Set(lcps).size}`);
  • Lines 2 to 5 hold the JavaScript bytes Lighthouse counted in each of the 20 runs of Ben's branch. It counts what came over the network, headers included, from Lighthouse CI's own server. So this isn't size-limit's number.
  • Lines 6 to 9 hold each run's LCP, to a tenth of a millisecond.
  • It prints Different byte counts: 1 and Different LCPs: 18.

Twenty runs gave one byte count and 18 different LCPs. Why it matters: a size check can sit close to today's size and still never fail by chance. One run is enough.

Real-life analogyA parcel on the post office scale

At the post office, you put a parcel on the scale: 2.4 kg. Put it on again, and it's still 2.4 kg. Then you ask how long it will take to arrive. The answer is "two to four days". The weight is a fact about the parcel. The trip depends on roads, weather and queues.

In real life: The parcel's weight
In web performance: The bytes of a build
In real life: The same reading every time
In web performance: The same bytes in every run
In real life: "Two to four days"
In web performance: LCP, a little different in each run
In real life: Roads and sorting queues
In web performance: The machine, the browser and what else is running

Compare with main

A pass tells Ben he's under the limit. It doesn't tell him how much he added. size-limit's README points to a GitHub action for that: size-limit-action. It runs size-limit on the pull request and on the branch it will merge into, here main. Then it writes a comment on the pull request:

size-limit-action's comment on Ben's pull requestText
## size-limit report 📦| Path               | Size                || ------------------ | ------------------- || JavaScript at load | 3.08 KB (+2.44% 🔺) || CSS at load        | 436 B (0%)          |
  • Line 1 is the comment's heading.
  • Lines 2 and 3 start a table with two columns: each check's name, and its size.
  • Line 4: the JavaScript grew by 2.44% against main. Line 5: the CSS didn't change.

Ben's timer added 75 bytes, with Brotli. The action writes sizes with a package called bytes, which counts 1,024 bytes as a KB. size-limit counts 1,000. So the same 3,158 bytes show as 3.08 KB here and 3.16 kB above. The action also fails the check when size-limit fails.

Another action, Preact's compressed-size-action, builds the pull request and its base branch too. It reports the change in every JavaScript file in dist/, with gzip unless you pick Brotli. It only reports: it has no limits to fail.

Why it matters: the comment turns a number into a conversation. Ben's +2.44% is a small cost everyone can see. For the replayed carousel, the same table said +1289.43%.

Lighthouse CI: collect, assert, upload

Bytes were the easy part. To see when the photo appears, you need a browser. Lighthouse CI is the Chrome team's tool for running Lighthouse in CI. You met it in lesson 8.5. Its guide sets up a build that runs Lighthouse "on every commit".

One command does it all: lhci autorun. It runs three steps:

  • collect runs Lighthouse on the page, several times, one run after another. numberOfRuns says how many. It's 3 unless you change it.
  • assert checks the reports against your assertions: rules like "LCP at most 1,100 ms". It prints each one that fails, and can fail the step.
  • upload sends the reports somewhere you can open them.

The settings live in a file called lighthouserc.json. Here is Ravi's first try:

Ravi's first lighthouserc.jsonJSON
{  "ci": {    "collect": {      "staticDistDir": "./dist",      "url": ["http://localhost/page.html"],      "numberOfRuns": 1    },    "assert": {      "assertions": {        "categories:performance": ["error", { "minScore": 1 }],        "largest-contentful-paint": ["error", { "maxNumericValue": 1059 }],        "total-blocking-time": ["error", { "maxNumericValue": 0 }],        "cumulative-layout-shift": ["error", { "maxNumericValue": 0 }]      }    },    "upload": { "target": "temporary-public-storage" }  }}
  • Line 4 tells Lighthouse CI to start its own small web server for the built files in dist/. Line 5 names the page. Lighthouse CI puts its server's port into the address.
  • Line 6 asks for one run.
  • Lines 10 to 13 are the assertions, one to a line. Each has a level, "error", and a limit.
  • minScore is the lowest score that passes, from 0 to 1. maxNumericValue is the highest number that passes. For LCP and Total Blocking Time (TBT), it's in ms. TBT adds up how long the main thread was too busy to answer a tap.
  • Line 16 uploads the report to temporary public storage.

Every limit came from that one run on main: a score of 1, which the report shows as 100; LCP 1,059 ms; no blocking time; no layout shift. Here is what the assert step printed on Ben's first attempt:

The assert step on Ben's first attemptText
$ npx lhci assertChecking assertions against 1 URL(s), 1 total run(s) 1 result(s) for http://localhost:PORT/page.html :   ✘  largest-contentful-paint failure for maxNumericValue assertion       Largest Contentful Paint       https://developer.chrome.com/docs/lighthouse/performance/lighthouse-largest-contentful-paint/         expected: <=1059           found: 1062.0998      all values: 1062.0998 Assertion failed. Exiting with status code 1.
  • Line 2 says it checked one page and one run.
  • Line 6 names the audit that failed, and the kind of limit: maxNumericValue. Lines 7 and 8 give the audit's title and its docs.
  • Line 10 is the limit, at most 1,059. Line 11 is what it found. Line 12 lists every run's value; here there was one.
  • Line 14 ends with exit code 1, so the step failed.

Each assertion has one of three levels:

  • error: checked, printed, and a failure fails the step.
  • warn: checked and printed, but the step still passes.
  • off: not checked.

You don't have to write every assertion yourself. A preset is a ready-made set. lighthouse:recommended expects a perfect score on every audit outside performance, and only warns on metrics like LCP. Its source code files them under "Flaky audits".

A status on the pull request

Ravi installs the Lighthouse CI GitHub App and puts its token in the workflow. Then the upload step also posts a status on the commit: a small line in the pull request with a mark and a link to the report. Ben's first attempt got lhci/url/page.html: Failed 1 assertion(s). The re-run got Passed.

The upload step has three places to send reports:

  • temporary-public-storage: free and quick. Anyone with the link can open the report, and it's deleted after 3 days to 5 weeks. It's run by a Google partner, Eris Ventures, not by Google.
  • lhci: your own Lighthouse CI server. It keeps reports private and for longer than a few days, graphs them over time, and shows the difference between two reports.
  • filesystem: saves the reports on the CI machine, for your own scripts.

Why it matters: the status puts the result where Ben already looks, on his pull request. One click opens the report that explains it.

Synthetic tests per pull request

Lighthouse in CI is a synthetic test. Synthetic means made, not natural. A script visits the page in a lab, with the same phone screen, network and CPU settings every time. Real visitors give field data instead. A synthetic test can run before anything goes live, on a page nobody has visited yet. Watching real visits is the next lesson.

Here is the workflow Ravi commits, as .github/workflows/performance.yml:

The workflow on every pull requestYAML
name: Performanceon: pull_request jobs:  checks:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v7        with:          ref: ${{ github.event.pull_request.head.sha }}      - uses: actions/setup-node@v7        with:          node-version: 22      - run: npm ci      - run: npm run build      - run: npx size-limit      - run: npx lhci autorun        env:          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
  • Line 1 names the workflow. Line 2, on: pull_request, runs it when a pull request opens, and when its branch gets new commits.
  • Line 6 asks for a fresh Linux machine from GitHub for the job, called a runner.
  • Lines 8 to 10 fetch the code. A pull request workflow normally tests the branch merged into main. Line 10 picks Ben's own last commit instead, as Lighthouse CI's docs suggest, so its status lands on his commit.
  • Lines 11 to 13 install Node.js 22. Line 14 installs the packages, Lighthouse CI too, at the versions in the lock file, package-lock.json.
  • Line 15 builds the page. Line 16 checks the bytes. Line 17 runs Lighthouse CI.
  • Lines 18 and 19 hand Lighthouse CI the app's token, so it can post the status. It comes from the repository's secrets, GitHub's safe store for passwords and tokens.

Here is how the collect step began in one of Ben's jobs:

The collect step in one jobText
$ npx lhci autorun✅  .lighthouseci/ directory writable✅  Configuration file found✅  Chrome installation foundHealthcheck passed! Started a web server on port PORT...Running Lighthouse 5 time(s) on http://localhost:PORT/page.htmlRun #1...done.Run #2...done.Run #3...done.Run #4...done.Run #5...done.Done running Lighthouse!
  • Lines 2 to 5 check that Lighthouse CI can run: a folder to write in, a settings file, and Chrome.
  • Line 7 starts the server for dist/. Line 8 starts five runs on the page.
  • Lines 9 to 13 are the runs, one after another. Line 14 says they're done.

The machine matters

A runner is someone else's computer. GitHub's ubuntu-latest runner has 4 processors and 16 GB of memory for public repositories, and 2 and 8 GB for private ones. Lighthouse's docs on variability ask for at least 2 dedicated cores, and recommend 4. They also warn: "Free CI environments and 'burstable' instances are typically quite volatile." And never run two Lighthouse runs at once on one machine. Lighthouse CI runs them one after another.

So how much do runs of the same commit wobble? The course ran the workflow's commands in 12 jobs on its laptop. Here is main, where nothing changed:

main's 20 runs against the first limitPop out in the code editor (opens in a new tab)JavaScript
// main, with nothing changed: 4 jobs of 5 Lighthouse runs. LCP in ms, to two decimals.const limit = 1059;const lcps = [  1058.07, 1059.03, 1058.64, 1054.42, 1053.66,  1058.2, 1055.81, 1053.97, 1053.1, 1053.72,  1057.69, 1055.56, 1058.49, 1052.96, 1053.51,  1059.95, 1054.62, 1053.78, 1054.01, 1053.1,]; const over = lcps.filter((lcp) => lcp > limit);console.log(`From ${Math.min(...lcps)} to ${Math.max(...lcps)} ms`);console.log(`Over ${limit}: ${over.length} of ${lcps.length} runs, ${over.join(" and ")}`);
  • Line 2 is Ravi's first limit. Lines 4 to 7 hold all 20 LCPs of main, one job to a row. They have two decimals, because one run was very close.
  • Line 10 keeps the runs over the limit. Lines 11 and 12 print the spread and the runs over.
  • It prints From 1052.96 to 1059.95 ms and Over 1059: 2 of 20 runs, 1059.03 and 1059.95.

main's own runs spread over 7 ms. Two of them crossed Ravi's limit, one by three hundredths of a millisecond. On Ben's branch, 1 run of 20 crossed it: his first attempt. Here is every run, close up and zoomed out:

Every Lighthouse run's LCP, as a tick: 20 on each branch. main 1,053.0 to 1,060.0 ms, Ben's timer 1,052.8 to 1,062.1 ms, the carousel 1,203.4 to 1,387.9 ms. Close up, 3 runs of main and Ben's timer are over the first limit of 1,059 ms. Zoomed out, main and Ben's timer are under the steady limit of 1,100 ms, and every carousel run is over it.Close up, 1,050 to 1,065 msfirst limit, 1,059 msmainBen's timer1,0501,0551,0601,065Zoomed out, 1,000 to 1,400 mssteady limit, 1,100 msmainBen's timerthe carousel1,0001,1001,2001,3001,400
Each tick is one run; taller ticks are over the dashed limit. Close up, the first limit sits inside the wobble, and 3 runs cross it. Zoomed out, the steady limit leaves room above main and Ben's timer, and every carousel run is past it.

Three more things the runs showed:

  • TBT was 0 and the score 100 in all 60 runs. On a page this small, nothing comes near those limits.
  • Lighthouse doesn't time the network for real. It works the load out from the page's requests, as lesson 8.3 showed. That takes most of the network's wobble away. Once in a while, a run still jumps: two carousel runs landed at 1,203.4 and 1,387.9 ms, far from the others.
  • These LCPs are higher than lesson 39.1's Lighthouse numbers. The page also came from a different server: Lighthouse CI's own server spoke HTTP/1.1, and 39.1's spoke HTTP/2. Compare Lighthouse CI's numbers only with Lighthouse CI's.

Why it matters: Ben's failure wasn't about his timer. main itself fails this check about one run in ten.

Thresholds that aren't flaky

Back to Ben's question. His timer didn't slow the page. The check failed because its line sat inside the wobble. A limit copied from one run lands wherever that run landed. Ravi copied a slow run, so only the slowest runs crossed. Had he copied main's fastest run, 1,053 ms, 19 of main's 20 runs would have failed.

A check like that is called flaky: it passes or fails on the same code, by chance.

Real-life analogyA smoke alarm over the toaster

A smoke alarm hangs right over the toaster. Every morning, the toast sets it off. After a week, someone takes the battery out. Then one night there's a real fire, and the alarm stays quiet. Move it a little way off, and it only sounds for real smoke.

In real life: An alarm right over the toaster
In web performance: A limit at one run's exact number
In real life: Toast every morning
In web performance: The normal wobble between runs
In real life: Taking the battery out
In web performance: Pressing Re-run without reading, or turning the check off
In real life: Moving the alarm a little way off
In web performance: A limit with room above the wobble

Read the middle run

Lighthouse CI already runs the page several times. The aggregationMethod option says how an assertion reads those runs:

  • median: the middle value.
  • optimistic: the value most likely to pass. For a maxNumericValue, that's the lowest. It's the default.
  • pessimistic: the value least likely to pass, the highest.
  • median-run: the value from one whole run, the one Lighthouse CI calls most representative. It picks the run closest to the middle in two numbers at once: when the first text or image appeared, and when the page became interactive.

Here are the five runs of the job that held Ben's first attempt:

One job's five runs, read three waysPop out in the code editor (opens in a new tab)JavaScript
// The job with Ben's first attempt: its 5 LCPs, in ms.const lcps = [1056.3, 1062.1, 1053.8, 1053.3, 1052.8];const sorted = [...lcps].sort((a, b) => a - b); console.log(`optimistic, the best run: ${sorted[0]}`);console.log(`median, the middle run: ${sorted[2]}`);console.log(`pessimistic, the worst run: ${sorted[4]}`);
  • Line 2 holds the five LCPs, in the order they ran. Line 3 sorts a copy, fastest first.
  • Lines 5 to 7 print the first, the third and the last.
  • It prints optimistic, the best run: 1052.8, median, the middle run: 1053.8 and pessimistic, the worst run: 1062.1.
Real-life analogyFive judges at a diving contest

At a diving contest, five judges score each dive. One judge is always kind, another always harsh. Take only the kind one, and every dive looks great. Take only the harsh one, and every dive looks poor. So the score drops the highest and the lowest marks, and keeps the middle.

In real life: Five judges
In web performance: Five Lighthouse runs
In real life: Only the kindest judge
In web performance: optimistic: the best run
In real life: Only the harshest judge
In web performance: pessimistic: the worst run
In real life: Dropping the highest and lowest
In web performance: median: the middle run

Try each way of reading the runs against Ravi's first limit:

Playground: one run, or five runs read three ways
One run, against 1,059 ms
  1. main2 of 20 runs over
  2. Ben's timer1 of 20 runs over
  3. The carousel, replayed20 of 20 runs over
Try it yourself
3 false alarms

Ravi's first try: each run is a job of its own, checked alone. Against Ravi's first limit of 1,059 ms, main and Ben's timer, where nothing that matters changed, failed 3 runs. The carousel failed 20 of 20.

From the course's recording in Chrome 157: 5 Lighthouse runs in each job, 4 jobs on each branch, the branches taking turns.

Leave room above the wobble

The middle run alone would have kept Ben's check green. But middle runs wobble too: main's four ranged from 1,054.0 to 1,058.1 ms. So Ravi also moves the line. He uses lesson 39.1's rule: the slowest run he has, rounded up to the next 100 ms. main's first job's slowest run was 1,059.0 ms, so the LCP limit becomes 1,100 ms. The other limits are his budgets from 39.1: TBT under 200 ms, a Cumulative Layout Shift (CLS) of at most 0.1, and a score of at least 0.9.

Ravi's steady lighthouserc.jsonJSON
{  "ci": {    "collect": {      "staticDistDir": "./dist",      "url": ["http://localhost/page.html"],      "numberOfRuns": 5    },    "assert": {      "assertions": {        "categories:performance": ["error", { "minScore": 0.9, "aggregationMethod": "median" }],        "largest-contentful-paint": ["error", { "maxNumericValue": 1100, "aggregationMethod": "median" }],        "total-blocking-time": ["error", { "maxNumericValue": 200, "aggregationMethod": "median" }],        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1, "aggregationMethod": "median" }]      }    },    "upload": { "target": "temporary-public-storage" }  }}
  • Line 6 asks for five runs.
  • Lines 10 to 13 read each assertion from the middle run, with "aggregationMethod": "median".
  • Line 11 is the new LCP limit, with room above every run main has had.

main and Ben's timer passed all 8 of their jobs. Ben merged his timer.

Test the alarm

A check that never fails can't be trusted either. So Ravi replays a known slowdown: Ben's carousel from lesson 39.1, with all of Swiper in app.js. If the new check is any good, it must fail:

The steady check on the carousel, replayedText
$ npx lhci assertChecking assertions against 1 URL(s), 5 total run(s) 1 result(s) for http://localhost:PORT/page.html :   ✘  largest-contentful-paint failure for maxNumericValue assertion       Largest Contentful Paint       https://developer.chrome.com/docs/lighthouse/performance/lighthouse-largest-contentful-paint/         expected: <=1100           found: 1355.7463      all values: 1358.3425, 1387.9397, 1353.8741, 1355.7463, 1352.8456 Assertion failed. Exiting with status code 1.
  • Line 2 says it checked five runs. Line 10 is the limit, 1,100.
  • Line 11 is the middle of the five. Line 12 lists all five: one run jumped to 1387.9397, and the middle run ignored it.

It failed in all 4 jobs. Even its best run, 1,203.4 ms, was over 1,100. Room above the wobble cost nothing: the real slowdown was far past the line. Slide the line yourself. Each job is checked by its middle run:

Playground: where to draw the line
Each job's middle run
  1. main1,058.1 ms, 1,054.0 ms, 1,055.6 ms, 1,054.0 ms: 0 of 4 over
  2. Ben's timer1,054.7 ms, 1,054.6 ms, 1,053.8 ms, 1,054.5 ms: 0 of 4 over
  3. The carousel, replayed1,355.7 ms, 1,354.9 ms, 1,355.1 ms, 1,355.7 ms: 4 of 4 over
Try it yourself
No false alarms, and the carousel is caught

Every job of main and Ben's timer passes, and every job of the carousel fails. Any limit from 1,060 to 1,350 ms does that. Ravi's 1,100 ms sits near the bottom, with room above the wobble.

Each job is checked by its middle run, as Ravi's steady check does. From the course's recording in Chrome 157.

Compare with main, in the same job

One more way to judge a change: measure main in the same job, on the same machine, at the same time. Then compare the two middle runs. Lesson 9.4 called this taking turns.

Real-life analogyTasting two teas side by side

Ravi tastes a new tea against the old one. He brews both from the same kettle, in the same cups, at the same moment. Then any difference is the tea's, not the water's. Tasting the new one today and remembering yesterday's tells him much less.

In real life: The same kettle, cups and moment
In web performance: The same machine, in the same job
In real life: The old tea
In web performance: main
In real life: The new tea
In web performance: The pull request
In real life: The difference in taste
In web performance: The difference between the two middle runs
Each branch against main, round by roundPop out in the code editor (opens in a new tab)JavaScript
// The middle LCP of each job, in ms. Each round ran main, then each branch, on the same machine.const main = [1058.1, 1054, 1055.6, 1054];const ben = [1054.7, 1054.6, 1053.8, 1054.5];const carousel = [1355.7, 1354.9, 1355.1, 1355.7]; const change = (branch) => branch.map((lcp, round) => (lcp - main[round]).toFixed(1)).join(", ");console.log(`Ben's timer: ${change(ben)}`);console.log(`The carousel: ${change(carousel)}`);
  • Lines 2 to 4 hold each job's middle LCP. Each round ran main first, then the other branches.
  • Line 6 subtracts main's middle run from the branch's, round by round.
  • It prints Ben's timer: -3.4, 0.6, -1.8, 0.5 and The carousel: 297.6, 300.9, 299.5, 301.7.

Ben's timer moved the middle run by a few milliseconds, either way: noise. The carousel added about 300 ms, every round. A Lighthouse CI server helps here too: it graphs each build over time, and shows the differences between two reports.

Last, pick the level. Use error for checks you trust: bytes, and the middle run against a limit with room. Use warn for numbers you want to watch that can cross a line by chance, as the recommended preset does. Why it matters: Ravi's checks now fail when something real changes, and only then.

Which check catches it?

Here are the two checks side by side. Read each row across: a question, then the answer for each check.

The bytes check and the lab check
QuestionBytes check (size-limit)Lab check (Lighthouse CI)
What it measuresThe size of the built files, with Brotli.A page load in a lab browser: LCP, TBT, CLS and the score.
Does it wobble?No. The same build always has the same bytes.Yes. Runs of the same commit differ a little.
How many runs?One.Several. Ravi uses five and reads the middle one.
Where does the limit go?Close to today's size.Above the wobble, with room to spare.
How long does it take?A few seconds.About a minute for five runs.

Some problems only real visits show. Sort each one by the check that would catch it first:

Which check catches it?
  • A pull request imports a whole date library into app.js
  • A carousel's styles land in app.css
  • The photo gets loading="lazy", so it starts downloading later
  • The reviews push the page down while it loads
  • A new script keeps the main thread busy for 300 ms at load
  • Taps get slow on old phones after five minutes on the page
  • The live server answers slowly at lunchtime
  • A chat widget is added from the tag manager
Try it yourself
0 of 8 correct

Sort each change by the check that would catch it before, or after, it goes live.

Choose a category for every card. You can change an answer at any time; Reset clears them all.

Common mix-ups

  • "A red check means the change made the page slower." No. main itself, with nothing changed, failed 2 of 20 one-run checks against a limit copied from one run.
  • "Re-run it until it's green." No. That keeps the luckiest run. It can hide a small real slowdown, and it teaches everyone to ignore the check.
  • "Lighthouse CI reads the middle run unless you say otherwise." No. The default is optimistic: for a maximum, the best run.
  • "The median alone makes a check steady." No. Middle runs wobble too. A limit also needs room above them.
  • "A size check needs several runs too." No. The same build always has the same bytes.
  • "Temporary public storage keeps reports private." No. Anyone with the link can open them, and they're deleted after 3 days to 5 weeks.

Practice exercises

Answer first, then use the hints.

Exercise 1 · Warm-upThe carousel's best run

This job of the carousel had one fast run. Read with the best run, does it pass? What does it print?

Starter codePop out in the code editor (opens in a new tab)JavaScript
// One job of the carousel, replayed: its 5 LCPs in ms, and Ravi's steady limit.
const lcps = [1357.7, 1355.7, 1356.5, 1203.4, 1354];
const limit = 1100;

const best = Math.min(...lcps);
console.log(best <= limit ? "passed" : "failed");

Answer, then press Check. Spacing and letter case don’t matter.

    Exercise 2 · Warm-upThe change in the comment

    This is how size-limit-action works out the change it writes. What does it print?

    Starter codePop out in the code editor (opens in a new tab)JavaScript
    // The JavaScript at load, in bytes with Brotli: main, and Ben's branch.
    const main = 3083;
    const ben = 3158;
    
    const change = ((ben - main) / main) * 100;
    console.log(`+${Math.ceil(change * 100) / 100}%`);

    Answer, then press Check. Spacing and letter case don’t matter.

      Exercise 3 · PracticeThe worst of five

      Read with the worst of five runs, how many of the 8 jobs of main and Ben's timer fail Ravi's first limit?

      Answer, then press Check. Spacing and letter case don’t matter.

        Exercise 4 · PracticeThe first false alarm

        Slide the LCP limit down. At which limit does the first false alarm appear, in ms?

        Answer, then press Check. Spacing and letter case don’t matter.

          Exercise 5 · ChallengeShow the passes too

          Ben's re-run printed no assertions at all. Which assert option makes Lighthouse CI print the ones that passed?

          Answer, then press Check. Spacing and letter case don’t matter.

            Exercise 6 · ChallengeWhat the tap fetches

            Open Ben's built Mint tea page in Chrome, with DevTools' Network panel open. Tap Start a 3-minute timer. How many new requests does the tap make?

            Answer, then press Check. Spacing and letter case don’t matter.

              Check your understanding

              For each question, ask: does this number wobble, and how many runs does the check read?

              The performance in CI quiz · 7 questionsScore: first tries count
              1. Question 1 of 7Ben's Lighthouse check failed, then passed when he re-ran the same commit. Why?

                Choose an answer to see the explanation.

              2. Question 2 of 7An assertion has no aggregationMethod. Which run does Lighthouse CI compare with its maxNumericValue?

                Choose an answer to see the explanation.

              3. Question 3 of 7What does this middle-run code print?

                Read the code, then predictPop out in the code editor (opens in a new tab)JavaScript
                const lcps = [1358.3, 1387.9, 1353.9, 1355.7, 1352.8];
                const sorted = [...lcps].sort((a, b) => a - b);
                
                console.log(sorted[2]);

                Choose an answer to see the explanation.

              4. Question 4 of 7Why can a size check use a limit close to today's number, when an LCP check can't?

                Choose an answer to see the explanation.

              5. Question 5 of 7Which upload target keeps the reports private, and keeps their history so builds can be compared?

                Choose an answer to see the explanation.

              6. Question 6 of 7An assertion is set to warn. What happens when it fails?

                Choose an answer to see the explanation.

              7. Question 7 of 7Why did Ravi run Ben's old carousel pull request through his new checks?

                Choose an answer to see the explanation.

              Key takeaways

              • A workflow on pull_request builds every change and runs the team's checks before anyone merges it.
              • Bytes don't wobble. A size check needs one run and can sit close to today's size. size-limit-action adds the change against main.
              • Lighthouse CI collects several runs, asserts limits at error, warn or off, and uploads the reports. A status lands on the pull request.
              • Lab timings wobble. A limit copied from one run fails by chance: main failed 2 of 20 such checks.
              • Read the middle of five runs, leave room above the wobble, and compare with main in the same job.
              • Test the check on a known slowdown. The replayed carousel failed every job.

              Remember the one-liner.
              Check bytes exactly and timings by the middle of several runs, with room, so a red check always means something real.

              Sources

              The facts on this page come from these docs and source files:

              Coming next: Monitoring and alerting. Checks on pull requests see the lab. Next, you watch real visits after a change goes live, and get told when they slow down.

              CompleteFrontend Clear concepts. Working examples.