Noise and variability

Same page, different numbers

You've changed something in the Leaf and Kettle store, and you want to know if the photo of mint tea now appears sooner. So first you measure the store as it is. You load it on your laptop with the CPU four times slower, like DevTools' mid-tier phone setting. The photo appears after 360 ms. To be sure, you load it again: 356 ms. Then 528. Then 644.

You didn't change anything between those loads. So which number is the store's real speed? And when you test your change, how will you know if a new number means it helped, or that you got lucky?

Here are those four loads. They're real: the course loaded the store in Chrome, one load after another.

1// The store, loaded four times in a row with 4x CPU. Nothing changed between loads.2// Each number is one load's LCP, in milliseconds.3const loads = [360, 356, 528, 644];4 5console.log(`The fastest load: ${Math.min(...loads)} ms.`);6console.log(`The slowest load: ${Math.max(...loads)} ms.`);

• Lines 1 and 2 are comments. LCP, Largest Contentful Paint, is the moment the biggest image or block of text appeared. On the store, that's the photo. A millisecond (ms) is a thousandth of a second.
• Line 3 holds the four recorded LCPs, in the order the loads ran.
• Lines 5 and 6 print The fastest load: 356 ms. and The slowest load: 644 ms.

The slowest load took 288 ms longer than the fastest: nearly twice as long. Nothing about the page changed. This lesson calls that wobble noise: the differences between runs of the same test, on the same page. How far the numbers reach, from the lowest to the highest, is their spread.

You've met noise before. In The Lighthouse score, the store scored 90 and then 89, like a bathroom scale that reads 60.2 kg and then 60.4. This lesson finds out where the wobble comes from, and what to do about it.

The course kept going. It loaded the store 45 times in a row, the same way each time. Here is every load, one dot each:

If noise were a gentle blur, the dots would pile up around one middle. They didn't. They fell into three clumps, with empty gaps between them. Something put each load in one clump or another. Finding that something is the first job of this lesson.

To find the store's red lights, the course loaded it four ways. In each round, it loaded the store once each way:

• No throttling: the laptop at full speed.
• 4x CPU: the page's main thread, the part of the browser that runs the page's JavaScript and draws it, made four times slower. That's DevTools' mid-tier phone preset.
• 4x CPU, busy laptop: the same, while a program kept every core of the laptop busy.
• 4x CPU, three at once: three loads started at the same moment.

It ran 9 rounds, taking turns, so anything that changed during the test hit all four ways alike. Each load ran in a fresh browser context, which is like a new private window: nothing saved from the load before. The store came from a server on the same laptop, so no real network was involved.

Pick a way. Each row is one load: the bar is the store's start-up work, and the mark is when the photo appeared.

Where noise comes from

Back to Ravi's red lights. What are the store's? Lighthouse's docs start with a warning: "Lighthouse performance scores will change due to inherent variability in web and network technologies, even if there hasn't been a code change." Then they list seven sources of it. Here they are in plain words:

• Page nondeterminism: the page itself changes, like an A/B test or an ad.
• Local network variability: your own network, such as a busy Wi-Fi router.
• Tier-1 network variability: the long-distance links between networks.
• Web server variability: a busy server answers more slowly.
• Client hardware variability: a different computer gives different numbers.
• Client resource contention: other programs competing for that computer.
• Browser nondeterminism: the browser runs its own work differently from load to load.

Now cross off what can't apply to the store's test. The store runs nothing random and shows no ads. Its server was on the same laptop, and one laptop ran every load. That leaves two rows: the browser itself, and other work on the laptop.

The same three jobs, in a different order

To see what the browser did, the course recorded twelve more loads with a trace: the timeline of everything the browser did, the one DevTools' Performance panel shows. Every trace had the same three long jobs on the main thread:

• Reading the photo. The photo, hero.svg, is a 580 KB drawing. The browser has to read, or parse, all of it before it can show it.
• The start-up work. This is the store's DOMContentLoaded listener: code that runs once the HTML has been read. It builds a search index by sorting 200,000 old order prices, then draws 24 teas.
• Drawing a frame. The browser works out where everything goes and paints it. The first frame with the photo in it gives the LCP.

Every load did all three, and reading the photo always came first. What changed was which of the other two came next:

In the top row, the frame went first, and the photo showed at 364 ms. In the bottom row, the start-up work went first. The frame waited for it, and LCP came at 652 ms. The middle row had a shorter start-up work, then the frame: 524 ms.

The page doesn't choose the order. The browser does, and it doesn't always choose the same way. The browser also can't paint in the middle of a job: the HTML standard's event loop draws frames between tasks, never during one. Lighthouse's docs call this browser nondeterminism: "Browsers have inherent variability in their execution of tasks that impacts the way webpages are loaded."

Here are all twelve traced loads, sorted by the order their jobs came in:

1// Twelve traced loads with 4x CPU. Each: [LCP, start-up work's start, its end], in ms.2const loads = [3  [652, 247.6, 520.8], [364, 349.2, 620.9], [524, 258.5, 402.8], [356, 343.4, 617.8],4  [652, 254.6, 529.2], [652, 255.5, 528.9], [364, 346.8, 619.3], [548, 250.1, 448.4],5  [368, 353.5, 498.5], [648, 250.9, 524.8], [652, 256.5, 530.1], [644, 251.6, 525.8],6];7// The browser paints between tasks, never in the middle of one.8// So an LCP before the start-up work ended means the frame came first.9function orderOf([lcp, start, end]) {10  if (lcp < end) return "frame first";11  return end - start < 200 ? "short start-up work first" : "long start-up work first";12}13 14const lcps = {};15for (const load of loads) (lcps[orderOf(load)] ??= []).push(load[0]);16for (const [order, list] of Object.entries(lcps)) {17  console.log(`${order}: ${list.length} loads, LCP ${Math.min(...list)} to ${Math.max(...list)} ms`);18}

• Lines 3 to 5 hold the twelve recorded loads. Each has its LCP, and when the start-up work started and ended, from the page's own timing.
• Lines 9 to 12 decide each load's order. Line 10 says the frame went first if the photo appeared before the start-up work ended. Otherwise the start-up work went first, and line 11 calls it short if it took under 200 ms.
• Lines 14 to 18 group the LCPs by order and print one line per group.
• The first line is long start-up work first: 6 loads, LCP 644 to 652 ms.
• The second is frame first: 4 loads, LCP 356 to 368 ms.
• The third is short start-up work first: 2 loads, LCP 524 to 548 ms.

The page's own timing comes from two APIs. LCP comes from LargestContentfulPaint entries. The start-up work's start and end come from the navigation entry's domContentLoadedEventStart and domContentLoadedEventEnd:

Why it matters: the store's clumps aren't random fuzz. Each clump is one order of the same three jobs. A number from one load tells you which order that load happened to get, not how fast the store is.

Four times slower, but not exactly

The order wasn't the only thing that moved. Here is the start-up work in the 9 rounds, with no throttling and with 4x CPU:

1// The store's start-up work in the nine rounds, in ms: with no throttling, and with 4x CPU.2const startup = {3  "no throttling": [40.7, 40.8, 40.9, 41.4, 34.9, 35.1, 41.3, 41.1, 41.3],4  "4x CPU": [146.3, 274.1, 272.1, 272.2, 145.5, 273.9, 272.6, 143.6, 276.4],5};6const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];7const usual = median(startup["no throttling"]);8 9for (const [way, list] of Object.entries(startup)) {10  const low = Math.min(...list);11  const high = Math.max(...list);12  console.log(`${way}: ${low} to ${high} ms, ${(low / usual).toFixed(1)} to ${(high / usual).toFixed(1)} times as long`);13}

• Lines 3 and 4 hold the start-up work's recorded length in each round.
• Line 7 takes the median length with no throttling as the usual length.
• Line 12 prints each way's shortest and longest run, and how many times the usual length each is.
• It prints no throttling: 34.9 to 41.4 ms, 0.9 to 1.0 times as long.
• Then it prints 4x CPU: 143.6 to 276.4 ms, 3.5 to 6.8 times as long.

"4x" didn't mean four times as long. The same code took 3.5 to 6.8 times as long. It ran at two speeds: about 146 ms in 3 rounds, and about 274 ms in the other 6.

On an empty page, the same setting was steady. In Choosing test conditions, DevTools' CPU benchmark, a small test loop, scored 3,192 on this laptop at full speed. At 4x it scored 774.5: about four times slower, run after run. Inside a page load, with other work around it, the slowdown wandered.

So how does Chrome slow a CPU down? DevTools' CPU setting sends Chrome the command Emulation.setCPUThrottlingRate. Chrome hands the rate to its CPU throttler. On Mac and Linux, Chrome's source shows how the throttler works:

• A separate thread wakes up about every 200 microseconds and sends the page's main thread a signal: a tap on the shoulder from the operating system. A microsecond is a thousandth of a millisecond.
• The main thread stops what it's doing. It works out how long it ran since its last stop, counting at most 1,000 microseconds.
• Then it waits, checking the clock, for run_duration_us * throttling_rate_percent / 100 - run_duration_us. At 4x, that's three times as long as it just ran.

It's like slowing a runner by blowing a whistle every few steps, and making them stand still three times as long as they just ran. A timer and a signal on your own laptop do the slowing. So 4x is what Chrome aims for, not what every task gets.

Why it matters: a CPU setting doesn't give every load the same slow computer. It brings noise of its own. Compare runs made with the same setting on the same laptop, and never read 4x as an exact multiplier.

A busy laptop, and three loads at once

The other row left on the docs' list is client resource contention: other programs competing for the same computer. The docs warn: "Running multiple instances of Lighthouse at once also typically distorts results due to this problem." The first playground tried two kinds of busy laptop. Here are their loads next to the loads with the laptop alone:

1// Nine rounds with 4x CPU, taking turns: each load's LCP in ms.2const lcp = {3  alone: [532, 652, 644, 644, 540, 660, 648, 528, 388],4  busy: [580, 612, 492, 636, 604, 628, 352, 616, 624],5  together: [6    444, 444, 424, 396, 400, 392, 416, 424, 400,7    388, 384, 676, 428, 432, 424, 408, 592, 696,8    384, 396, 396, 708, 416, 704, 396, 400, 684,9  ],10};11const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];12 13for (const [way, list] of Object.entries(lcp)) {14  console.log(`${way}: median ${median(list)} ms, ${Math.min(...list)} to ${Math.max(...list)} ms`);15}

• Line 3 holds the nine loads with 4x CPU and nothing else running.
• Line 4 holds nine loads made while a program kept one thread spinning on each of the laptop's 8 cores, like a big build running in the background.
• Lines 6 to 8 hold 27 loads: three at a time, for nine rounds.
• Line 14 prints each way's median and range.
• The first line is alone: median 644 ms, 388 to 660 ms.
• The second is busy: median 612 ms, 352 to 636 ms.
• The third is together: median 416 ms, 384 to 708 ms.

You might expect a busy laptop to make every load slower. It didn't. Three loads at once made the median 228 ms faster. The order had changed: with three at once, the frame went first in 21 of 27 loads. Alone, it went first in 1 of 9.

Why it matters: a faster number from a busy machine isn't good news. It's a different test. Run one test at a time, on a machine doing nothing else, and write down what else was running.

How many runs you need

You've met the usual answer already. In The Lighthouse score, you stepped on the bathroom scale a few times and took the middle reading. Scripting Lighthouse ran Lighthouse several times for the same reason. Lighthouse's docs put a number on it: "The median Lighthouse score of 5 runs is twice as stable as 1 run."

The median is the middle value once the values are sorted. The middle one, not the lowest. Here are the store's first five loads:

1// The store's first five loads in a row, with 4x CPU. LCP in ms.2const loads = [360, 356, 528, 644, 652];3const sorted = [...loads].sort((a, b) => a - b);4 5console.log(`Sorted: ${sorted.join(", ")} ms`);6console.log(`The lowest is ${sorted[0]} ms. The middle one is ${sorted[2]} ms.`);

• Line 2 holds the first five of the 45 loads in a row.
• Line 3 sorts a copy of them, from fastest to slowest.
• Line 5 prints Sorted: 356, 360, 528, 644, 652 ms.
• Line 6 prints The lowest is 356 ms. The middle one is 528 ms.

It's tempting to report the lowest: it's your page at its best. But the lowest is the luckiest load. On the bathroom scale, it's the reading you'd like to believe, not the one you should. Only 14 of the 45 loads landed in the fast clump. 23 took 640 ms or more.

So is 528 ms the store's number? Take the next five loads, and the middle one is 544 ms. The five after that: 356 ms. To see how far this goes, the course cut all 45 loads into groups, the way several people might each run the test a few times and report a median:

1// 45 loads with 4x CPU, one after another: LCP in ms.2const lcp = [3  360, 356, 528, 644, 652, 672, 652, 356, 544, 348, 640, 356, 644, 352, 352,4  648, 352, 644, 356, 660, 516, 640, 656, 640, 648, 356, 640, 640, 640, 644,5  360, 512, 512, 524, 520, 652, 524, 368, 656, 652, 644, 364, 652, 644, 360,6];7const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];8 9for (const size of [1, 3, 5, 9]) {10  const medians = [];11  // Cut the loads into groups of this size, in order, and keep each group's median.12  for (let start = 0; start + size <= lcp.length; start += size) {13    medians.push(median(lcp.slice(start, start + size)));14  }15  console.log(`groups of ${size}: ${medians.length} medians, ${Math.min(...medians)} to ${Math.max(...medians)} ms`);16}

• Lines 3 to 5 hold all 45 loads, in the order they ran.
• Line 9 tries four group sizes: single loads, then groups of 3, 5 and 9.
• Lines 12 to 14 cut the loads into groups of that size, in order, and keep each group's median.
• Line 15 prints how many medians there were, and the lowest and highest.
• The first two lines are groups of 1: 45 medians, 348 to 672 ms and groups of 3: 15 medians, 352 to 652 ms.
• The last two are groups of 5: 9 medians, 356 to 652 ms and groups of 9: 5 medians, 356 to 644 ms.

More loads barely helped. Single loads ranged over 324 ms. Medians of nine still ranged over 288 ms. Nine people, each taking a median of five, could report anything from 356 ms to 652 ms for the same page.

The store's loads work like that bus. A median lands in whichever clump holds its group's middle load. About half of the 45 loads were in the two fast clumps (22), and the rest in the slow one (23). So the median flips between clumps, however many loads you take. In Reading distributions, a spread with two humps was called bimodal. The store's has three.

Pick a group size. Each dot is one load, coloured by its order. Each flat line is one group's median:

So how many loads do you need? It depends on the shape of the spread. Read each row of this table as a shape, an example from the store, and what to do:

Tools built for comparing versions take many samples. tachometer, a benchmarking tool from Google, takes "a minimum of 50 samples" from each version by default. Then it keeps taking more until the difference is clear, or its time runs out.

Why it matters: look at the loads before you decide how many to take. A handful is enough for one clump, but not for several.

Medians and stable environments

Keep the run nearest the middle

For a number to report, take the median. One odd load can drag an average a long way, but it can't move a median much, as Benchmarking without fooling yourself shows.

Sometimes you need a whole run, though: a trace to open in DevTools, or a report to share. You can't take the median of five traces, so you keep the run nearest the middle.

You met Lighthouse's helper for this in Scripting Lighthouse: computeMedianRun. Its source says what it looks for: "the run that's closest to the median of the FCP and the median of the TTI." Those are two metrics:

• FCP, First Contentful Paint: when the first text or image appeared.
• TTI, Time to Interactive: in Lighthouse's words, "the amount of time it takes for the page to become fully interactive." Since Lighthouse 10, the report doesn't show it, but "it is still accessible in the Lighthouse result JSON."

The source explains the choice: "We use FCP and TTI because they represent the earliest and latest moments in the page lifecycle." It skips the score on purpose. The source says: "We avoid the median of single measures like the performance score because they can still exhibit outlier behavior at the beginning or end of load."

The course ran Lighthouse five times on the store, with DevTools throttling, so each run was really measured. Here is computeMedianRun, shortened, on those five runs:

1// Five Lighthouse runs of the store with DevTools throttling: score, FCP and TTI in ms.2const runs = [3  { score: 65, fcp: 3031.9, tti: 5301 },4  { score: 63, fcp: 2872.3, tti: 5298.7 },5  { score: 66, fcp: 3017.7, tti: 5287.7 },6  { score: 64, fcp: 2880.3, tti: 5292.8 },7  { score: 64, fcp: 2908.6, tti: 5310.8 },8];9const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];10const fcp = median(runs.map((run) => run.fcp));11const tti = median(runs.map((run) => run.tti));12// computeMedianRun, shortened: the run closest to both medians wins. A tie goes to the lower TTI.13const distance = (run) => (fcp - run.fcp) ** 2 + (tti - run.tti) ** 2;14const kept = [...runs].sort((a, b) => distance(a) - distance(b) || a.tti - b.tti)[0];15 16console.log(`median FCP ${fcp} ms, median TTI ${tti} ms`);17console.log(`kept: run ${runs.indexOf(kept) + 1}, score ${kept.score}`);18console.log(`scores: ${runs.map((run) => run.score).join(", ")}`);

• Lines 3 to 7 hold the five recorded runs: each one's score, FCP and TTI.
• Lines 10 and 11 take the median FCP and the median TTI.
• Line 13 measures how far each run is from both medians. Line 14 sorts the runs by that distance and keeps the closest. If two runs tie, the one with the lower TTI wins.
• Line 16 prints median FCP 2908.6 ms, median TTI 5298.7 ms.
• Line 17 prints kept: run 5, score 64. Line 18 prints scores: 65, 63, 66, 64, 64.

Lighthouse's real computeMedianRun kept the same run, run 5. Its score, 64, also happens to be the median score. Nothing makes sure of that: the score isn't what it compares.

These runs slowed the network too, so the photo took seconds to arrive. LCP came at about 5.3 seconds and barely moved: from 5,329.4 ms to 5,352.7 ms. The CPU's noise still showed in TBT, Total Blocking Time: how long, in total, long tasks blocked the main thread between FCP and TTI. It went from 155.3 ms to 248 ms.

A steady machine

A median calms the noise down. A steady machine makes less of it in the first place.

Lighthouse's docs list ways to move your tests onto solid tiles:

• Enough computer: "Minimum 2 dedicated cores (4 recommended)". A core is one of the processor's workers. Dedicated means nothing else shares it.
• No shared cloud machines: avoid "burstable" or "shared-core" instance types. Some cloud machines run at a lower baseline speed and burst above it. Others share one physical core, taking turns with other machines.
• One test at a time: "DO NOT collect multiple Lighthouse reports at the same time on the same machine." You saw why: three loads at once changed the order.
• A nearby server: "Use localhost or a machine on the same exact network whenever stability is a concern." localhost means the same computer, as the store's server was here.
• Nothing else running: "Isolate your client environment from external influences like anti-virus software and browser extensions."

Why it matters: a number without its setup can't be compared with anything. Next to every result, write down the machine, the Chrome version, the settings, and what else was running.

Comparing before and after

Now the question from the start: you've made a change. Did it help? The course tried a real fix. The store's start-up work builds a search index straight away, though nothing on the first screen needs it. The fix leaves out that one line, buildIndex();, at start-up. The Add to cart click still builds the index, as it always did.

Two habits make a comparison honest: take turns, and test the page against itself first.

Take turns

Suppose you load the old store ten times in the morning, and the fixed one ten times after lunch. If a backup started at lunch, every "after" load ran on a busier laptop. You'd be measuring the backup, not the fix.

So take turns instead: old, new, old, new. Then whatever changes over time hits both sides alike. Taking turns in a fixed order is called round-robin. tachometer works this way: "It round-robins between the two files, running each at least 50 times."

Test the page against itself first

An A/A test compares a page with itself: version A against the same version A. Nothing changed, so any difference it finds is noise. It tells you how big a difference noise alone can make.

The course ran fifteen rounds, taking turns: the store, the same store again, and the fix. First, the store against itself:

1// Fifteen rounds with 4x CPU, taking turns: the store, then the same store again. LCP in ms.2const before = [524, 648, 624, 640, 364, 644, 652, 636, 640, 512, 640, 356, 640, 512, 520];3const again = [644, 520, 660, 652, 372, 652, 356, 360, 652, 644, 648, 636, 660, 640, 640];4const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];5const gaps = before.map((ms, round) => Math.abs(again[round] - ms));6 7console.log(`Median before: ${median(before)} ms. Median again: ${median(again)} ms.`);8console.log(`In one round, the same page differed by ${Math.max(...gaps)} ms.`);

• Lines 2 and 3 hold the store's two sets of loads. They ran in turns, one from each set, round after round.
• Line 5 works out, round by round, how far apart the two loads were.
• Line 7 prints Median before: 636 ms. Median again: 644 ms.
• Line 8 prints In one round, the same page differed by 296 ms.

Against itself, the store got 8 ms slower. That's a regression, a change for the worse, from nothing. And a single round could have "shown" a 296 ms difference. So one pair of loads proves nothing, and neither does a small gap between two medians.

Count the pairs

Medians hide a lot. Here is a fairer question: if you picked one load from each side at random, how often would the new side win? Pair every before load with every after load: 15 loads by 15, or 225 pairs. Then count the pairs where the after load was faster:

1// Fifteen rounds, taking turns: the store, the store again, and the fix. LCP in ms.2const lcp = {3  before: [524, 648, 624, 640, 364, 644, 652, 636, 640, 512, 640, 356, 640, 512, 520],4  again: [644, 520, 660, 652, 372, 652, 356, 360, 652, 644, 648, 636, 660, 640, 640],5  after: [380, 388, 392, 368, 388, 376, 364, 364, 352, 376, 384, 352, 380, 368, 384],6};7const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];8 9// Pair every before load with every other load. Count the pairs where the other was faster; a tie counts half.10function fasterShare(before, other) {11  let wins = 0;12  for (const a of before) for (const b of other) wins += b < a ? 1 : b === a ? 0.5 : 0;13  return wins / (before.length * other.length);14}15 16for (const version of ["again", "after"]) {17  const share = Math.round(fasterShare(lcp.before, lcp[version]) * 100);18  console.log(`${version}: median ${median(lcp[version])} ms against ${median(lcp.before)}, faster in ${share}% of pairs`);19}

• Lines 3 to 5 hold the fifteen rounds: the store, the store again, and the fix.
• Lines 10 to 14 pair every before load with every load of the other version. They count the pairs where the other was faster. A tie counts half.
• Line 18 prints each version's median, and the share of pairs it won.
• It prints again: median 644 ms against 636, faster in 34% of pairs.
• Then it prints after: median 376 ms against 636, faster in 89% of pairs.

The store against itself won only 34% of its pairs. The fix won 89%, far beyond anything noise did in the A/A test. Not 100%: two of the store's own loads, 356 and 364 ms, had the frame go first, and were as fast as the fix.

Now look at the fix's range. Every load after the fix landed between 352 ms and 392 ms: one clump. With no index to build, the start-up work took under 10 ms, so it no longer raced the first frame. Pick a version to set beside the store. Each tick is one load, and the tall mark is the median:

So a change is believable when two things hold:

• it's bigger than the difference your A/A test found;
• it wins most of the pairs, not just the medians.

For small changes, you need many more loads and a confidence interval. tachometer's README explains one: "Loosely speaking, a confidence interval is a range of plausible values for a parameter like runtime." It reports "95% confidence intervals for the percent change from one benchmark to another." When that range spans zero, it can't name a winner: "We can't tell which is faster, because the difference was too small."

Why it matters: the fix, not the noise, deserves the credit only when it beats what the A/A test showed.

Adds noise or fights it?

Each item below makes lab results either noisier or steadier. Sort it, using what the store's loads showed and what the docs advise.

Common mix-ups

• "A busy machine only makes loads slower." No. Three loads at once made the store's median 228 ms faster, because the frame went first more often.
• "More runs always settle the median." No. With clumps, the median jumps between them. Medians of nine still ranged from 356 to 644 ms.
• "4x CPU means everything takes four times as long." No. The store's start-up work took 3.5 to 6.8 times as long.
• "The fastest run is the page's real speed." No. It's the luckiest run. Report the median.
• "One faster load proves the fix." No. The same page, round by round, differed by up to 296 ms.
• "Lighthouse keeps the run with the median score." No. It keeps the run closest to the median FCP and the median TTI.

Practice exercises

Answer first, then use the hints.

// Nine rounds with 4x CPU, taking turns: each load's LCP in ms.
const lcp = {
  alone: [532, 652, 644, 644, 540, 660, 648, 528, 388],
  busy: [580, 612, 492, 900, 604, 628, 352, 616, 624],
  together: [
    444, 444, 424, 396, 400, 392, 416, 424, 400,
    388, 384, 676, 428, 432, 424, 408, 592, 696,
    384, 396, 396, 708, 416, 704, 396, 400, 684,
  ],
};
const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];

for (const [way, list] of Object.entries(lcp)) {
  console.log(`${way}: median ${median(list)} ms, ${Math.min(...list)} to ${Math.max(...list)} ms`);
}

// 45 loads with 4x CPU, one after another: LCP in ms.
const lcp = [
  360, 356, 528, 644, 652, 672, 652, 356, 544, 348, 640, 356, 644, 352, 352,
  648, 352, 644, 356, 660, 516, 640, 656, 640, 648, 356, 640, 640, 640, 644,
  360, 512, 512, 524, 520, 652, 524, 368, 656, 652, 644, 364, 652, 644, 360,
];
const median = (list) => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)];

for (const size of [1, 3, 5, 9, 15]) {
  const medians = [];
  // Cut the loads into groups of this size, in order, and keep each group's median.
  for (let start = 0; start + size <= lcp.length; start += size) {
    medians.push(median(lcp.slice(start, start + size)));
  }
  console.log(`groups of ${size}: ${medians.length} medians, ${Math.min(...medians)} to ${Math.max(...medians)} ms`);
}

Check your understanding

For each question, ask where the spread comes from, what shape it has, and whether the comparison took turns.

Key takeaways

• A lab number is one sample from a spread, even when nothing about the page changes.
• The browser's own order of work, the CPU throttling and other work on the machine all move it.
• A busy machine doesn't only add time. It can change the test, and even make it look faster.
• Report the median, not the lowest. When you need a whole run, keep the one nearest the middle.
• Look at the shape first. With clumps, medians jump between them, however many loads you take.
• Take turns between versions, run an A/A test, and count how often after beats before.

Sources

The facts on this page come from these docs, standards and source files:

• Isolation: browser contexts, Playwright docs.
• Score Variability, Lighthouse docs.
• Event loop processing model, HTML Standard.
• inspector_emulation_agent.cc, setCPUThrottlingRate, Chromium source.
• thread_cpu_throttler.cc, Chromium source.
• tachometer README, Google.
• median-run.js, Lighthouse source.
• interactive.js, the Time to Interactive audit, Lighthouse source.
• Changelog, 10.0.0, Lighthouse.
• default-config.js, the hidden metrics, Lighthouse source.
• total-blocking-time.js, the Total Blocking Time audit, Lighthouse source.
• Burstable performance instances, AWS docs.
• General-purpose machine family, shared-core VMs, Google Cloud docs.

Coming next: More lab tools. DevTools and Lighthouse aren't the only lab tools. You'll meet sitespeed.io, the Firefox Profiler, Safari's timelines and Perfetto, and when to reach for each.