Performance experiments
Test a speed change on real visitors: split them by chance at the edge, measure clicks with an interval, value speed with a slowdown, and avoid the traps.
- 01A/B test a speed changeSplit real visitors by chance, check that the speed really changed, and measure the business number with z and an interval.
- 02Put a value on speedSlow a page down on purpose, read what each second costs, and work out how many visitors a small effect needs.
- 03Run it at the edge, and trust itPick each visitor's group before the HTML leaves, keep the groups apart in caches, and catch uneven groups and early looks.
What is a faster page worth?
Meera has built the fix from Ravi's report: a smaller home photo, loaded first. In the lab, the photo now appears after 1.3 seconds instead of 6.4. The owner reads the numbers and asks one more question: "Will more people look at our teas?"
It's a fair question. Meera's hours cost money, and the owner wants to know what they buy. So Ravi plans to test the fix on real visitors. Here are both pages on Asha's phone, the slow phone setting from lesson 40.1:
With the fix, the photo is whole about five seconds sooner. Why it matters: the lab shows how fast each page is. It can't show what visitors do about it.
So how do you find out what speed is worth? The lab has no customers. Comparing fast visits with slow ones doesn't work either. Lesson 1.2 showed why. Slow visits come from slower phones and networks, and those visitors may click less anyway.
The answer is the A/B test from lesson 13.2, aimed at a new number. Split real visitors by chance. Give each group one version. Then count what each group does. Lesson 13.2 counted milliseconds. This lesson counts clicks.
A gardener wants to know if a new fertiliser grows more tomatoes. Comparing her best plants with her worst won't tell her. The best ones may just get more sun. So she flips a coin for each row: heads get the fertiliser, tails don't. In autumn, she counts the tomatoes, row by row.
- In real life: The rows of tomato plants
- In an experiment: Ravi's phone visitors
- In real life: A coin flip for each row
- In an experiment: Chance picking each visitor's version
- In real life: The new fertiliser
- In an experiment: The faster page
- In real life: Counting tomatoes in each row
- In an experiment: Counting clicks in each group
A performance experiment shows real visitors different speeds, picked by chance, and counts what each group does. It tells you what speed is worth in clicks or sales, which no lab can. To trust the answer, keep the groups fair, and read the result once, at the end.
Here is Ravi's whole test in one picture:
Each phone visitor lands in one of three groups, a third each:
- original: the home page as it is.
- fast: the fixed page.
- slow: the page as it is, held back 500 ms before it's sent.
The edge is a server close to the visitor that runs a little code before a page is sent. Lesson 35.4 ran pages there. Ravi's edge picks each visitor's group. A beacon is the short report a page sends home, as in lesson 12.4. The next four sections take one part each: the A/B test, the slowdown, the edge, and the traps.
How the course recorded it
- The shop: Ravi's home page from lesson 40.1, in the three versions. A small edge server on the course's computer sent them over HTTPS, at a made-up address. Its code is in the edge section.
- The visitors: 840 phone visits over a week, 120 a day, from 8:00 to 22:00 on the shop's clock. Each was a new visitor in real Chrome 157, with its own phone speed and network, drawn the way lesson 13.1 drew them. The shop's clock followed each visit's planned time, so a week of visits could run back to back. 35 more came on the first morning, and 9 came back on the last evening.
- The clicks: Chrome measured each visit's LCP, the moment the biggest image or text appears: here, the photo. Whether the visitor then clicked "See all our teas" came from the course's model below. Then the visitor really clicked it with the mouse, or went on to another page.
- The lab: the pictures use Asha's phone: Lighthouse's phone screen, DevTools' Slow 4G network, and a CPU 4× slower. Each version loaded 5 times, taking turns. The pictures show the middle load.
Real visitors decide for themselves. The course's visitors follow a simple rule instead:
// The course's model of a visitor: the later the photo, the less likely a click.function clickChance(lcpMs) { const chance = 0.5 - 0.06 * (lcpMs / 1000); return Math.max(0.05, chance);} console.log(`Photo at 1.3 s: ${Math.round(clickChance(1300) * 100)}% click`);console.log(`Photo at 6.4 s: ${Math.round(clickChance(6400) * 100)}% click`);console.log(`Photo at 9 s: ${Math.round(clickChance(9000) * 100)}% click`);- Line 2 starts the rule. It takes the visit's LCP, in milliseconds (ms).
- Line 3 starts from a 50% chance, and takes away 6 points for each second of waiting. A point is one percent: 50% minus 6 points is 44%.
- Line 4 never lets the chance fall below 5%.
- Lines 7 to 9 print
Photo at 1.3 s: 42% click,Photo at 6.4 s: 12% clickandPhoto at 9 s: 5% click.
This rule is the course's own, not a law of nature. Its shape comes from real tests. Google's searchers searched less as its delays grew, as lesson 1.2 showed. At Bing, a straight line fit the loss well. The sizes, 50% and 6 points a second, are the course's: big enough for one week of a small shop to show them. Everything else is real: the pages, the edge, the loads, the beacons and the statistics.
Test the fix on real visitors
Before the first visitor arrives, Ravi writes down three things. Changing them later would let him pick the answer he likes.
- Who is in it: phone visits to the home page. That's where the audit found the slow photo. Computers get the page as it is, outside the test.
- What to count: the share of visits that click "See all our teas". That's the shop's business number for this page.
- How long: one whole week, read once at the end. Kohavi's guide from lesson 13.2 says to run tests "for at least a week or two", so every day of the week is in.
The week's result
Here is the week, group by group:
// A week of phone visits to the home page, split three ways by the edge's coin.const week = { original: { visits: 302, clicks: 101, p75: 3960 }, fast: { visits: 268, clicks: 128, p75: 600 }, slow: { visits: 270, clicks: 65, p75: 4544 },}; for (const [group, { visits, clicks, p75 }] of Object.entries(week)) { console.log(`${group}: ${Math.round((clicks / visits) * 100)}% of ${visits} clicked, p75 LCP ${p75} ms`);}- Lines 3 to 5 are recorded. Each holds a group's visits, its clicks, and its p75 LCP: three visits in four were at it or faster.
- Line 8 goes through the groups. Line 9 prints the share that clicked, and the p75.
- It prints
original: 33% of 302 clicked, p75 LCP 3960 msandfast: 48% of 268 clicked, p75 LCP 600 ms. - Then it prints
slow: 24% of 270 clicked, p75 LCP 4544 ms.
Read the p75s first. The fixed page's is 600 ms, against 3,960 ms for the page as it is. The slow page's is 584 ms later than the original's: about half a second, as planned. Why it matters: check this first. If the fast group's LCP hadn't dropped, its clicks would say nothing about speed.
Now the clicks. 48% of the fast group clicked, against 33% of the original group. The slow group clicked 24%.
Is the gap real?
Lesson 13.2 asked this of milliseconds, with Mann-Whitney. Clicks are simpler: each visit either clicked or didn't. NIST's handbook of statistics has a z test for two shares like these:
// Clicks out of visits in the week: the fixed page, and the page as it is.const fast = { visits: 268, clicks: 128 };const original = { visits: 302, clicks: 101 }; const pooled = (fast.clicks + original.clicks) / (fast.visits + original.visits);const se = Math.sqrt(pooled * (1 - pooled) * (1 / fast.visits + 1 / original.visits));const z = (fast.clicks / fast.visits - original.clicks / original.visits) / se;console.log(`z = ${z.toFixed(2)}: ${Math.abs(z) > 1.96 ? "not chance" : "could be chance"}`);- Lines 2 and 3 are recorded: each group's visits and clicks.
- Line 5 is the share that clicked in both groups together. If the fix did nothing, both groups would share this one rate.
- Line 6 is the standard error: how far apart two groups' shares usually land by chance alone, with this many visits.
- Line 7 divides the gap between the shares by it. That's z.
- Line 8 prints
z = 3.48: not chance.
As in lesson 13.2, a z beyond 1.96, either way, means p is below 0.05. 3.5 is far past it. So the gap isn't luck.
How big is the gap?
z says the gap is real. The bootstrap says how big it is. Resample each group's visits 500 times, as lesson 13.2 did, and keep the middle 90% of the gaps. The fixed page brought 14.3 more clicks per 100 visits. Its 90% interval runs from 7.6 to 20.7.
So Ravi tells the owner: "On phones, the faster home page brought about 14 more clicks on our teas per 100 visits. Our test of 570 visits puts it between 8 and 21." That's lesson 40.2's rule: the answer first, with how sure you are.
Rakuten 24, an online shop in Japan, tested a speed change the same way. For a month, half its visitors got a landing page tuned for Core Web Vitals. The other half got the original. web.dev reports that nothing else differed, and that the faster page had a 33.13% higher conversion rate.
Ron Kohavi and his co-authors have run thousands of experiments at Microsoft, Amazon, LinkedIn and Booking.com. They add a warning: an idea that worked on one site "may not work as well for another." That's why Ravi tests on his own visitors.
Slow the page down on purpose
The test answered one question: what this fix is worth. Next month, the owner will ask about the tea list. Must Ravi build every fix before he knows its value?
No. He can measure what time itself is worth. Kohavi's team writes that the best way is to "isolate just that factor using a slowdown experiment, i.e., add a delay."
A café is thinking about a faster coffee machine. It can't test a machine it doesn't have. But it can make some coffee slower. For one table in three, picked by a coin, the barista waits an extra minute before serving. If those tables order less, the café learns what a minute costs. That's roughly what the new machine would earn.
- In real life: One table in three, picked by a coin
- In an experiment: The slow group, picked by the edge
- In real life: The extra minute before serving
- In an experiment: The 500 ms the edge holds the page back
- In real life: Orders lost per minute of waiting
- In an experiment: Clicks lost per second of LCP
- In real life: What the faster machine would earn
- In an experiment: What a future speed fix is worth
Big sites have done exactly this. Google held its search results back by 100 to 400 ms for one group of users, as lesson 1.2 showed. Bing slowed one tenth of its users by 100 ms, and another tenth by 250 ms, for two weeks. Every 100 ms of speed turned out to be worth 0.6% of revenue. Kohavi's team put it this way: an engineer who makes the server 10 ms faster "more than pays for his fully-loaded annual costs."
A slowdown measures a loss. Why does it tell you what a gain is worth? Near today's speed, a little more waiting and a little less change the number by about the same amount. Kohavi's team calls this a linear approximation. They tried several delays, and found it "very reasonable for Bing". It holds near today's speed. If the site gets much faster, or its visitors change, measure again.
Ravi's slow group
Ravi's edge holds the slow group's page back 500 ms before it sends a single byte. Google's first kind of delay worked the same way. The slow group's photo came 511 ms later on average. Its clicks fell from 33% to 24%, with z = -2.46: past 1.96. So what is a second worth? Here it is, worked out from each group:
// Each group's share that clicked, and its mean LCP in seconds, from the week.const original = { clicked: 0.334, lcp: 3.23 };const fast = { clicked: 0.478, lcp: 0.52 };const slow = { clicked: 0.241, lcp: 3.74 }; const perSecond = (quick, later) => ((quick.clicked - later.clicked) / (later.lcp - quick.lcp)) * 100;console.log(`The fix: ${perSecond(fast, original).toFixed(1)} points per second`);console.log(`The slowdown: ${perSecond(original, slow).toFixed(1)} points per second`);- Lines 2 to 4 are recorded: each group's share that clicked, and its mean LCP in seconds. The mean is the average: add them all up, then divide by how many there are.
- Line 6 divides the change in clicks by the change in LCP, and turns it into points.
- Line 7 prints
The fix: 5.3 points per second. Line 8 printsThe slowdown: 18.2 points per second.
Two answers to one question, and one is more than three times the other. Which is right? Here you can check, because the clicks come from the course's model. For these very visits, the model's chances add up to a 15.5-point gain for the fix, and a 3.0-point loss for the slowdown. That's about 6 points a second, the model's own rate. The fix measured 14.3: close. The slowdown measured 9.4: three times too big.
Each dot below is a group, with its 90% interval as a bar:
The slow group's loss has a 90% interval from 3.2 to 15.2 points. The model's 3.0 sits just outside it. A 90% interval misses one time in ten, and this was one of those times. Half a second is a small step, so a little luck in the clicks becomes a big error per second. The fix moved LCP by almost three seconds, so the same luck shrinks.
Kohavi's team has a name for this warning. When a result looks too strong, they call out Twyman's law: "Any figure that looks interesting or different is usually wrong!" Bing lost 0.6% of revenue per 100 ms. A loss of 28% of Ravi's clicks for half a second is that kind of figure. Why it matters: a small delay needs far more visitors than a big fix.
How many visitors would it take?
Kohavi's guide gives a quick formula. It works out how many visitors a test needs to find a gap 9 times in 10, with 95% confidence:
// Kohavi's formula: visitors in all groups for 95% confidence and 90% power.const rate = 0.33;const gap = 0.03;const groups = 2; const visitors = ((4 * groups * Math.sqrt(rate * (1 - rate))) / gap) ** 2;console.log(`${Math.round(visitors).toLocaleString("en-US")} visitors`);console.log(`${Math.ceil(visitors / 120)} days of Ravi's phone visits`);- Line 2 is the share that clicks now, from the original group.
- Line 3 is the gap to find. In the course's model, half a second costs 3 points.
- Line 4 counts the groups: the page as it is, and the slow one.
- Line 6 is the rule: four times the groups, times the spread of one click, divided by the gap, then squared. For a yes-or-no number, the spread is the square root of rate × (1 − rate).
- Line 7 prints
15,723 visitors. Line 8 prints132 days of Ravi's phone visits.
That's months of slowing down half his phone visitors. This is why the famous slowdown tests come from Google and Bing. Try other gaps:
- Share that clicks now33.0%
- Gap to find3 points
- Visitors, both groups together15,723
- Days at 120 phone visits a day132
Kohavi's formula for two groups, 95% confidence and 90% power. Halve the gap, and you need four times the visitors.
Halve the gap, and you need four times the visitors. Why it matters: a small shop can measure a big fix. For the value of small delays, it should borrow from published tests.
Read this table across: each way to put a value on speed, what it tells you, and what to watch for.
| Way | What it tells you | What to watch for |
|---|---|---|
| Compare fast and slow visits in your RUM | That speed and clicks go together. | Slow visits come from slower phones and networks. Other differences hide in the gap. |
| A/B test a fix | What this one fix is worth, on your own visitors. | You have to build the fix first. It can change more than the speed. |
| Slow a page down on purpose | What a second is worth today, for any fix you plan next. | It costs real clicks while it runs. A small delay needs many visitors. |
A slowdown has a price, too: its visitors really wait. In the course's model, half a second costs 3 clicks in every 100 visits. Bing slowed a tenth of its users in each delay group, not half.
Split visitors at the edge
Ravi's test needs each visitor to get their group's page from the very first byte. That rules out choosing the group in a script on the page:
- The fix lives in the HTML: a different
<img>tag. A script on the page runs after that HTML has arrived. By then, the browser is already fetching the old photo. - Testing tools that change the page in the browser often hide it first. Vercel's guide calls this an anti-flicker snippet. The page stays invisible until the tool picks a group, or until a safety timer runs out, "commonly set to four seconds". Meanwhile, "the browser paints nothing, and LCP absorbs the whole wait."
So the choice must come before the page is sent. Vercel's docs say its Routing Middleware "runs globally before the cache". They list "assigning visitors to an A/B test" among its uses.
At a festival gate, a steward gives each newcomer a coloured wristband, picked at random. Inside, every stall serves you by the colour on your wrist. Come back tomorrow with the band on, and you keep your colour. Nobody has to choose again.
- In real life: The steward at the gate
- In an experiment: The edge function, before the page is sent
- In real life: Picking a colour at random
- In an experiment: Picking a group for a new visitor
- In real life: The wristband
- In an experiment: The home-test cookie
- In real life: Stalls that serve by colour
- In an experiment: Each group getting its own page
Here is Ravi's edge function. The course's recording ran this exact file for every visit:
// edge.js: runs for every visit to the home page, before the cache.const GROUPS = ["original", "fast", "slow"]; export async function onHomeRequest(request, { cache, origin, random }) { if (!/Mobi/.test(request.headers["user-agent"])) return { body: await origin("original"), headers: {} }; let group = /home-test=(\w+)/.exec(request.headers.cookie ?? "")?.[1]; const isNew = !GROUPS.includes(group); if (isNew) group = GROUPS[Math.floor(random() * GROUPS.length)]; const key = `/?group=${group}`; const body = cache.get(key) ?? cache.put(key, await origin(group)); if (group === "slow") await new Promise((done) => setTimeout(done, 500)); const headers = { "cache-control": "no-store" }; if (isNew) headers["set-cookie"] = `home-test=${group}; Path=/; Max-Age=2592000`; return { body, headers };}- Line 4 gets the request and three helpers: the edge's
cache,originandrandom().originbuilds a page on the shop's own server.random()gives a number from 0 up to 1. - Line 5 sends computers the page as it is, outside the test. To spot a phone, MDN suggests looking for "Mobi" in the User-Agent, the name a browser sends with each request.
- Line 6 looks for a
home-testcookie, which holds the visitor's group. - Lines 7 and 8: a visitor with no group is new.
random()times 3, rounded down, gives 0, 1 or 2. That picks one of the three groups, by chance. - Line 9 builds the cache key: the address, plus the group.
- Line 10 takes that key's copy from the cache. If there isn't one, it builds a fresh page and stores it.
- Line 11 holds the slow group's page back 500 ms.
- Lines 12 and 13 set the headers. A new visitor gets the cookie, with
Max-Age=2592000. MDN says that's the cookie's life in seconds: 30 days. - Line 14 sends the page.
The cookie keeps a visitor in one group. Kohavi's guide says a returning user "should be assigned to the same variant on each successive visit". Fastly's guide gives the reason: so visitors "don't perceive that the website is changing for no reason." The course checked it. On the last evening, 9 of day 1's visitors came back. All 9 kept their group, and the edge picked no new one.
Line 12 follows a tip from Fastly's guide, too. When an answer sets a cookie, make sure the browser doesn't keep a copy of it. no-store does that.
Each page reports its own group
The pages send their own beacon:
// ab-rum.js: sends the page's group, its LCP and the click home when the page is hidden.const visit = { group: document.body.dataset.group, lcp: null, clicked: false }; webVitals.onLCP(({ value }) => { visit.lcp = value;});document.querySelector('main a[href="/teas"]').addEventListener("click", () => { visit.clicked = true;}); addEventListener("visibilitychange", () => { if (document.visibilityState === "hidden") navigator.sendBeacon("/rum", JSON.stringify(visit));});- Line 2 starts the visit's record. Its group comes from the page itself: each version has
data-groupon its<body>. - Lines 4 to 6 keep the LCP, with the web-vitals library from lesson 12.1.
- Lines 7 to 9 note a click on "See all our teas".
- Lines 11 to 13 send the record with
navigator.sendBeacon()once the page is hidden: the visitor clicked through, or left.
Why read the group from the page, when the cookie has it? The cookie says which page the visitor should get. The page says which one they did get. The next section shows how those two can differ.
Keep the groups apart in caches
Every visitor asks for the same address, /. Lesson 15.4 showed that a cache files each copy under a cache key, usually built from the address. So a test of the page itself must put the group in the key:
- Cloudflare's A/B example rewrites each group to its own address on the origin. Cloudflare's default key is the full address, so each group's page gets its own key. A custom key can also add a cookie's value.
- Fastly's guide sends the group to the origin in a request header. The origin's answer names that header in
Vary, so Fastly can "cache multiple variants of the response". - Vercel's key is the request's method, address, host, deployment and scheme. Cookies aren't in it.
- Ravi's edge builds its own key, on line 9.
Ravi's first version had a different line 9: const key = "/";. One key for every group:
The cookies were still fair. The pages weren't. That's the first trap.
Avoid the traps that fool a test
Uneven groups: a sample ratio mismatch
Ravi first started the test on the morning of day 0, with that one-key line. At noon, he counted the beacons from each page:
// The first morning: beacons from each page, by the group the page says it is.const pages = { original: 0, fast: 11, slow: 24 };const expected = 35 / 3; let chi2 = 0;for (const seen of Object.values(pages)) chi2 += (seen - expected) ** 2 / expected;const p = Math.exp(-chi2 / 2);console.log(`chi-square ${chi2.toFixed(1)}, p = ${p.toFixed(6)}`);console.log(p < 0.0005 ? "Mismatch: stop and find the cause" : "The split looks fair");- Line 2 is recorded: the beacons from each page, by the group the page says it is.
- Line 3 is what a fair three-way split expects: a third of the 35 visits for each group.
- Lines 5 and 6 add up the chi-square. For each group, take its count minus the expected count, square it, and divide by the expected count. NIST's handbook uses this sum to test whether counts fit a plan.
- Line 7 turns it into p. With three groups, the chi-square's p works out to exactly
Math.exp(-chi2 / 2). - Line 8 prints
chi-square 24.7, p = 0.000004. Line 9 printsMismatch: stop and find the cause.
Groups whose sizes don't match the plan have a name: a sample ratio mismatch, or SRM. Microsoft's experiment team checks every test for one before anyone sees a result. Its line is p < 0.0005: strict on purpose, so false alarms are rare.
A teacher takes a class to a museum. She draws names to split the children among three guides. At lunch, she counts heads: one guide has nobody, and another has twice as many as planned. Before she asks which guide was best, she finds out where the children went.
- In real life: Drawing names for the three groups
- In an experiment: The edge picking each visitor's group
- In real life: A third of the class planned for each guide
- In an experiment: A third of the visits planned for each page
- In real life: The head count at lunch
- In an experiment: The beacons from each page
- In real life: Finding the lost group before asking who had fun
- In an experiment: Finding the cause before reading any result
Where did the original pages go? The cookies were fair: 13, 10, 12. The cache wasn't. Its first copy, at 08:00, was the fast page, and every cookie got it for two hours. At 10:02, the next copy was the slow page. Microsoft's paper on SRMs describes this pattern. When the mismatch is strongest on the first day, look for "time related factors like caching".
The paper compares an SRM with a fever: a sign that something is wrong, not what. It says an SRM "in most cases completely invalidates experiment results". It found one in about 6% of Microsoft's experiments. So Ravi fixed line 9, threw the morning away, and started again on day 1. The week's pages split 302, 268, 270, with p = 0.27: fair. Check all three counts:
- Original13, against 11.7 for a fair split
- Fast10, against 11.7 for a fair split
- Slow12, against 11.7 for a fair split
- Chi-square0.4
- p0.818731
- Below Microsoft's line, 0.0005?No: the split looks fair
Each row is one group's count. The chi-square test asks how often a fair three-way split would land this far from equal thirds.
One more cause matters in speed tests. The paper notes that a slower page can lose logs, because visitors leave before they're sent. A faster page can show more visitors. Why it matters: count the groups from the pages' own beacons, and check the split before you read anything else.
Looking too early
Ravi planned one look, at the end of day 7. What would a look each evening have told him? Pick a day:
- Original: clicked101 of 302, 33.4%
- Fast: clicked128 of 268, 47.8%
- Slow: clicked65 of 270, 24.1%
- Fast against original+14.3 points, z = 3.48, past 1.96: looks real
- Slow against original−9.4 points, z = -2.46, past 1.96: looks real
Each row counts every visit from day 1 to the day you picked. z is NIST's test for two shares, as in the lesson's code.
Look at day 2. The slow group was 21.5 points behind, with z = -2.86. Stop there, and Ravi would report a loss 7 times the model's. That same evening, the fixed page's z was only 0.40. Stop there, and the fix seems to do nothing. By day 7, both were closer to the model. Why it matters: early looks swing both ways, so a test stopped at a good-looking moment reports luck.
Two equally good football teams play. Suppose one coach could end the match the moment her team scored. Her team would win far more than half its matches. The game is only fair if it runs the full 90 minutes.
- In real life: Two equally good teams
- In an experiment: Two groups with the same page
- In real life: Ending the match the moment you lead
- In an experiment: Stopping the first day z passes 1.96
- In real life: Winning more than half the matches
- In an experiment: Finding false differences far more than 5% of the time
- In real life: The full 90 minutes
- In an experiment: The size you chose before the start
The course measured this. It took the original group's real visits and split them in two by a coin, 1,000 times. Both halves saw the same page, so any difference is luck. Kohavi's guide calls this an A/A test. At 95% confidence, it should find a difference about 5% of the time.
- Tests run1,000, each with the same page on both sides
- When Ravi looksOnce, at the end of day 7
- Tests that found a difference50 of 1,000, 5.0%
- What the 1.96 line promisesAbout 5% when there's no difference
Each test splits the original group's real visits in two by a coin. Both halves saw the same page, so every difference it finds is luck.
Looking once found a difference 50 times in 1,000. Looking every evening, and stopping at the first z past 1.96, found one 171 times. Johari's team at Stanford calls p-values "wholly unreliable" when people keep watching a test and stop when it suits them. So decide how many visitors before you start, and read the result once. Or use a method built for watching, such as their "always valid" p-values.
Too few visitors, too short a test
- Too few visitors. The slowdown showed it. Kohavi's guide: "A common mistake is to run experiments that are underpowered." Work out the visitors before you start.
- Too short. Run whole weeks, so weekdays and weekends both count. Some effects also change over time. Google's searchers searched less the longer its delay lasted.
- New things. Some people try a new design just because it's new. Kohavi's team found these novelty effects uncommon, but suggests running two weeks and looking for them.
- Groups that remember. After Google removed its 400 ms delay, that group still searched less for weeks, as lesson 1.2 showed. Don't reuse a slowed group in the next test straight away.
Which trap is this?
Each problem below could fool a performance experiment. Sort it into the trap it falls into:
- The edge keeps one copy of the home page for every group
- Visitors on the slow page leave before its beacon is sent
- Ravi stops on day 2 because z passed 1.96
- A dashboard checks z every hour, and the first green light wins
- 840 visitors to size a 3-point loss
- A shop with 30 visits a day tests a 100 ms delay
- The test runs from Monday to Wednesday
- A slowdown runs for two days, though losses grow week by week
Sort each problem by the trap it falls into.
Common mix-ups
- "A faster page always earns more, by the same amount everywhere." No. Kohavi's team warns that a result from one site may not work as well on another. Test on your own visitors.
- "Compare fast and slow visits in your RUM to put a value on speed." No. Slow visits come from different visitors. Only a split by chance leaves speed as the one difference.
- "A script on the page can A/B test a speed change." No. By the time it runs, the browser is already loading the old page's files. Pick the group at the edge.
- "If the cookies split fairly, the groups are fair." No. On Ravi's first morning, the cookies were fair and the cache sent the wrong pages. Count the pages' own beacons.
- "Stop as soon as z passes 1.96." No. In the course's A/A tests, daily looks found far more false differences. Decide the size first, and look once.
- "A result past 1.96 has the right size." No. Ravi's slowdown passed 1.96, and still measured three times the model's loss. A week of a small shop sizes a small effect badly.
- "A slowdown test is free." No. Its visitors really wait. Google's lost searches lasted weeks after its delay ended.
Practice exercises
Answer first, then use the hints.
The model gets a visit whose photo appeared at 4,000 ms. What does it print?
// The course's model of a visitor: the later the photo, the less likely a click.
function clickChance(lcpMs) {
const chance = 0.5 - 0.06 * (lcpMs / 1000);
return Math.max(0.05, chance);
}
console.log(`Photo at 4 s: ${Math.round(clickChance(4000) * 100)}% click`);Photo at 4 s: 26% click. 26% is above the 5% floor, so line 4 keeps it.
With the cache key fixed, the morning's pages split 12, 11 and 12. What does the last line print?
// A morning with the key fixed: beacons from each page.
const pages = { original: 12, fast: 11, slow: 12 };
const expected = 35 / 3;
let chi2 = 0;
for (const seen of Object.values(pages)) chi2 += (seen - expected) ** 2 / expected;
const p = Math.exp(-chi2 / 2);
console.log(`chi-square ${chi2.toFixed(1)}, p = ${p.toFixed(6)}`);
console.log(p < 0.0005 ? "Mismatch: stop and find the cause" : "The split looks fair");The split looks fair. Chi-square is about 0.06, so p is about 0.97, far above 0.0005.
Looking every evening, how many of the A/A tests found a difference?
171. Every one of them is a false alarm: both halves saw the same page. Looking once found 50.
How many visitors, in both groups together, does a 10-point gap need?
1,415. A gap 10 times bigger than 1 point needs 100 times fewer visitors: the rule squares the gap.
Open the fixed page's trace in DevTools' Performance panel. Which image file was the LCP?
green-tea-800.jpg. On Asha's 412 px screen the photo's slot is 380 px wide, and each CSS pixel is 1.75 screen pixels: 665 in all. So srcset picks the 800 px photo.
In Cloudflare's A/B testing example, which two path prefixes does the Worker put in front of the address?
/control and /test. Each group's page has its own address on the origin, so each gets its own copy in the cache.
Check your understanding
For each question, ask: is the comparison fair, is the split even, and is the test big enough?
Question 1 of 7Why can't Ravi value speed by comparing his fast visits with his slow ones in his RUM data?
Choose an answer to see the explanation.
Question 2 of 7Why does Ravi choose each visitor's group at the edge, instead of in a script on the page?
Choose an answer to see the explanation.
Question 3 of 7What does this test of a 5-point gap print?
Read the code, then predict// A made-up test: 200 visitors a group, 30% and 25% clicked. const fast = { visits: 200, clicks: 60 }; const original = { visits: 200, clicks: 50 }; const pooled = (fast.clicks + original.clicks) / (fast.visits + original.visits); const se = Math.sqrt(pooled * (1 - pooled) * (1 / fast.visits + 1 / original.visits)); const z = (fast.clicks / fast.visits - original.clicks / original.visits) / se; console.log(`z = ${z.toFixed(2)}: ${Math.abs(z) > 1.96 ? "not chance" : "could be chance"}`);Choose an answer to see the explanation.
Question 4 of 7On the first morning, the cookies split fairly, but one page sent no beacons at all. What should Ravi do?
Choose an answer to see the explanation.
Question 5 of 7Midway through the week, the fixed page's z passes 1.96. Why not stop the test there?
Choose an answer to see the explanation.
Question 6 of 7Bing slowed some users by 250 ms. Why does that say anything about what a speed-up would be worth?
Choose an answer to see the explanation.
Question 7 of 7Ravi's two estimates of what a second is worth disagree a lot. Why trust the fix's estimate more than the slowdown's?
Choose an answer to see the explanation.
Key takeaways
- A performance experiment shows real visitors different speeds, picked by chance, and counts what they do. No lab can.
- Check that the speed changed first. Then test the business number: z says whether the gap is real, the interval how big.
- A slowdown puts a value on time itself. Near today's speed, a little slower and a little faster cost and earn about the same.
- Small effects need many visitors. Work out how many before you start.
- Pick the group at the edge, before the HTML leaves. Keep it in a cookie, and put it in every cache key.
- Count the pages' own beacons, and check the split first. A mismatch means stop, fix, and start again. Read the result once.
Remember the one-liner.
Split by chance, check the split, look once, and let real visitors say what speed is worth.
Module 40 in one look
- The audit: field data says where, the lab says why, a ranked list says what first.
- The report: the answer first, proof for each claim, a name and a day for each fix.
- The experiment: real visitors, split by chance, show what the fix is worth.
Sources
The facts on this page come from these papers and docs:
- Speed Matters for Google Web Search, Jake Brutlag, Google.
- Seven Rules of Thumb for Web Site Experimenters, Ron Kohavi, Alex Deng, Roger Longbotham and Ya Xu.
- Practical Guide to Controlled Experiments on the Web, Ron Kohavi, Randal M. Henne and Dan Sommerfield.
- Rakuten 24's investment in Core Web Vitals, web.dev.
- How can we determine whether two processes produce the same proportion of defectives?, NIST/SEMATECH e-Handbook of Statistical Methods.
- Bootstrap Plot, NIST/SEMATECH e-Handbook of Statistical Methods.
- Routing Middleware, Vercel docs.
- A/B testing on ecommerce storefronts without hurting Core Web Vitals, Vercel Knowledge Base.
- Browser detection using the user agent, MDN.
- Set-Cookie header, MDN.
- A/B Testing (VCL), Fastly docs.
- web-vitals, Google Chrome on GitHub.
- A/B testing with same-URL direct access, Cloudflare Workers docs.
- Cache keys, Cloudflare docs.
- Purging Vercel CDN cache: cache keys, Vercel docs.
- Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, Aleksander Fabijan and others, KDD.
- Diagnosing Sample Ratio Mismatch in A/B Testing, Microsoft Research, Experimentation Platform.
- Chi-Square Goodness-of-Fit Test, NIST/SEMATECH e-Handbook of Statistical Methods.
- Always Valid Inference: Continuous Monitoring of A/B Tests, Ramesh Johari, Leo Pekelis and David Walsh.
Coming next: The business case for performance. You'll turn tests like Ravi's, and published studies, into a case for speed work.