Proving a change worked

Did it really get faster?

The Leaf and Kettle team wants the mint tea page to show up sooner. Its LCP, Largest Contentful Paint, is the moment the biggest thing on the screen appears. On this page, that's the big photo of mint tea. So the team saves the photo again at a lower JPEG quality, and ships it on a Monday.

A week later, they open the dashboard and put the two weeks side by side:

1// The mint tea photo's size in bytes, and p75 LCP in ms, a week before and a week after the change.2const before = { photoBytes: 102809, p75: 500 };3const after = { photoBytes: 21521, p75: 588 };4 5console.log(`The photo got ${Math.round((1 - after.photoBytes / before.photoBytes) * 100)}% smaller.`);6console.log(`p75 LCP went from ${before.p75} to ${after.p75} ms.`);

• Line 1 is a comment. A millisecond (ms) is a thousandth of a second.
• Lines 2 and 3 are recorded. Each holds the photo's size in bytes, and that week's p75 LCP: the time that three visits in four matched or beat.
• Line 5 prints The photo got 79% smaller.
• Line 6 prints p75 LCP went from 500 to 588 ms.

The photo is about a fifth of its old size. Yet the page got 88 ms slower. Did the smaller photo slow the page down? Should the team put the old one back?

No. The smaller photo really did make the page faster. Something else fooled the dashboard.

Reading performance data showed three reasons a p75 can move: the page changed, the mix of visitors changed, or chance brought different visitors. When you change the page on purpose, your job is to rule out the other two. Benchmarking without fooling yourself showed the same for timing code: one timing proves nothing.

The course tested this one change in three ways, with 588 visits in Chrome. Here they are side by side:

• Before and after. A quiet week on the old photo, then a week with a phone campaign on the new one. That's what the dashboard showed.
• An A/B test. One week and 240 visitors. Chance decides which photo each visitor gets.
• Lab builds. 36 builds of the page in a row, each loaded 3 times on one phone-like setup. Two of them bring a bigger photo back by mistake.

In the first two, each visit drew its own CPU speed and network at random, as in Reading performance data. Only the share of phones followed a plan. The photo comes in three versions:

• Photo a is the original: 102,809 bytes.
• Photo b is the new one, saved at a lower JPEG quality: 21,521 bytes.
• Photo c sits in between, at 29,453 bytes. Only the lab uses it.

The next sections take those steps in order: compare fairly, rule out chance, catch a build that got slower, and report the result.

Before/after vs A/B comparisons

Start with the dashboard's way. Ship the change, then compare a week before it with a week after it. This is a before/after comparison. So why did it point the wrong way?

Split each week by device, the way Segmenting field data did:

1// p75 LCP in ms, for phones on their own and desktops on their own, each week.2const before = { phones: 1072, desktops: 224 };3const after = { phones: 692, desktops: 216 };4 5console.log(`Phones: ${before.phones} to ${after.phones} ms`);6console.log(`Desktops: ${before.desktops} to ${after.desktops} ms`);

• Lines 2 and 3 are recorded: each week's p75 LCP for phones on their own, and for desktops on their own.
• Line 5 prints Phones: 1072 to 692 ms.
• Line 6 prints Desktops: 224 to 216 ms.

Phones got 380 ms faster. Desktops got 8 ms faster. Yet everyone together got 88 ms slower. How can both groups improve while the total gets worse? Count who came each week:

1// How many of each week's 120 visits came from phones.2const phoneVisits = { before: 40, after: 83 };3 4for (const [week, phones] of Object.entries(phoneVisits)) {5  console.log(`The week ${week}: ${Math.round((phones / 120) * 100)}% phones`);6}

• Line 2 is recorded: how many of each week's 120 visits came from phones.
• Line 4 goes through the two weeks. Line 5 turns each count into a share of 120.
• It prints The week before: 33% phones and The week after: 69% phones.

The week after had a campaign that brought in phone visitors: about twice the share. Phones were slower than desktops, even with the new photo: 692 ms against 216. So the week with more phones had a slower p75, though every visit in it got the smaller photo.

That's the mix from Reading performance data, like the swimming pool when the families arrive. This time it changed in the same week as the photo, and it moved p75 more than the photo did. Why it matters: a before/after comparison measures everything that changed between the two weeks. Your code is only one of those things.

Racing side by side

The way out is to stop comparing two different weeks.

That race is an A/B test. Ron Kohavi, Randal Henne and Dan Sommerfield wrote a practical guide to experiments on websites. It describes the simplest A/B test like this: "users are randomly exposed to one of two variants: control (A), or treatment (B)".

A variant is one version of the page. The control is the page as it is, here photo a. The treatment is the change, here photo b.

Both variants run at the same time. So a campaign, a holiday or a slow afternoon reaches both of them alike.

Chance decides who gets which

With both variants running together, one question is left: who gets which?

Picking by chance like this is called random assignment. The guide is firm about it: "The key here is 'random.' Users cannot be distributed 'any old which way'; no factor can influence the decision."

When only chance picks, phones and desktops, fast networks and slow ones, all spread over both variants. Then, in the guide's words, "the only thing consistently different between the two variants is the change". So a difference in the results comes from the change. That's how an A/B test can show a cause, and a before/after can't.

The course's test went one step further. It drew 120 phone visitors and 120 desktop visitors. Then chance gave 60 of each kind photo a, and 60 photo b. So both photos got exactly 50% phones. The guide adds one rule for real sites: a visitor who comes back "should be assigned to the same variant on each successive visit".

Here is what that fair start gave:

1// p75 LCP in ms, from one week: 240 visitors, each given photo a or photo b at random.2const photos = {3  a: { everyone: 612, phones: 872, desktops: 208 },4  b: { everyone: 448, phones: 640, desktops: 176 },5};6 7for (const who of ["everyone", "phones", "desktops"]) {8  const change = photos.b[who] - photos.a[who];9  console.log(`${who}: ${photos.a[who]} to ${photos.b[who]} ms, ${change} ms (${Math.round((change / photos.a[who]) * 100)}%)`);10}

• Line 1 says how the visitors were split.
• Lines 2 to 5 are recorded: each photo's p75 LCP, for everyone and for each device.
• Line 7 goes through the three groups. Line 8 subtracts photo a's p75 from photo b's. Line 9 prints the change, and the change as a share of photo a's p75.
• It prints everyone: 612 to 448 ms, -164 ms (-27%) and phones: 872 to 640 ms, -232 ms (-27%).
• Then it prints desktops: 208 to 176 ms, -32 ms (-15%).

The smaller photo made the page faster for everyone: 164 ms faster at p75. Phones gained the most, 232 ms. The course gave phones slower networks than desktops, and a slow network takes the longest to download a big photo.

Here are the two ways side by side. Read across each row:

Why it matters: test a change as an A/B whenever you can. If you can only compare before and after, use the habits from Reading performance data: compare the same days of the week, and split by segment.

Tests for skewed data

The A/B test says photo b is 164 ms faster at p75. Could that be luck? In Reading performance data, two hours with the same page and the same mix landed 280 ms apart. So a gap of 164 ms between two groups of visitors isn't proof on its own.

Could it be chance?

Statisticians start from the dull answer: the change did nothing. That's called the null hypothesis. Kohavi's guide describes it as the variants being no different, with any gap down to "random fluctuations".

Then a test works out a p-value. NIST's handbook of statistics defines it as "the probability of the test statistic being at least as extreme as the one observed given that the null hypothesis is true." In plain words: if the photo made no difference, how often would chance give a gap at least this big? For the coin, about one time in six.

A small p-value means chance rarely does this. The usual line is 0.05, or one time in 20. A result below it is called statistically significant. The line has a price. Kohavi's guide says that at the usual setting, "5% of the time we will incorrectly conclude that there is a difference when there is none."

Why not compare the means?

The classic test for two groups is the t-test. It compares their means, their averages. But timings have a long tail, as Reading performance data showed: a few very slow visits pull the mean far up. In this A/B test, the quickest visit on each photo took 88 ms. The slowest took over 2,000 ms, more than 20 times as long. One or two visits like that can move a mean a long way.

SciPy, a Python library for science, offers another test. Its docs call the Mann-Whitney U test "a non-parametric version of the t-test for independent samples". They suggest the t-test itself when the means are "normally distributed": shaped like a bell curve. Mann-Whitney doesn't need that. It doesn't use the times at all, only their order.

Ranks: places in a line

Think of a sports day that records finishing places, not times. A runner who stops to tie a shoelace finishes last, and last is all that counts. A minute behind counts the same as a second behind. Mann-Whitney looks at visits that way. Start small, with the first 3 visitors on each photo:

1// LCP of the first 3 visitors on each photo in the A/B test, in ms.2const a = [344, 908, 160];3const b = [192, 132, 728];4 5// Line all six up, quickest first. A visit's rank is its place in the line.6const line = [...a, ...b].sort((x, y) => x - y);7const rank = (lcp) => line.indexOf(lcp) + 1;8console.log(`The line: ${line.join(", ")}`);9console.log(`Photo a's ranks: ${a.map(rank).join(", ")}`);10console.log(`Photo b's ranks: ${b.map(rank).join(", ")}`);

• Lines 2 and 3 are recorded: the LCP of the first 3 visitors on each photo.
• Line 6 puts all six in one line, quickest first.
• Line 7 gives a visit's rank: its place in that line. indexOf() counts from 0, so line 7 adds 1.
• Line 8 prints The line: 132, 160, 192, 344, 728, 908.
• Line 9 prints Photo a's ranks: 4, 6, 2. Line 10 prints Photo b's ranks: 3, 1, 5.

Ranks only care about order. The slowest visit, 908 ms, is last in the line, and a visit can't fall further back than last, however slow it was. That's why ranks suit a long tail. One of the exercises lets you try it.

Counting the pairs

Now the test itself. Pair every photo a visit with every photo b visit. Then count the pairs where photo b's visit was the quicker one:

1// The same six visits. Pair each photo a visit with each photo b visit: 3 × 3 = 9 pairs.2const a = [344, 908, 160];3const b = [192, 132, 728];4 5let u = 0;6for (const x of a) for (const y of b) if (y < x) u++;7console.log(`Photo b was quicker in ${u} of ${a.length * b.length} pairs.`);

• Line 1 says how many pairs there are: 3 visits times 3 visits makes 9.
• Line 5 starts the count, u, at 0.
• Line 6 goes through every pair. When photo b's visit y is quicker than photo a's visit x, it adds 1.
• Line 7 prints Photo b was quicker in 6 of 9 pairs.

That count is the test's U. If the photos made no difference, photo b would be quicker in about half the pairs: 4.5 of 9. Here it was quicker in 6. That's a little more than half, like seven heads in ten flips.

SciPy's code gets U from the ranks instead. Add up photo a's ranks: 4 + 6 + 2 = 12. Take away 3 × 4 ÷ 2, which is 6. That leaves 6, the same U. When two visits take exactly the same time, they share the average of their ranks, and their pair counts as half.

From U to z

Is 6 of 9 far enough from half to mean anything? To judge, the test measures how far U sits from the middle, in steps of its usual wobble. That number is z. Here it is for the first 12 visitors on each photo:

1// LCP of the first 12 visitors on each photo in the A/B test, in ms.2const a = [344, 908, 160, 1132, 140, 116, 612, 132, 172, 256, 1144, 284];3const b = [192, 132, 728, 160, 164, 584, 592, 100, 492, 640, 600, 188];4 5let u = 0;6for (const x of a) for (const y of b) u += y < x ? 1 : y === x ? 0.5 : 0;7const middle = (a.length * b.length) / 2;8const spread = Math.sqrt((a.length * b.length * (a.length + b.length + 1)) / 12);9const z = (u - middle) / spread;10console.log(`U = ${u} of ${a.length * b.length} pairs, z = ${z.toFixed(2)}: ${Math.abs(z) > 1.96 ? "a real difference" : "can't tell yet"}`);

• Lines 2 and 3 are recorded: the LCP of the first 12 visitors on each photo. That makes 12 × 12 = 144 pairs.
• Line 6 counts the pairs where photo b was quicker. A tie counts as half.
• Line 7 is the middle: half of the 144 pairs, 72. If the photos made no difference, U would land near it.
• Line 8 is the standard error: how far U usually lands from the middle by chance alone, with this many visits.
• Line 9 divides. z says how many standard errors U sits from the middle.
• Line 10 prints U = 75 of 144 pairs, z = 0.17: can't tell yet.

Lines 7 and 8 use the same formulas as SciPy's own code. SciPy also corrects line 8 for ties, and so do the course's numbers for all the visitors.

So where does 1.96 come from? NIST's table of the normal curve, the bell curve, lists it among the "critical values". Chance alone pushes z above 1.96 in 2.5% of tests, and below −1.96 in another 2.5%. That's 5% in all. So a z beyond 1.96, either way, means p is below 0.05.

0.17 is nowhere near 1.96. With 12 visitors a photo, the test can't tell. Now run it on all 120 visitors a photo: z = 2.61. That's past 1.96, so p < 0.05. More visitors don't change the photo. They shrink the wobble, the way a poll of 1,000 people beats a poll of 10, so a real gap can stand out.

A test can't fix an unfair comparison

Now run Mann-Whitney on the dashboard's two weeks. z = −2.16. Its size is past 1.96, so the test says the weeks really differ. The minus sign says which way: the week after was quicker in fewer than half the pairs. So the week after was the slower one.

The test is right. The week after's visits really were slower. But a test only rules out chance. It can't tell a slower photo from a week with more phones. Fairness has to come from how you compare: both photos at once, and chance deciding who gets which.

How big, and how sure?

A test answers "is the gap real?". It doesn't say how big the gap is. For that, use the bootstrap from Reading performance data on each photo's visits. Resample both, and subtract photo a's p75 from photo b's. Do it 500 times, and keep the middle 90% of the differences.

That middle 90% is the change's 90% interval. Here it is for both comparisons, for everyone and for each device:

Read each line against the zero line:

• A/B, everyone: −164 ms, with an interval from −272 to −76 ms. The whole line sits left of zero, so photo b really was faster.
• Before/after, everyone: +88 ms, from −132 to +216 ms. The line crosses zero, so it can't even tell which way p75 moved.
• Before/after, by device: phones sit well left of zero, and desktops sit around it. The total's dot lands right of zero because the mix changed.

Why it matters: report the interval with the change. "164 ms faster, somewhere from 76 to 272 ms" tells a reader how sure you are. "27% faster" on its own doesn't.

Try it yourself. Pick a comparison and a group of visitors. The playground's title gives the change in p75, its 90% interval, and the size of z:

The third button, Lab builds, shows a different kind of test. That's next.

Regression detection

So far, the question was: did my change help? Teams also ask a second question, every day: did some change make things worse without anyone meaning to? A change that makes a page slower is a regression.

Teams can run automatic checks on every new build of their site, a setup called continuous integration, or CI. A performance check in CI loads the page in the lab and times it. The course did the same for 36 builds in a row, with 3 runs per build. Every run used one phone-like setup: a CPU slowed down 4 times, 150 ms of latency, and a download speed of 400,000 bytes a second.

Two mistakes slipped in:

• Build 13 shipped photo c instead of photo b. That's 7,932 bytes more: a small regression.
• Build 25 shipped the original, photo a. That's another 73,356 bytes: a big one.

Here are all 108 runs, build by build:

The big jump is easy to see. The small one isn't. Why not just flag any build whose runs are slower than the build before?

Single runs wobble. Mozilla's performance sheriffs watch Firefox's test results and find its regressions. Their docs explain: "Generally a test reports values that are in a range instead of a consistent value. The larger the range of 'normal' results, the more noise we have."

Here, one run of build 12 took 536 ms. Build 12 shipped photo b, like builds 1 to 11, and its other two runs took 468 and 464 ms. That 536 was noise.

Comparing two windows

The sheriffs work from Perfherder, "a tool that takes data points from log files and graphs them over time". Its alerts don't compare single runs. At each point, its code runs what it calls "a Students t-test on the two sets of revision data". One set is the values just before the point. The other is the values from the point on. It wants at least 12 values in each.

The course's version uses 4 builds of 3 runs: 12 runs on each side. Here it is at build 25:

1// LCP of the 12 lab runs before build 25, and the 12 from build 25 on, in ms.2const before = [472, 464, 468, 464, 460, 464, 464, 464, 476, 464, 464, 476];3const after = [644, 652, 644, 644, 660, 644, 640, 644, 644, 648, 652, 652];4 5const mean = (runs) => runs.reduce((sum, run) => sum + run, 0) / runs.length;6const variance = (runs) => runs.reduce((sum, run) => sum + (run - mean(runs)) ** 2, 0) / (runs.length - 1);7const jump = mean(after) - mean(before);8const t = jump / Math.sqrt(variance(before) / before.length + variance(after) / after.length);9console.log(`The mean rose ${jump.toFixed(1)} ms. t = ${t.toFixed(1)}: ${t > 7 ? "alert" : "no alert"}`);

• Lines 2 and 3 are recorded: the 12 runs before build 25, and the 12 from build 25 on.
• Line 5 works out a mean: add the runs up, and divide by how many there are.
• Line 6 works out a variance, a measure of how spread out the runs are. It's the average squared distance from the mean, dividing by one less than the count, as Perfherder's code does.
• Line 7 is the jump: the mean after, minus the mean before.
• Line 8 is t: the jump divided by its standard error, which comes from both windows' variances. Perfherder's calc_t() uses the same formula.
• Line 9 prints The mean rose 180.7 ms. t = 81.7: alert.

Perfherder's code raises an alert when t is above 7. Then it marks the point where t is highest, higher than both its neighbours, as the likely cause. One difference: Perfherder gives the runs nearest the point more weight in its means. The course's version weights every run the same.

81.7 is far above 7. The runs barely wobble, so a jump of over 180 ms stands far out, like a heavy loaf every morning. The playground's Lab builds view works out t at every build. It peaks at build 25.

The small regression hides

Now the same windows at build 13, where photo c arrived:

1// LCP of the 12 lab runs before build 13, and the 12 from build 13 on, in ms.2const before = [480, 456, 444, 460, 456, 468, 456, 452, 460, 536, 468, 464];3const after = [460, 464, 460, 464, 468, 488, 460, 468, 468, 460, 464, 468];4 5const mean = (runs) => runs.reduce((sum, run) => sum + run, 0) / runs.length;6console.log(`Mean before: ${mean(before).toFixed(1)} ms. Mean after: ${mean(after).toFixed(1)} ms.`);

• Lines 2 and 3 are recorded: the 12 runs before build 13, and the 12 from build 13 on.
• Line 6 prints Mean before: 466.7 ms. Mean after: 466.0 ms.

The mean went down, though photo c is bigger. Look at line 2 again. Build 12's slow 536 ms run sits in the "before" window, and pulls its mean up. Without that one run, the before window's mean would be 460.4 ms, below the after window's. Put these two windows through the same t, and t is 0.1: no alert. One noisy run made a regression look like an improvement.

Mozilla's sheriffs know this trouble. A regression smaller than a test's usual range, they write, is "obvious in the graph over time, but hard to tell until you have many future data points." So take more runs. Compare all 36 runs of builds 1 to 12 with all 36 runs of builds 13 to 24:

• A t-test on the means reaches only 1.5, short of even 1.96. A few slow runs make the variance big, and t divides by it. Take out the slowest run on each side, 536 and 544 ms, and t climbs to 2.5.
• Mann-Whitney gives z = −2.63, past 1.96. The minus sign says builds 13 to 24 were quicker in fewer than half the pairs: they were slower. The rank test found the regression, slow runs and all.

Why it matters: a big regression shows up in a few runs. A small one needs many runs, and a test that a few slow runs can't push around.

Reporting results with uncertainty

The team has its answer. Now it has to tell everyone else, in a way they can check. Think of Ravi again. "I lost weight" is a claim. "I lost this much, give or take this much, weighing myself every morning on the same scale" is a result. A performance report needs the same parts:

1// The A/B test's result: each photo's p75 LCP, the 90% interval of the change, the visitors, and Mann-Whitney's z.2const result = { a: 612, b: 448, low: -272, high: -76, visits: 120, z: 2.61 };3 4console.log(`p75 LCP: ${result.a} ms to ${result.b} ms, ${result.b - result.a} ms`);5console.log(`90% interval: ${result.low} to ${result.high} ms, ${result.visits} visits per variant`);6console.log(`Mann-Whitney z = ${result.z}: ${Math.abs(result.z) > 1.96 ? "p < 0.05" : "not significant"}`);

• Line 2 is recorded: the A/B test's two p75s, the 90% interval of the change, the visitors per photo, and Mann-Whitney's z.
• Line 4 prints p75 LCP: 612 ms to 448 ms, -164 ms.
• Line 5 prints 90% interval: -272 to -76 ms, 120 visits per variant.
• Line 6 prints Mann-Whitney z = 2.61: p < 0.05.

Two warnings for anyone reading a report:

• Significant doesn't mean big. In the A/B test, desktops got 32 ms faster, with z = 2.93. That's real, and small. Whether 32 ms is worth the work is a question for the team, not for the test.
• Not significant doesn't mean no change. The first 12 visitors a photo gave z = 0.17. All 120 gave 2.61. The change was there all along.

Can you trust it?

Every result below comes from this lesson's recordings. Sort each one: trust it, wait for more visits or runs, or don't compare it this way at all.

Common mix-ups

• "Before/after is good enough." No. Here it said the page got 88 ms slower, for a change the A/B test measured as 164 ms faster.
• "A significant result proves the change caused it." No. The two weeks differed significantly, with z = −2.16, because of the phones. A test rules out chance, not an unfair comparison.
• "Significant means big." No. Desktops got z = 2.93 for a change of 32 ms.
• "Not significant means no change." No. 12 visitors a photo gave z = 0.17. 120 gave 2.61.
• "One slow run is a regression." No. Build 12's 536 ms run sat next to runs of 468 and 464 ms, on the same photo as builds 1 to 11.
• "Comparing the means is always enough." No. On 36 runs a side, t reached only 1.5. Mann-Whitney found the small regression in the same runs.
• "A percentage is a report." No. Give the interval, the visits, the test, and who it covers.

Practice exercises

Answer first, then use the hints.

// The A/B test's result for desktops: each photo's p75 LCP, the 90% interval of the change, the visitors, and Mann-Whitney's z.
const result = { a: 208, b: 176, low: -68, high: -4, visits: 60, z: 2.93 };

console.log(`p75 LCP: ${result.a} ms to ${result.b} ms, ${result.b - result.a} ms`);
console.log(`90% interval: ${result.low} to ${result.high} ms, ${result.visits} visits per variant`);
console.log(`Mann-Whitney z = ${result.z}: ${Math.abs(result.z) > 1.96 ? "p < 0.05" : "not significant"}`);

// The same visits, but photo a's slowest now took ten times as long.
const a = [344, 9080, 160];
const b = [192, 132, 728];

// Line all six up, quickest first. A visit's rank is its place in the line.
const line = [...a, ...b].sort((x, y) => x - y);
const rank = (lcp) => line.indexOf(lcp) + 1;
console.log(`The line: ${line.join(", ")}`);
console.log(`Photo a's ranks: ${a.map(rank).join(", ")}`);
console.log(`Photo b's ranks: ${b.map(rank).join(", ")}`);

// LCP of the 12 lab runs before build 26, and the 12 from build 26 on, in ms.
const before = [464, 460, 464, 464, 464, 476, 464, 464, 476, 644, 652, 644];
const after = [644, 660, 644, 640, 644, 644, 648, 652, 652, 652, 644, 640];

const mean = (runs) => runs.reduce((sum, run) => sum + run, 0) / runs.length;
const variance = (runs) => runs.reduce((sum, run) => sum + (run - mean(runs)) ** 2, 0) / (runs.length - 1);
const jump = mean(after) - mean(before);
const t = jump / Math.sqrt(variance(before) / before.length + variance(after) / after.length);
console.log(`The mean rose ${jump.toFixed(1)} ms. t = ${t.toFixed(1)}: ${t > 7 ? "alert" : "no alert"}`);

Check your understanding

For each question, ask four things. Was the comparison fair? Does the test suit skewed timings? Were there enough visits? Does the result say how sure it is?

Key takeaways

• To prove a change worked, run both versions at the same time, and let chance decide who gets which: an A/B test.
• Before/after mixes in everything else that changed. If it's all you have, compare the same days and split by segment.
• A p-value asks how often chance alone would give a gap this big. Below 0.05 is called significant.
• For skewed timings, use a rank test like Mann-Whitney to ask whether, and a bootstrap interval to say how much.
• Catch regressions by comparing windows of many runs. Small ones need more runs than big ones.
• Report the change, its interval, the visits, and the test.

Sources

The facts on this page come from these docs:

• Practical Guide to Controlled Experiments on the Web, Ron Kohavi, Randal M. Henne and Dan Sommerfield (ExP Platform).
• Critical values and p values, NIST/SEMATECH e-Handbook of Statistical Methods.
• scipy.stats.mannwhitneyu, SciPy.
• _mannwhitneyu.py, version 1.18.0, SciPy.
• Cumulative Distribution Function of the Standard Normal Distribution, NIST/SEMATECH e-Handbook of Statistical Methods.
• Bootstrap Plot, NIST/SEMATECH e-Handbook of Statistical Methods.
• Performance Sheriffing, Firefox Source Docs.
• perfalert: detect_changes, Mozilla Treeherder.

Coming next: Time to first byte, the first lesson of Fast servers, caches, and networks. It takes apart the wait before a page's first byte arrives: redirects, connection setup, and server time.