Building a RUM pipeline

From beacon to dashboard

Asha opens the mint tea page, clicks Add to cart, and leaves. As the page is hidden, it sends one beacon to the shop's server. A beacon is a small message. This one holds every number the web-vitals library measured on her visit. Sending performance data showed how to get it out of the page. Ravi's visit sends one too. So does every other visit.

Now the team ships a new release of the page. It moves the mint tea photo to a different image server. The next morning, someone asks: did the page get slower? And if it did, for whom?

Here is that morning: every visit's beacon, piled up at the shop's server.

Each card is one visit's beacon, named for its browser, its release and its number, counting from 0. So chrome-a-0 is Chrome's first visit to release a.

The beacons hold the answer. But one beacon is just a few numbers about one visit. Here is the top card's whole beacon, from the course's first visit:

1// The batch the first Chrome visit to release a sent home. Times are in ms, rounded to a tenth here.2const beacon = {3  visit: "chrome-a-0",4  release: "a",5  metrics: [6    { name: "FCP", value: 84 }, { name: "TTFB", value: 6.1 }, { name: "LCP", value: 400 },7    { name: "INP", value: 160 }, { name: "CLS", value: 0 },8  ],9};10 11console.log(`${beacon.visit} sent ${beacon.metrics.length} metrics.`);12console.log(`Its LCP was ${beacon.metrics.find((metric) => metric.name === "LCP").value} ms.`);

• Line 1 is a comment. Times are in milliseconds (ms), thousandths of a second, rounded to a tenth.
• Lines 2 to 9 hold the beacon. visit names the visit. release says which version of the page it saw.
• Lines 6 and 7 hold its five metrics. Each has a name and a value.
• Line 11 prints chrome-a-0 sent 5 metrics. Line 12 finds the LCP and prints Its LCP was 400 ms.

The five names are the metrics web-vitals measures:

• FCP, First Contentful Paint: when the first text or image appeared.
• TTFB, Time to First Byte: when the first byte of the page arrived.
• LCP, Largest Contentful Paint: when the biggest image or block of text appeared. Here, that's the photo.
• INP, Interaction to Next Paint: how long the page took to answer a click.
• CLS, Cumulative Layout Shift: a score for how far things on the page jumped.

So this visit saw the photo at 400 ms. Was that good? Was the new release worse? One beacon can't say. You need all of them, checked, kept and added up.

That's the job of a RUM pipeline. RUM, real-user monitoring, means measuring the visits of real people. Real-user monitoring explains the idea. A pipeline is a chain of steps, where each step hands its result to the next.

The course built one, and ran real visits through it:

• The shop: the mint tea page, with web-vitals on it. When the page is hidden, it sends all its metrics in one beacon, a batch.
• Two releases: release a holds the photo back for 300 ms. Release b moves it to a slower image server: 600 ms. The course's own server plays both image servers.
• A network: each visit's photo also waits a random extra time, from 0 to 897 ms, and 125.3 ms on average, like a real network.
• The visitors: Chrome, Firefox and WebKit each made 20 visits to release a, then 20 to release b. WebKit is Safari's engine, the part of a browser that turns a page into what you see. Each visit did what Asha did: load the page, click Add to cart, and leave.

The pages send the beacons. After them come four stages, each with one job. Here is each one, with what passed through it in the course's run:

• The endpoint, the address beacons are posted to, checks each one. It let all 120 batches through, and refused the 6 broken bodies the course sent it.
• Storage keeps every metric, one line each: 520 lines.
• Percentiles sum up many visits in one number, like "three visits in four were this fast or faster". This lesson works out the p50, p75 and p95.
• The dashboard shows the answer. An alert sends someone a message when a number gets worse. The course's dashboard showed 3 alerts.

The rest of the lesson follows that one beacon through each stage, and ends with the answer to the morning's question.

A collection endpoint

The first stop is the endpoint. On the course's server, it's the address /rum. The page posts its beacon there with navigator.sendBeacon(). MDN says "The data is sent as an HTTP POST request." A POST request carries its data in a body. Here, the body is the batch, written as JSON text.

The endpoint can't trust what arrives. Anything on the internet can post to a public address: an old version of your page, a broken script, someone's test. Junk that gets into storage ends up in every percentile you work out later.

Is it JSON at all?

The first checks look at the body as a whole. Is it small enough to bother with, and is it JSON?

1function refuseBody(text) {2  if (text.length > 16384) return "too big";3  try {4    JSON.parse(text);5  } catch {6    return "not JSON";7  }8  return null;9}10console.log(refuseBody("LCP=1200") ?? "fine");11console.log(refuseBody('{"visit":"chrome-a-1"}') ?? "fine");

• Line 2 refuses a body longer than 16,384 characters, before parsing it. The course picked that limit to be far bigger than any real batch.
• Lines 3 to 7 try JSON.parse(). If the text isn't JSON, it throws an error. The catch on line 5 turns that into the reason "not JSON".
• Line 8 returns null: no reason to refuse.
• Line 10 tries LCP=1200, one of the broken bodies the course sent. Line 11 tries a small piece of JSON.
• ?? picks its right side when its left side is null, so a body with no reason prints fine. They print not JSON and fine.

A beacon from a real page can't be huge anyway: MDN says its queued data "is limited to 64 KiB (65,536 bytes)." But the endpoint can't tell your page from anything else.

Is each number believable?

A body can be perfect JSON and still be nonsense. So the endpoint checks each metric in the batch too:

1const METRICS = ["TTFB", "FCP", "LCP", "INP", "CLS"];2function refuseMetric(metric) {3  if (!METRICS.includes(metric.name)) return "unknown metric";4  if (typeof metric.value !== "number" || !Number.isFinite(metric.value)) return "value is not a number";5  if (metric.value < 0) return "value out of range";6  return null;7}8console.log(refuseMetric({ name: "LCP", value: 336 }) ?? "fine");9console.log(refuseMetric({ name: "FID", value: 80 }) ?? "fine");10console.log(refuseMetric({ name: "LCP", value: -5 }) ?? "fine");11console.log(refuseMetric({ name: "LCP", value: "fast" }) ?? "fine");12console.log(refuseMetric(JSON.parse('{"name":"LCP","value":1e999}')) ?? "fine");

• Line 1 lists the five metrics the endpoint accepts. Line 3 refuses any other name.
• Line 4 refuses a value that isn't an ordinary number. Number.isFinite() is false for text, and for Infinity and NaN.
• Line 5 refuses a value below 0. None of the five metrics can be negative.
• Lines 8 to 12 try five metrics. They print fine, unknown metric, value out of range, value is not a number and value is not a number.

Line 12 is a sneaky one. JSON's standard doesn't allow Infinity as a value. But 1e999, a 1 followed by 999 zeros, is too big for a JavaScript number. So JSON.parse() turns it into Infinity, and line 4 catches it.

The course's real endpoint also refuses a batch with no visit, an unknown release, or no metrics. And it refuses any time above 60,000 ms, a whole minute, and any CLS above 100: ceilings no real visit should reach.

Adding what the page didn't send

A good batch is nearly ready to file. First, like the post room's two stamps, the endpoint adds two things it can work out for itself:

• The engine, from the user agent: the text that names the browser, sent with every request. Field data from every browser showed how to read it.
• The time it arrived. The course stored milliseconds since its server started. A real endpoint would store the date and time.

1// Part of chrome-a-0's batch, and two things only the server knows.2const batch = { visit: "chrome-a-0", release: "a" };3const metric = { name: "FCP", value: 84 };4const engine = "chrome"; // worked out from the request's user agent5const receivedAt = 2831; // ms since the server started6 7const line = { receivedAt, ...batch, engine, ...metric };8console.log(JSON.stringify(line));

• Line 2 holds the overview's visit and release. Line 3 holds its first metric.
• Lines 4 and 5 hold the two stamps, as the course's server worked them out.
• Line 7 builds one line. ...batch copies the batch's fields into the new object, and ...metric copies the metric's.
• Line 8 prints it as JSON: {"receivedAt":2831,"visit":"chrome-a-0","release":"a","engine":"chrome","name":"FCP","value":84}

That's exactly the first line the endpoint wrote to its file. Every metric in a batch becomes a line like it. Then the endpoint answers 204 No Content: MDN says the request "has succeeded", with nothing to send back. The page never reads the answer anyway. sendBeacon() only tells it whether the beacon was queued.

A bad body gets 400 Bad Request, which MDN says is for "something the server considered to be a client error." The course's endpoint sent the reason back as JSON, and stored nothing.

All 120 real batches passed. Then the course posted 6 broken bodies straight to /rum, the way a broken client might. Each got 400 and its own reason:

• not JSON, refused as not JSON
• no visit, refused as no visit
• unknown metric, refused as unknown metric
• negative value, refused as value out of range
• value is text, refused as value is not a number
• too big, refused as too big

Here is the endpoint as a picture:

Why it matters: your percentiles can only be as honest as your endpoint is strict.

Storing raw events

The endpoint has a checked, stamped line. Where does it go? Into storage, as a raw event. An event here is one metric from one visit: one LCP, or one INP. Raw means just as it came, before anything is added up or rounded.

Why not keep a running total instead? Think of a shop that keeps every receipt, not only the day's total. From the receipts, you can always work out a total, or any other number. From a total, you can never get the receipts back. Raw events are your receipts.

The course's endpoint added each event as one line of JSON at the end of a file. This format is called JSON Lines, and its site calls it "a great format for log files". Adding a line never touches the lines already there. A database table with the same columns works too.

1// The first two lines of the endpoint's file: one JSON object per metric, one per line.2const file = `{"receivedAt":2831,"visit":"chrome-a-0","release":"a","engine":"chrome","name":"FCP","value":84}3{"receivedAt":2831,"visit":"chrome-a-0","release":"a","engine":"chrome","name":"TTFB","value":6.1000001430511475}`;4 5const events = file.split("\n").map((line) => JSON.parse(line));6for (const event of events) console.log(`${event.visit}, ${event.engine}: ${event.name} ${event.value}`);

• Lines 2 and 3 are recorded: the first two lines of the endpoint's file. Both come from the overview's visit.
• Line 5 splits the file into lines, and turns each one back into an object with JSON.parse().
• Line 6 prints one sentence per event: chrome-a-0, chrome: FCP 84 and chrome-a-0, chrome: TTFB 6.1000001430511475.

Look at that TTFB, Time to First Byte. In the overview it was 6.1, rounded for showing. The file keeps exactly what the browser sent. You can always round later, when you show it, but you can't un-round.

Each line holds six fields, and each one answers a question you'll ask later:

• receivedAt: when did it arrive? You'll want to split by day.
• visit: which visit was it? It ties a visit's metrics together, so you can count visits, not lines.
• release: which version of the page? That's the morning's question.
• engine: which engine measured it? You'll compare each engine with itself.
• name and value: which metric, and its number.

After the 120 visits, the file held 520 lines in 52,654 bytes. That's about 101 bytes per event. Chrome sent five metrics per visit. Firefox and WebKit sent four, with no CLS, as Field data from every browser explained.

Why it matters: the file can answer any question, even one you haven't thought of yet. The price is its size, and the time it takes to read.

Computing percentiles

The file now holds 60 LCPs for release a, and 60 for release b. How do you turn 60 numbers into one you can compare?

It's tempting to use the average, or mean: add them all up, and divide by how many there are. web.dev warns against it: "Averages are problematic because they don't represent any single user's session." Its advice: "Whenever possible, rely on percentiles instead of averages."

That circle is a percentile. The 75th percentile, or p75, is the value that 75% of the values are at or below. The simplest way to find it is by nearest rank:

1. Sort the values, smallest first.
2. Work out the position: 75% of the count, rounded up.
3. Read the value at that position.

1// The first four Chrome visits to release a: their LCPs in ms.2const lcp = [400, 336, 452, 756];3const sorted = [...lcp].sort((a, b) => a - b);4const position = Math.ceil(0.75 * sorted.length);5 6console.log(`In order: ${sorted.join(", ")}`);7console.log(`p75 is number ${position} in line: ${sorted[position - 1]} ms`);

• Line 3 copies the four LCPs with [...lcp], and sorts the copy. sort() changes the array it sorts, so the copy leaves the original alone.
• (a, b) => a - b sorts numbers from smallest to largest. Without it, sort() compares them as text, and would put 1228 before 324.
• Line 4 works out the position: 0.75 × 4 is 3. Math.ceil() rounds up, for counts where 75% isn't a whole number.
• Line 6 prints In order: 336, 400, 452, 756. Line 7 prints p75 is number 3 in line: 452 ms. It reads sorted[position - 1], because arrays count from 0.

Now see why the mean misleads. Chrome made 20 visits to release a. One of them, chrome-a-18, drew the longest random wait of all, 897 ms. Its LCP was 1,228 ms:

1// Chrome's 20 LCPs for release a, in ms. The 1228 is visit chrome-a-18.2const lcp = [400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364];3const mean = (values) => (values.reduce((sum, value) => sum + value, 0) / values.length).toFixed(1);4const p75 = (values) => [...values].sort((a, b) => a - b)[Math.ceil(0.75 * values.length) - 1];5const others = lcp.filter((value) => value !== 1228);6 7console.log(`All 20 visits: mean ${mean(lcp)} ms, p75 ${p75(lcp)} ms`);8console.log(`Without the slowest: mean ${mean(others)} ms, p75 ${p75(others)} ms`);

• Line 2 holds Chrome's 20 recorded LCPs. Lines 3 and 4 work out a mean and a p75.
• Line 5 makes a second list, without the 1,228 ms visit.
• Line 7 prints All 20 visits: mean 503.2 ms, p75 524 ms. Line 8 prints Without the slowest: mean 465.1 ms, p75 524 ms.

One visit moved the mean by 38.1 ms. The p75 didn't move at all. Like the broken kettle, one slow visit can move the p75 by one place in line at most. Why it matters: with a mean, one unlucky visit can look like a slower page.

Now take all 60 visits to release a, and three percentiles. p50, the median, is the middle: half the visits were this fast or faster. p95 is near the slow end: only one visit in twenty was slower.

1// LCP from the 60 visits to release a, in ms: Chrome's 20, Firefox's 20, then WebKit's 20.2const lcp = [3  400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364,4  453, 331, 454, 438, 355, 344, 514, 387, 356, 408, 419, 604, 337, 528, 409, 375, 403, 327, 539, 333,5  380, 376, 852, 376, 448, 388, 440, 412, 536, 356, 784, 456, 684, 856, 348, 428, 444, 388, 628, 380,6];7 8// The p-th percentile by nearest rank: the smallest value that p% of values are at or below.9const percentile = (values, p) => [...values].sort((a, b) => a - b)[Math.max(Math.ceil((p / 100) * values.length) - 1, 0)];10console.log(`mean ${(lcp.reduce((sum, value) => sum + value, 0) / lcp.length).toFixed(1)} ms`);11for (const p of [50, 75, 95]) console.log(`p${p} ${percentile(lcp, p)} ms`);

• Lines 2 to 6 are recorded: all 60 LCPs, 20 per engine.
• Line 9 is nearest rank again, for any p.
• Line 10 prints mean 472.3 ms. Line 11 prints p50 428 ms, p75 514 ms and p95 784 ms.

The typical visit, at p50, took 428 ms. The slowest quarter took more than 514 ms. The mean, 472.3 ms, sits between them, and describes neither group.

So which one should you report? web.dev says that to check the Core Web Vitals thresholds, "you'll need your report to display the value of each metric at the 75th percentile." It adds that the 95th may be worth watching too, for "those who are on very slow networks or devices." So report the p75, and keep an eye on the p95. Reading performance data adds error bars and sample sizes.

Percentiles don't add up

Reading every line each time someone opens the dashboard gets slower as the file grows. So a pipeline can also keep summary rows: small totals for one group of events, worked out once.

The catch is what goes on the label. A count adds up, and so does a total. A p75 doesn't. Here are release a's raw events behind three rows, one per engine:

1// Release a's LCPs in ms, 20 visits per engine: the raw events behind three summary rows.2const lcp = {3  chrome: [400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364],4  firefox: [453, 331, 454, 438, 355, 344, 514, 387, 356, 408, 419, 604, 337, 528, 409, 375, 403, 327, 539, 333],5  webkit: [380, 376, 852, 376, 448, 388, 440, 412, 536, 356, 784, 456, 684, 856, 348, 428, 444, 388, 628, 380],6};7const p75 = (values) => [...values].sort((a, b) => a - b)[Math.ceil(0.75 * values.length) - 1];8const rows = Object.values(lcp).map(p75);9 10console.log(`Each engine's p75: ${rows.join(", ")} ms`);11console.log(`The average of those: ${(rows.reduce((sum, value) => sum + value, 0) / rows.length).toFixed(1)} ms`);12console.log(`The p75 of all 60 visits: ${p75(Object.values(lcp).flat())} ms`);

• Lines 3 to 5 are recorded: each engine's 20 LCPs.
• Line 8 works out one p75 per row. Line 10 prints Each engine's p75: 524, 453, 536 ms.
• Line 11 averages those three, and prints The average of those: 504.3 ms.
• Line 12 works out the p75 from all 60 raw values. It prints The p75 of all 60 visits: 514 ms.

Averaging the rows gave 504.3 ms, but the real p75 was 514 ms. A p75 depends on where every value sits in the line, and each row threw the line away.

So what can a row keep instead? The SRE book, Google's book on running production systems, has an answer: "collect request counts bucketed by latencies (suitable for rendering a histogram), rather than actual latencies". A bucket is a range, like 300 ms up to 400 ms. The row keeps how many visits fell in each bucket.

1// Three summary rows: how many of release a's LCPs fell in each 100 ms bucket.2// The buckets start at 300 ms. The last one holds everything from 1,200 ms up.3const rows = {4  chrome: [6, 7, 4, 0, 2, 0, 0, 0, 0, 1],5  firefox: [9, 7, 3, 1, 0, 0, 0, 0, 0, 0],6  webkit: [8, 6, 1, 2, 1, 2, 0, 0, 0, 0],7};8const everyone = rows.chrome.map((count, bucket) => count + rows.firefox[bucket] + rows.webkit[bucket]);9console.log(`Everyone: ${everyone.join(", ")}`);10console.log(`Visits counted: ${everyone.reduce((sum, count) => sum + count, 0)}`);

• Lines 4 to 6 are recorded: how many of each engine's LCPs fell in each 100 ms bucket.
• Line 8 adds the three rows together, bucket by bucket.
• Line 9 prints Everyone: 23, 20, 8, 3, 3, 2, 0, 0, 0, 1. Line 10 prints Visits counted: 60.

Now count along to the 45th visit, which is the p75 of 60. 23 visits were under 400 ms, 43 under 500, and 51 under 600. So the p75 is in the bucket from 500 ms up to 600 ms. The raw p75, 514 ms, is right there. Buckets can't say 514 exactly, but they find the right bucket, and they add up.

Read this table across. Each row is one thing a summary row could keep, and whether it adds up:

The same buckets make a picture of both releases:

When there are too many beacons

A busy site can get far more beacons than it needs. Then you can keep only some visits, and drop the rest. That's sampling. It works like a hand counter at the door of a busy shop. When it's too busy to count everyone, the doorman clicks for every second customer, and doubles the count when the shop closes.

In Sending performance data, the page sampled a whole visit at a time. The endpoint can sample the same way. Here, it keeps every second one of Chrome's visits:

1// Chrome's 20 LCPs in ms for each release, in the order the visits happened.2const lcp = {3  a: [400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364],4  b: [676, 964, 700, 712, 672, 652, 704, 628, 668, 676, 680, 648, 764, 764, 672, 732, 688, 664, 704, 848],5};6const p75 = (values) => [...values].sort((a, b) => a - b)[Math.ceil(0.75 * values.length) - 1];7const sample = (values) => values.filter((value, visit) => visit % 2 === 0);8 9for (const release of ["a", "b"]) {10  const kept = sample(lcp[release]);11  console.log(`Release ${release}: kept ${kept.length} of 20, p75 ${p75(kept)} ms. All 20: p75 ${p75(lcp[release])} ms.`);12}

• Lines 3 and 4 are recorded: Chrome's 20 LCPs for each release.
• Line 7 keeps visits 0, 2, 4 and so on. visit % 2 is the remainder after dividing by 2, so it's 0 for every second visit.
• Line 11 prints how many visits were kept, and both p75s: Release a: kept 10 of 20, p75 504 ms. All 20: p75 524 ms. and Release b: kept 10 of 20, p75 704 ms. All 20: p75 712 ms.

The sample's p75s are close, but not equal: 20 ms and 8 ms off. And they tell the same story. Release b was slower by 200 ms in the sample, and by 188 ms in all 20 visits.

Two rules keep a sample honest:

• Keep or drop a whole visit, never single metrics.
• Store the rate, the share of visits you keep, with each event. Scale counts by it: each of the 10 kept visits stands for 2. A p75 from a fair sample needs no scaling.

Dashboards that answer questions

Back to the morning's question: did release b make the page slower, and for whom?

A dashboard is a page of numbers and charts that people check. The SRE book says "Dashboards should answer basic questions about your service". One of the questions it says monitoring should answer is "Is my site slower than it was last week?"

So start from the question, and show only what answers it. Think of a station's departures board. It answers "is my train on time?" at a glance, and nothing else. Here, the question needs the p75 LCP for both releases, split by engine:

1// p75 LCP by engine, for release a and release b, in ms, from 20 visits each.2const p75 = {3  chrome: { a: 524, b: 712 },4  firefox: { a: 453, b: 834 },5  webkit: { a: 536, b: 768 },6};7 8for (const [engine, { a, b }] of Object.entries(p75)) {9  console.log(`${engine}: ${a} to ${b} ms, ${Math.round(((b - a) / a) * 100)}% worse`);10}

• Lines 2 to 6 are recorded: the p75 LCP for each engine and release, worked out from the stored events.
• Line 9 prints one row per engine. The change is (b - a) / a, the rise as a share of the old value, times 100.
• It prints chrome: 524 to 712 ms, 36% worse, firefox: 453 to 834 ms, 84% worse and webkit: 536 to 768 ms, 43% worse.

The answer is yes, and for everyone. Every engine's p75 LCP got worse, by 36 to 84%. The photo now came from a slower server, and every visitor waited for it.

Why split by engine? One number for everyone could hide a change that hit only one engine.

Here is the dashboard for all the stored events. Pick a metric and a percentile:

Alerts on regressions

A dashboard answers when someone looks. An alert looks for you. It's a rule that runs on the new numbers, and sends someone a message when one got worse. A change for the worse is a regression. The numbers you compare against are the baseline: here, release a.

The SRE book calls an alert sent to someone's pager a page. Its rule: "Every page should be actionable." It also warns what happens when they aren't: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts".

Here's a first rule: alert when a p75 got more than 10% worse. It runs over every metric in every engine:

1// p75 for release a and release b, by metric and engine, in ms, from 20 visits each.2const p75 = {3  "LCP chrome": [524, 712], "LCP firefox": [453, 834], "LCP webkit": [536, 768],4  "INP chrome": [152, 152], "INP firefox": [160, 160], "INP webkit": [152, 152],5  "TTFB chrome": [2.3, 2.6], "TTFB firefox": [3, 2], "TTFB webkit": [2, 1],6};7 8// Alert when the p75 got more than 10% worse.9const worse = ([before, after]) => after > before * 1.1;10const alerts = Object.entries(p75).filter(([, pair]) => worse(pair));11console.log(`${alerts.length} alerts`);12for (const [name, [before, after]] of alerts) console.log(`${name}: ${before} to ${after} ms`);

• Lines 2 to 6 are recorded: the p75 for release a and release b, for three metrics in three engines.
• Line 9 is the rule: the new p75 is more than 1.1 times the old one.
• Line 10 keeps the pairs where the rule fires. Line 11 prints 4 alerts.
• Line 12 prints each one: LCP chrome: 524 to 712 ms, LCP firefox: 453 to 834 ms, LCP webkit: 536 to 768 ms and TTFB chrome: 2.3 to 2.6 ms.

The first three are real. The last one is toast. Chrome's TTFB got 13% worse, but by 0.3 ms, less than a thousandth of a second. Nobody could notice that, and nobody could fix it. A share alone can't tell a big change from a tiny number getting a little bigger. So the course's rule also asks for an amount: at least 50 ms worse. The exercises let you run it.

There's a second trap: too few values. Watch what a p75 does with four visits at a time, when nothing changed at all:

1// WebKit's 20 LCPs for release a, in ms, in the order the visits happened.2const lcp = [380, 376, 852, 376, 448, 388, 440, 412, 536, 356, 784, 456, 684, 856, 348, 428, 444, 388, 628, 380];3const p75 = (values) => [...values].sort((a, b) => a - b)[Math.ceil(0.75 * values.length) - 1];4 5for (let start = 0; start < lcp.length; start += 4) {6  console.log(`Visits ${start} to ${start + 3}: p75 ${p75(lcp.slice(start, start + 4))} ms`);7}8console.log(`All 20 visits: p75 ${p75(lcp)} ms`);

• Line 2 holds WebKit's 20 recorded LCPs for release a, in the order the visits happened.
• Lines 5 to 7 work out a p75 for each group of four visits. Line 8 works one out for all 20.
• Line 6 prints Visits 0 to 3: p75 380 ms, Visits 4 to 7: p75 440 ms and Visits 8 to 11: p75 536 ms.
• Then Visits 12 to 15: p75 684 ms and Visits 16 to 19: p75 444 ms. Line 8 prints All 20 visits: p75 536 ms.

Same page, same release. Yet the p75 of four visits ran from 380 to 684 ms. Compare visits 0 to 3 with visits 12 to 15, and it looks 80% and 304 ms worse: enough for both the share and the amount. With four values, the p75 is the third in line, so two slow visits make it slow.

So the course's full rule has three parts. It fires only when release b's p75 is:

• more than 10% worse than the baseline,
• and at least 50 ms worse,
• with at least 15 values on each side.

The dashboard playground uses this rule in its Alert column. The three numbers are the course's own choice. Pick an amount your users would notice, and enough values to steady the p75. Proving a change worked goes further, with tests made for skewed data like this.

Which stage?

Each job below belongs to one stage of the pipeline. Sort it into the stage that does it.

Common mix-ups

• "The mean is good enough." No. One slow visit moved Chrome's mean by 38.1 ms, and its p75 not at all.
• "Storing the summaries is enough." No. Raw events can answer tomorrow's question too.
• "Average each day's p75 to get the month's." No. Release a's three engine p75s averaged 504.3 ms, but the real p75 was 514 ms. Add bucket counts instead.
• "Every beacon that arrives is real." No. Anything can post to a public address. Six broken bodies were refused.
• "Any percentage rise is a regression." No. Chrome's TTFB rose 13%, which was 0.3 ms. A rule needs an amount too.
• "A sample's counts are the real counts." No. Each kept visit stands for several. With one visit in two, double the counts.
• "One dashboard should show everything." No. Start from one question, and show what answers it.

Practice exercises

Answer first, then use the hints.

// p75 for release a and release b, by metric and engine, in ms, from 20 visits each.
const p75 = {
  "LCP chrome": [524, 712], "LCP firefox": [453, 834], "LCP webkit": [536, 768],
  "INP chrome": [152, 152], "INP firefox": [160, 160], "INP webkit": [152, 152],
  "TTFB chrome": [2.3, 2.6], "TTFB firefox": [3, 2], "TTFB webkit": [2, 1],
};

// Alert when the p75 got more than 10% worse.
const worse = ([before, after]) => after > before * 1.1 && after - before >= 50;
const alerts = Object.entries(p75).filter(([, pair]) => worse(pair));
console.log(`${alerts.length} alerts`);
for (const [name, [before, after]] of alerts) console.log(`${name}: ${before} to ${after} ms`);

// LCP from the 60 visits to release a, in ms: Chrome's 20, Firefox's 20, then WebKit's 20.
const lcp = [
  400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364,
  453, 331, 454, 438, 355, 344, 514, 387, 356, 408, 419, 604, 337, 528, 409, 375, 403, 327, 539, 333,
  380, 376, 852, 376, 448, 388, 440, 412, 536, 356, 784, 456, 684, 856, 348, 428, 444, 388, 628, 380,
];

// The p-th percentile by nearest rank: the smallest value that p% of values are at or below.
const percentile = (values, p) => [...values].sort((a, b) => a - b)[Math.max(Math.ceil((p / 100) * values.length) - 1, 0)];
console.log(`mean ${(lcp.reduce((sum, value) => sum + value, 0) / lcp.length).toFixed(1)} ms`);
for (const p of [99]) console.log(`p${p} ${percentile(lcp, p)} ms`);

// Chrome's 20 LCPs in ms for each release, in the order the visits happened.
const lcp = {
  a: [400, 336, 452, 756, 352, 324, 504, 476, 452, 524, 328, 716, 384, 440, 536, 492, 436, 564, 1228, 364],
  b: [676, 964, 700, 712, 672, 652, 704, 628, 668, 676, 680, 648, 764, 764, 672, 732, 688, 664, 704, 848],
};
const p75 = (values) => [...values].sort((a, b) => a - b)[Math.ceil(0.75 * values.length) - 1];
const sample = (values) => values.filter((value, visit) => visit % 4 === 0);

for (const release of ["a", "b"]) {
  const kept = sample(lcp[release]);
  console.log(`Release ${release}: kept ${kept.length} of 20, p75 ${p75(kept)} ms. All 20: p75 ${p75(lcp[release])} ms.`);
}

Check your understanding

For each question, ask which stage of the pipeline it's about, and what would go wrong without it.

Key takeaways

• A RUM pipeline turns beacons into answers: an endpoint, storage, percentiles, then a dashboard and alerts.
• The endpoint refuses a bad body with 400 and a reason. It stamps a good one with the engine and the time, files it, and answers 204.
• Store every metric raw, one line each, with the value exactly as sent. Round only when you show it.
• Report the p75, watch the p95, and don't trust the mean: one slow visit can drag it.
• A p75 doesn't add up. Summary rows keep counts, totals and counts per bucket, which do.
• Sample whole visits, store the rate, and scale counts by it, not percentiles.
• Build each dashboard around one question, split by what matters, like engine and release.
• Alert only when the p75 is worse by a share and by an amount, from enough values.

Sources

The facts on this page come from these standards and docs:

• Navigator: sendBeacon() method, MDN.
• RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format, IETF.
• 204 No Content, MDN.
• 400 Bad Request, MDN.
• JSON Lines, jsonlines.org.
• Best practices for measuring Web Vitals in the field, web.dev.
• Array.prototype.sort(), MDN.
• Monitoring Distributed Systems, Google SRE book.

Coming next: Segmenting field data. Split the same data by device, network, country, page type and navigation type, to find who is slow and why.