Monitoring and alerting
Watch real visits after every release: dashboards for engineers and for leaders, release markers, p75 alerts that ignore noise, and a synthetic check on a schedule.
- 01Build dashboards people readSplit field data by device and LCP element for engineers, and give leaders a weekly trend instead of a pass or fail.
- 02Mark every releasePut code releases and tag manager versions on the charts, and in every beacon, so a change lines up with its cause.
- 03Alert on what's realAlert on the p75, skip hours with too few visits, wait for a lasting change, and schedule a synthetic check that sees what RUM can't.
A slowdown no pull request showed
It's Friday morning at the Leaf and Kettle. Ravi opens the team's dashboard with his first cup of tea. One line on it has jumped. On phones, the Mint tea page got slower on Thursday, and it stayed slow.
That's strange. In lesson 39.2, Ravi's team set up checks that run on every pull request: each change someone asks to merge into the site's code. All week, every check passed. So what changed? And why did no check see it? Here are the two numbers that worried him:
// p75 LCP of phone visits, in msconst wednesday = 640;const friday = 1360; const times = (friday / wednesday).toFixed(1);console.log(`Friday's p75 was ${times} times Wednesday's`);console.log(friday <= 2500 ? "Both are under 2,500 ms: good" : "Friday is over 2,500 ms");- Line 1 is a comment. LCP, Largest Contentful Paint, is when the biggest image or block of text appears. A millisecond (ms) is a thousandth of a second.
- Lines 2 and 3 hold the p75 of each day's phone visits. That's the time three visits in four were at or under.
- Line 5 divides Friday's number by Wednesday's. Line 6 prints
Friday's p75 was 2.1 times Wednesday's. - Line 7 checks Google's line for a good LCP, 2,500 ms. It prints
Both are under 2,500 ms: good.
So for phone visitors, the wait more than doubled. Yet by Google's line, the page is still good. Why it matters: a number that only says pass or fail would have said nothing was wrong.
The answer sits in the tag manager. A tag manager lets Meera add scripts to the page from a website, without changing Ravi's code. Each time she publishes, a new version goes live, as lesson 24.3 showed. On Thursday at 10:20, she published version 22. It added a newsletter pop-up. There was no pull request, so no check ran. Here is what a phone showed, on Wednesday and on Thursday:
On Thursday, the tea photo still came on time. Then the pop-up appeared, with a bigger photo. LCP counts the biggest thing on the screen, whoever put it there. So the pop-up's photo became the LCP, and the LCP came later. web.dev warns about the same thing with cookie notices: on phones, they cover more of the screen, and they can become the LCP.
Why it matters: a tag can change your LCP without touching your code. Only a check that watches the live page can see it. Here is the whole week:
Phones jumped on Thursday and stayed up. Computers didn't move. On a wide screen, the pop-up is a small card in the corner, smaller than the tea photo. And Ravi's own release on Tuesday, web-41, changed nothing.
You don't stare at the fuel gauge the whole drive. You glance at it now and then. When fuel runs low, a light comes on, and it stays on until you fill up. A light that flashed at every bump would soon be ignored. And each time the car comes back from the garage, the repair goes in the service book.
- In real life: The gauges you glance at
- In monitoring: A dashboard of real visits
- In real life: The low fuel light
- In monitoring: An alert
- In real life: A light that stays on, not one that flickers at every bump
- In monitoring: An alert that fires only for a lasting change
- In real life: The service book
- In monitoring: Release markers: what changed, and when
Monitoring means watching real visits to the live page, all the time. Mark every release on the charts, so a change lines up with its cause. Send an alert only when a slowdown is real, lasts, and is measured on enough visits. Checks on pull requests see your code before it ships. Monitoring sees whatever reaches visitors: code, tags, servers and networks.
The rest of the lesson answers four questions:
- Who reads a dashboard, and what should it show them?
- How do you know which release did it?
- When should an alert fire, and when should it stay quiet?
- How often should a robot check the page, and what can't it see?
How the course recorded it
- The week: one week of the live page, squeezed into one run in Chrome 157. The course made every visit, in order: 1,386 of them, few at night and most in the evening. Each phone and computer had its own speed and network, drawn the way lesson 13.1 drew them.
- The visitors: about one in three was new. The rest were regulars who already get the newsletter, so their browser had the shop's newsletter cookie. LCP stops counting at a visitor's first tap or scroll, as web.dev explains. So each visitor waited until the page had settled, then followed a link.
- The page: lesson 39.2's Mint tea page, with the tag manager's script. Its rum.js sent each visit's numbers home, as in lesson 12.4.
- The sites: the shop, the tag manager and the pop-up each had a made-up address, served over HTTPS from the course's computer. The tag manager and the pop-up are stand-ins. The pop-up is the size of lesson 24.3's pop-up, and its photo is 92 kB.
- The robot: every hour, a fresh Chrome loaded the page as Asha's phone: Slow 4G, the CPU 4 times slower, and no cookies.
Dashboards for engineers and for leaders
A dashboard is a page of charts built from the beacons: the small messages each visit sends home with its numbers. Lesson 12.4 built one to answer a question. web.dev's guide to measuring adds one more thought: think about who needs to see the data. At the Leaf and Kettle, two kinds of people do.
In a café kitchen, the cooks watch a screen of live orders: what's waiting, for how long, and for which table. The owner doesn't. Once a week, she reads one sheet: sales, waiting times, and how they compare with last week. Same café, same orders, two very different views.
- In real life: The cooks' live screen
- In monitoring: The engineers' dashboard: daily and hourly, split many ways
- In real life: Which table each order is for
- In monitoring: Splits: device, LCP element, release
- In real life: The owner's weekly sheet
- In monitoring: The leaders' report: a few numbers, once a week
- In real life: "Compared with last week"
- In monitoring: The trend, not just pass or fail
What engineers need
Engineers need to find the cause. So their dashboard splits the numbers. Here are the phone visits after Meera's release, split by the element their LCP came from:
// Phone visits from Thursday 10:20 on, by the LCP element each beacon namedconst elements = { "html>body>main>img.hero": 387, "html>body>aside.offer>img.offer-photo": 207,}; for (const [element, visits] of Object.entries(elements)) { console.log(`${element}: ${visits} visits`);}- Lines 3 and 4 hold the counts. Each name is a selector: a CSS-style address of the element, from the page down to the image. Ravi's beacon sends it, as the next section shows.
- Lines 7 to 9 print one line for each element:
html>body>main>img.hero: 387 visitsandhtml>body>aside.offer>img.offer-photo: 207 visits.
207 of 594 phone visits named the pop-up's photo: the new visitors. Regulars never saw the pop-up. Why it matters: the element tells Ravi where to look before he opens a single trace. Now split the whole week yourself. Each row shows the p75 before and after Meera's release:
- Every visitbefore 564 ms (647 visits), after 1,208 ms (739 visits)
One number for every visit, phones and computers together. Each row shows the p75 LCP before Thu 10:20 and after it.
Every visit together still shows the jump, because most visits come from phones. By device, you see that only phones changed. By LCP element, the cause has a name. Next to each number, show how many visits it came from. As lesson 13.1 showed, a p75 of a few visits is little more than a guess.
What leaders need
Leaders don't need hourly charts. They need to know if the site is getting faster or slower, and why. web.dev's article on fixing site speed across a company says it plainly: "what's not reported will not be focused on." It suggests putting speed in the weekly report to the business. Read this table across: each row asks one question of both dashboards.
| Question | Engineers' dashboard | Leaders' report |
|---|---|---|
| Who reads it? | Ravi, Ben, and anyone fixing the page. | The shop's owners, Meera, and anyone deciding what to work on. |
| How often? | Every day, and every hour after a change. | Once a week. |
| Which numbers? | p75 of LCP, INP and CLS, with the visits behind each one. | A few: phones' p75 LCP this week, last week, and the change. |
| Split how? | By device, LCP element and release. | By device at most. |
| What is it for? | Finding the cause of a change. | Seeing the trend, and deciding where to spend time. |
One trap for leaders is pass or fail. Ravi's phones went from 640 to 1,360 ms, and stayed good. A report that only said "good" would have shown nothing. So give leaders the trend: this week, last week, and the change.
Google's numbers move slowly
Many leaders look at Google's numbers, in Search Console and PageSpeed Insights. Both come from CrUX, the data Chrome collects from real visits, from lesson 10.1. And both cover the past 28 days. The CrUX History API keeps one point a week. It's updated each Monday, with data up to the Saturday before. Here is what those points would show for Ravi's phones:
// Phones' p75 LCP over 28 days, at each weekly point, in msconst points = [ { newDays: 0, p75: 640 }, { newDays: 2.6, p75: 664 }, { newDays: 9.6, p75: 760 }, { newDays: 16.6, p75: 1008 }, { newDays: 23.6, p75: 1276 }, { newDays: 28, p75: 1428 },]; for (const { newDays, p75 } of points) { console.log(`${newDays} of 28 days after the release: ${p75} ms`);}- Lines 3 to 8 hold one point a week.
newDaysis how many of its 28 days came after the release. Each day before it counts like an average day of the week before 10:20 on Thursday, and each day after it like an average day after. - Line 3 is the Saturday before the release. Line 4 is the first Saturday after it: only 2.6 of its 28 days are new.
- Lines 11 to 13 print one line a point, from
0 of 28 days after the release: 640 msto28 of 28 days after the release: 1428 ms.
The first point after the release barely moves. It takes four more weeks to show the whole change. Why it matters: CrUX tells you how the month went. To catch a slowdown on the day, you need your own field data. web.dev recommends it for exactly this: "more detailed and immediate feedback".
Marking every release
On Friday, Ravi's first question was one the SRE book, Google's book on running services, lists: "what else happened around the same time?" A release marker answers it. It's a line or a tick on a chart, at the moment something changed.
Ana keeps a garden diary. The day she switched to a new plant food, she wrote it down. Weeks later, the tomato leaves turn yellow. She flips back, finds the note, and knows where to start. Without it, she'd be guessing between the weather, the soil and the seeds.
- In real life: A note in the diary
- In monitoring: A release marker on the chart
- In real life: Switching to a new plant food
- In monitoring: A release: code, tags or settings
- In real life: Yellow leaves, weeks later
- In monitoring: A slower p75 on the dashboard
- In real life: Flipping back to the note
- In monitoring: Lining the change up with its marker
What counts as a release? The SRE book calls a change a push: "Any change to a service's running software or its configuration." That's more than Ravi's code:
- Code: mark each deploy. SpeedCurve, a monitoring service, can test the site each time you add a deploy, from your CI workflow or by hand. Sentry, an error and performance tool, ties each event to the release name sent with it.
- Tags: Google Tag Manager keeps a publish history, "so you can see when versions were live and who published them."
- Everything else: a new setting on the CDN, the servers that deliver your files, or a new image format. DebugBear, another monitoring service, suggests a chart annotation for changes like these.
Ravi's week has two markers: web-41 on Tuesday at 14:00, and tags version 22 on Thursday at 10:20. Only the second lines up with the jump.
Put the release in every beacon
A marker says when. Ravi also wants each visit to say what it ran. So his rum.js now sends that with every beacon:
import { onCLS, onINP, onLCP } from "web-vitals/attribution";import { RELEASE } from "./release.js"; const metrics = []; export function startRum() { onLCP((metric) => metrics.push({ name: "LCP", value: metric.value, element: metric.attribution.target })); onINP((metric) => metrics.push({ name: "INP", value: metric.value })); onCLS((metric) => metrics.push({ name: "CLS", value: metric.value })); addEventListener("visibilitychange", () => { if (document.visibilityState !== "hidden" || metrics.length === 0) return; const beacon = { release: RELEASE, tags: window.tagManager?.version ?? null, device: matchMedia("(max-width: 600px)").matches ? "phone" : "desktop", metrics: metrics.splice(0), }; navigator.sendBeacon(`rum${location.search}`, JSON.stringify(beacon)); });}- Line 1 imports from web-vitals' attribution build: a slightly bigger version of the library whose metrics also say why, in an
attributionobject. - Line 2 imports the release's name. The build writes it into release.js: here,
web-41. - Line 7 keeps LCP's value and its element,
metric.attribution.target: the selector you saw above. - Lines 12 to 17 build the beacon. Line 13 adds the release, line 14 the tag manager's version, and line 15 the device. The stand-in tag manager puts its version in
window.tagManager. Google Tag Manager has a built-in variable, Container Version, for the same job. - Line 18 sends it with
navigator.sendBeacon()when the page is hidden, as in lesson 12.4.
Now the visits can be grouped by what was live:
// p75 LCP of phone visits, by what was live, in msconst live = [ { release: "web-40", tags: 21, p75: 664 }, { release: "web-41", tags: 21, p75: 612 }, { release: "web-41", tags: 22, p75: 1428 },]; for (const { release, tags, p75 } of live) { console.log(`${release} with tags ${tags}: ${p75} ms`);}- Lines 3 to 5 hold the p75 for each mix of release and tags version the week had.
- Lines 8 to 10 print
web-40 with tags 21: 664 ms,web-41 with tags 21: 612 msandweb-41 with tags 22: 1428 ms.
Ravi's own release changed nothing. The tags version did. Why it matters: with the release in every beacon, "which change was it?" takes one split, not an afternoon of guessing.
Alert thresholds on percentiles
The dashboard showed the jump on Friday. Ravi wants to hear about it on Thursday. And nobody should sit and watch charts: the SRE book says its teams avoid any job that needs someone to "stare at a screen to watch for problems." That's what an alert is for. It's a rule that runs every hour on the new beacons, and sends someone a message when it fires.
Lesson 12.4 wrote a rule that compared two releases. This one runs on a clock. It needs three choices:
- which number to watch;
- how many visits an hour needs before the rule reads it;
- how long a slowdown must last before anyone is told.
A parent takes a child's temperature right after a game of tag. It's a little high, but running about warms anyone up. So they let the child rest, and take it again an hour later. Only two high readings in a row, taken at rest, make them call the doctor.
- In real life: One reading
- In monitoring: One hour's p75
- In real life: Right after a game of tag
- In monitoring: An hour with only two or three visits
- In real life: Waiting for a reading at rest
- In monitoring: Skipping hours with too few visits
- In real life: Two high readings in a row
- In monitoring: Two hours in a row over the line
- In real life: Calling the doctor
- In monitoring: Sending the alert
Which number?
Here is the same hour, 11:00 to 12:00, on Wednesday and on Thursday. Thursday's is the first whole hour after the release:
// The phone visits between 11:00 and 12:00, in msconst wednesday = [1232, 704, 244, 696, 296, 724, 1060, 276];const thursday = [608, 728, 260, 3172, 1340, 252, 212, 1448]; const mean = (lcps) => Math.round(lcps.reduce((sum, lcp) => sum + lcp, 0) / lcps.length);const rank = (lcps, share) => [...lcps].sort((a, b) => a - b)[Math.ceil(lcps.length * share) - 1];for (const [day, lcps] of Object.entries({ wednesday, thursday })) { console.log(`${day}: mean ${mean(lcps)}, p50 ${rank(lcps, 0.5)}, p75 ${rank(lcps, 0.75)}`);}- Lines 2 and 3 hold the LCPs of that hour's phone visits, in the order they came.
- Line 5 works out the mean, the average. Line 6 sorts a copy and picks a rank, as lesson 13.1 counted percentiles. The p50 is the middle one. The p75 is the one three quarters of the way up.
- It prints
wednesday: mean 654, p50 696, p75 724andthursday: mean 1003, p50 608, p75 1340.
In this hour, the p50 even went down. With 8 visits, the middle ones were regulars, who never saw the pop-up. The p75 jumped: 3 of the 8 visits saw the pop-up, and the p75 sits among the slowest quarter. The mean rose too, mostly because one visit took 3,172 ms.
The whole week shows the same pattern. After the release, phones' p50 went from 432 to 668 ms. The p75 went from 640 to 1,428 ms.
The SRE book warns that "a simple average can obscure these tail latencies, as well as changes in them." It says the median, the p50, "emphasizes the typical case." web.dev adds that the median "can mask" slow visits. That's why Core Web Vitals, Google's main speed numbers, are judged at the 75th percentile. Why it matters: a slowdown that hits a third of your visitors shows clearly in the p75.
Where to draw the line
Ravi uses lesson 39.1's rule again: the slowest value he has, rounded up to the next 100 ms. He takes the hours from Monday to Wednesday with at least 8 phone visits. Their slowest p75 gives a line of 900 ms. The same rule gives 800 ms for the mean and 700 ms for the p50.
Too few visits
Now the quiet hours. At night, only two or three phones visit. Here is one of those hours:
// The 2 phone visits between 01:00 and 02:00 on Thursday, in msconst lcps = [1884, 412];const line = 900; const p75 = [...lcps].sort((a, b) => a - b)[Math.ceil(lcps.length * 0.75) - 1];console.log(`p75 of ${lcps.length} visits: ${p75} ms`);console.log(p75 > line ? "Over the line: alert!" : "Under the line");- Line 2 holds the visits. Line 3 is Ravi's p75 line.
- Line 5 picks the p75. With 2 visits, that's rank 2 of 2: the slower one.
- It prints
p75 of 2 visits: 1884 msandOver the line: alert!
One slow phone in the middle of the night, and the rule fires. Nothing had changed on the page. A p75 of a few visits is mostly luck, as lesson 13.1 showed. Google's SRE workbook says alerting gets "particularly sensitive to nonsignificant events during low-traffic periods." So Ravi's rule skips any hour with fewer than 8 phone visits.
Two hours in a row
One more guard. A single bad hour can be bad luck too, such as a few slow phones at once. So the rule waits for two hours in a row over the line. Prometheus, an open-source monitoring tool, does this with a for setting. Until the time is up, its alert is "pending". SpeedCurve can delay an alert "until a threshold is crossed multiple times."
// Thursday, hour by hour: phone visits, and their p75 in msconst hours = [ { until: "09:00", visits: 8, p75: 532 }, { until: "10:00", visits: 8, p75: 400 }, { until: "11:00", visits: 8, p75: 816 }, { until: "12:00", visits: 8, p75: 1340 }, { until: "13:00", visits: 10, p75: 1036 }, { until: "14:00", visits: 8, p75: 1244 },];const line = 900; let inARow = 0;for (const hour of hours) { if (hour.visits < 8) continue; inARow = hour.p75 > line ? inARow + 1 : 0; if (inARow === 2) console.log(`Alert at ${hour.until}: two hours in a row over ${line} ms`);}- Lines 3 to 8 hold Thursday's hours: when each one ended, its phone visits, and their p75.
- Line 10 is the line. Line 12 counts how many hours in a row were over it.
- Line 14 skips an hour with fewer than 8 visits. Line 15 adds one when the p75 is over the line, or starts again from 0.
- Line 16 prints
Alert at 13:00: two hours in a row over 900 ms.
Meera published at 10:20. The alert went out at 13:00, under three hours later. Why it matters: Ravi heard about it on Thursday, not on Friday morning.
Waiting has a price, and Google's SRE workbook spells it out. The alert comes later. And if the value dips under the line for one hour, the count starts again, so a slowdown that flickers might never alert. The workbook prefers a longer window instead, such as the last few hours read together. Both ideas do the same job: they put more visits behind each decision.
Now try five rules on the whole week. Each square is one hour of phone visits:
- Linemean over 800 ms
- False alarms, before the release5: Mon 05:00, Tue 00:00, Tue 02:00, Wed 02:00, Thu 02:00
- First alert after the releaseThu 12:00
- Alerts sent after the release9
Each square is one hour of phone visits, Monday at the top. Filled squares were over the line and faint ones under it. An alert is sent when an hour is over, and stays on until an hour is under. A red ring marks each alert, and the dashed square is the hour of Meera's release.
Watch two things: how many alerts each rule sends before the release, when nothing had changed, and when it first fires after it. One more setting matters. SpeedCurve sends an alert only when a budget's status changes: "Repeated overages do not trigger repeated alerts." Ravi's rule works the same way: one message when a slowdown starts, not one every hour. Why it matters: the SRE book warns that when alerts come too often, people "second-guess, skim, or even ignore" them.
Synthetic monitoring on a schedule
Ravi's alert needs real visits, and at 3 in the morning there are hardly any. So he also points a robot at the live page. In lesson 39.2, a synthetic test was a scripted visit in a lab, on every pull request. A synthetic monitor is the same kind of visit, on a timer, against the live site. Ravi's robot checks the page every hour, as Asha's phone, with no cookies.
A chain of cafés sends a mystery shopper to each shop at set times. She orders the same tea every visit and times the queue. If the queue gets longer, the owner knows by the next visit. But she's one person ordering one drink. She never sees how the regulars are treated.
- In real life: The mystery shopper
- In monitoring: The robot: one scripted visit
- In real life: The same order every time
- In monitoring: The same phone and network, and no cookies
- In real life: Her set times
- In monitoring: The schedule
- In real life: The regulars she never sees
- In monitoring: Real visitors the robot can't stand in for
The robot's line comes from the same rule: its slowest check from Monday to Wednesday, rounded up, 1,400 ms. Those 72 checks ranged from 1,288 to 1,340 ms, much closer together than real visits. Lab loads still wobble, as lesson 39.2 found. On Thursday at 07:00, one check took 1,408 ms, just over the line, with nothing changed. The next one took 1,312 ms. So the robot gets Ravi's rule too: two checks in a row over the line.
On Wednesday at 11:00, the robot's LCP was 1,296 ms. Thursday at 11:00 was its first check after Meera's release: 3,540 ms, with the pop-up's photo as the LCP. Every check after it took 3,524 to 3,696 ms. So the robot's alert went out at 12:00, an hour before the alert on real visits.
Why it matters: the robot doesn't wait for visitors. It checks the same way every time, so its numbers stay close together, and two checks over the line mean something.
How often?
More checks cost more. DebugBear's docs say once a day is enough for most sites. They suggest more often if you deploy several times a day, so you're "alerted to regressions more quickly", or if your numbers wobble a lot. SpeedCurve's docs give an example: your own site 3 times a day, and a competitor's once, to keep your monthly checks down.
const release = 10 + 20 / 60; // Thursday 10:20, in hours for (const every of [1, 6, 24]) { const next = Math.ceil(release / every) * every; console.log(`Every ${every} h: ${168 / every} checks a week, the first ${(next - release).toFixed(1)} h after the release`);}- Line 1 is Thursday 10:20, in hours since midnight.
- Line 4 rounds up to the next check. With a check every 6 hours, that's 12:00.
- Line 5 prints one line for each schedule. The first is
Every 1 h: 168 checks a week, the first 0.7 h after the release. Every 6 hours, the first check comes 1.7 h after it. Once a day, 13.7 h after.
Pick a schedule, and see which of the robot's real checks it keeps:
- Checks a week168
- Last check before the releaseThu 10:00: LCP 1,296 ms
- First check after itThu 11:00: LCP 3,540 ms, the pop-up's photo
- Time from release to that check0 h 40 min
- Checks over the line before the release1: Thu 07:00
- Alert, at the second check in a row over the lineThu 12:00
The robot's line is 1,400 ms. Fewer checks cost less, and the first one after Meera's release comes later.
You can also check right after a change. SpeedCurve can run its tests each time you add a deploy, from your CI workflow or by hand. But Meera's publish wasn't a deploy, so only the schedule caught it.
What the robot can't see
The robot found the pop-up within the hour. The field data, from real visits, told Ravi the rest:
- Who was hit. The robot is always a new visitor. In the field, 207 of 594 phone visits had the pop-up's photo as their LCP. The regulars had the cookie.
- How badly. The robot has one phone on one network. In the field, one new visitor in ten waited over 2,716 ms.
- Computers. The robot only checks a phone. Only the field could say computers didn't change.
- Taps. The robot never taps, so it has no INP, the time the page takes to answer a tap. web.dev notes that some lab tools "won't report a page's INP because they only observe the loading of a page without any interactions."
Google's SRE workbook warns about the other side too. If a problem hits real users but not the robot, "the successful artificial requests hide the real user signal." A/B tests, which show different visitors different versions, add noise too. SpeedCurve warns that its robots "end up in random test buckets." Read this table across to compare the robot with real-user monitoring (RUM):
| Question | The hourly robot | Real-user monitoring |
|---|---|---|
| Who visits? | One made-up visitor: Asha's phone, on Slow 4G, every time. | Every real visitor, on their own phone or computer and network. |
| Cookies? | None. It's a new visitor on every check. | Whatever each visitor has. Regulars carry the newsletter cookie. |
| When? | On its schedule, even at 3 in the morning with nobody there. | Only when people visit. Quiet hours have few visits. |
| Does it tap or scroll? | No. It loads the page and waits, so it has no INP. | Yes. Real taps give each visit its INP. |
| How soon does it notice? | At its next check: here, 40 minutes after the release. | Once enough visits pile up: here, under three hours. |
| What can't it tell you? | How many real visitors were hit, and how badly. | Why: it needs the release markers and the LCP element to say. |
Why it matters: use both. The robot is quick, steady, and awake at night. The field says who was hit, and how badly.
Which check sees it first?
Ravi now has three ways to catch a slowdown: the checks on each pull request, the hourly robot, and alerts on real visits. Sort each problem by the one that would see it first:
- A pull request adds a big date library to app.js
- A pull request adds
loading="lazy"to the tea photo - Meera publishes a pop-up for new visitors from the tag manager
- The shop's server gets slow at 3 in the morning
- A tag slows the page only for people with the newsletter cookie
- Taps on the brewing timer get slow on old phones
- The page gets slow on one mobile network in one country
Sort each problem by the check that would notice it first.
Common mix-ups
- "If every pull request passes its checks, the live site can't get slower." No. Meera's pop-up went live from the tag manager, with no pull request. Phones' p75 more than doubled.
- "The average shows every slowdown." No. A mean moves for one very slow visit as easily as for many, so it fires on noise. And the SRE book warns it can hide changes in the slowest visits.
- "The p50 is the safest number to alert on." No. It shows the typical visit. In the hour after the release, it went down while a third of the visits got slower.
- "Every hour's p75 is worth an alert." No. With 2 visits, the p75 is the slower of the two. Skip hours with too few visits.
- "If Search Console says good, nothing got slower." No. It covers 28 days, and its line for good is 2,500 ms. Ravi's phones stayed good while their p75 more than doubled.
- "A synthetic monitor sees what visitors see." No. It's one visitor, on one phone, with no cookies. It can't say how many real visitors were hit.
- "More alerts keep you safer." No. When alerts come too often, people learn to ignore them.
Practice exercises
Answer first, then use the hints.
Ravi tries waiting for three hours in a row. What does it print?
// Thursday, hour by hour: phone visits, and their p75 in ms
const hours = [
{ until: "09:00", visits: 8, p75: 532 },
{ until: "10:00", visits: 8, p75: 400 },
{ until: "11:00", visits: 8, p75: 816 },
{ until: "12:00", visits: 8, p75: 1340 },
{ until: "13:00", visits: 10, p75: 1036 },
{ until: "14:00", visits: 8, p75: 1244 },
];
const line = 900;
let inARow = 0;
for (const hour of hours) {
if (hour.visits < 8) continue;
inARow = hour.p75 > line ? inARow + 1 : 0;
if (inARow === 3) console.log(`Alert at ${hour.until}: three hours in a row over ${line} ms`);
}Alert at 14:00: three hours in a row over 900 ms. The hours to 12:00, 13:00 and 14:00 were all over the line. Waiting one hour more sends the alert one hour later: that's the price of a longer wait.
Another night hour had three phone visits. What does the first line print?
// The 3 phone visits between 05:00 and 06:00 on Monday, in ms
const lcps = [320, 348, 952];
const line = 900;
const p75 = [...lcps].sort((a, b) => a - b)[Math.ceil(lcps.length * 0.75) - 1];
console.log(`p75 of ${lcps.length} visits: ${p75} ms`);
console.log(p75 > line ? "Over the line: alert!" : "Under the line");p75 of 3 visits: 952 ms. 3 × 0.75 is 2.25, rounded up to rank 3: the slowest of the three. One slow phone made the hour cross the 900 ms line, so the second line is Over the line: alert!.
After the release, what was the p75 LCP of the visits whose LCP was the pop-up's photo, in ms?
2,072 ms. Those are the new visitors on phones after the release: the ones who saw the pop-up. The visits whose LCP was the tea photo barely changed.
Read every hour's p75, however few visits it had. How many alerts go out before the release?
6. Every one came from a quiet hour with fewer than 8 phone visits, where one or two slow phones decided the p75. Skipping those hours removes them all.
Open Thursday's Mint tea page in Chrome, on a phone-sized screen. Which element is the LCP?
img.offer-photo, the pop-up's photo. On a phone's screen, it's as wide as the screen, so it's bigger than the tea photo. On a wide screen, the pop-up is a small card, and the tea photo stays the LCP.
Prometheus has a second setting next to for. It keeps an alert firing for a while after the condition stops, so it doesn't switch on and off. What is it called?
keep_firing_for. It keeps the alert firing for that long after its condition was last met, so a value that dips under the line for a moment doesn't end the alert and start a new one.
Check your understanding
For each question, ask: what reached the visitors, which number shows it, and how many visits stand behind that number?
Question 1 of 7Meera's pop-up reached visitors on Thursday, and every check on every pull request had passed. Why did no check stop it?
Choose an answer to see the explanation.
Question 2 of 7After the release, a third of phone visits got much slower and the rest didn't change. Which hourly number showed it most clearly?
Choose an answer to see the explanation.
Question 3 of 7What does this code print, the p75 of four visits by nearest rank?
Read the code, then predictconst lcps = [380, 1460, 420, 510]; const sorted = [...lcps].sort((a, b) => a - b); console.log(sorted[Math.ceil(lcps.length * 0.75) - 1]);Choose an answer to see the explanation.
Question 4 of 7Why does Ravi's alert skip hours with fewer than 8 phone visits?
Choose an answer to see the explanation.
Question 5 of 7Ravi's rule waits for two hours in a row over the line. What does that cost?
Choose an answer to see the explanation.
Question 6 of 7Search Console showed Ravi's phone LCP as good all week. What does that say about Thursday's change?
Choose an answer to see the explanation.
Question 7 of 7The hourly robot caught the pop-up at its first check after the release. What could it not tell Ravi?
Choose an answer to see the explanation.
Key takeaways
- A slowdown can reach visitors without a pull request. Meera's tag manager release did, so only checks on the live page saw it.
- Engineers need splits, by device, LCP element and release, with the visits behind each number. Leaders need a weekly trend, not pass or fail.
- Mark every push on the charts, code and tags alike, and send the release and the tags version in every beacon.
- Alert on the p75, skip hours with too few visits, wait for a lasting change, and send one message per problem.
- CrUX covers 28 days, so it shows a change weeks late.
- A synthetic monitor is quick, steady and awake at night. Real-user data says who was hit, and how badly. Use both.
That's Module 39. Lesson 39.1 drew the lines: budgets from real data. Lesson 39.2 checked every pull request against them, with room for the wobble. This lesson watches what really reaches visitors, whatever path it took.
Remember the one-liner.
Watch real visits, mark every release, and alert only on slowdowns that are real, lasting, and measured on enough visits.
Sources
The facts on this page come from these docs and books:
- Monitoring Distributed Systems, Google, Site Reliability Engineering.
- Service Level Objectives, Google, Site Reliability Engineering.
- Alerting on SLOs, Google, The Site Reliability Workbook.
- Getting started with measuring Web Vitals, web.dev.
- Fixing website speed cross-functionally, web.dev.
- Best practices for cookie notices, web.dev.
- Largest Contentful Paint (LCP), web.dev.
- Interaction to Next Paint (INP), web.dev.
- web-vitals 6.2.3: the attribution build, Google Chrome.
- CrUX API, Chrome for Developers.
- CrUX History API, Chrome for Developers.
- Publishing, versions, and approvals, Tag Manager Help.
- Built-in variables for web containers, Tag Manager Help.
- Releases, Sentry Docs.
- Chart annotations, DebugBear docs.
- Deployment testing (Synthetic), SpeedCurve docs.
- Performance budgets and alerts, SpeedCurve docs.
- Alerting rules, Prometheus docs.
- Test scheduling, DebugBear docs.
- Schedule test times, SpeedCurve docs.
- A/B tests interfering with metrics and alerts, SpeedCurve docs.
Coming next: Running a performance audit. You'll start from field data like Ravi's, pick the pages and visitors that matter, and turn what you find into a ranked list of fixes.