Choosing a RUM tool

What a RUM tool does for you

Leaf and Kettle's team runs its own RUM pipeline now. RUM, real-user monitoring, means measuring the visits of real people. When Asha leaves the mint tea page, her visit sends a beacon. The shop's endpoint checks it, storage keeps it, and the dashboard shows the p75. Building a RUM pipeline built every stage.

It works. But the team has to run it. Someone keeps the server up, makes room as the file of events grows, and fixes the dashboard when it breaks. So one morning, someone asks: should we keep running our own, or pay for a RUM tool?

There are many RUM tools. So how do you choose? And once a tool is on your page, how do you know it sends what it says?

Four jobs, three ways to share them

Every RUM tool does the same four jobs as the team's pipeline:

• Measure in the page. Work out LCP, INP and the other metrics, the way the web-vitals library does.
• Send each result out of the page as a report, a small message with a metric in it. It goes to a collector, the server address that receives reports. Sending performance data showed how.
• Store every report, like the pipeline's file of raw events.
• Show the results: percentiles, a dashboard and alerts.

Tools differ in who does which job. A tool's code for the page is its SDK, short for software development kit. Past the SDK, there are three ways to share the work:

• Build your own. A small library like web-vitals measures. Your code sends, and your servers store and show. That's the team's pipeline today.
• Run an open-source tool yourself. The tool's SDK measures and sends. You run its collector, its storage and its dashboards on your own servers. Open source means anyone may read the code, change it and run it.
• Pay for a hosted service. The SDK measures and sends. The company that makes the tool, the vendor, stores and shows the data on its own servers. That's what hosted means.

Here are the three ways side by side:

None of the three is always best. The rest of this lesson shows what each one costs you.

Four kinds of tool

Hosted tools come in three kinds, by what else they do. Open-source tools you run make a fourth. The picture names a few tools of each kind, and the next four sections take one kind each.

The course's test

To see what tools really do, the course put five of them on a copy of the mint tea page, one at a time:

• web-vitals, with a six-line do-it-yourself setup;
• Sentry and Datadog RUM, two observability platforms;
• Grafana Faro and Elastic APM RUM, two open-source tools.

Each had its smallest real setup. A server on the course's computer played every collector, and noted when each report arrived. Then Chrome made 5 visits per tool for each of two endings, 50 visits in all. Each visit went like this:

1. Load the page, and wait one and a half seconds.
2. Click Add to cart. Its click handler keeps the main thread busy for 150 ms. The main thread is the one thread that runs the page's JavaScript and answers clicks. So every visit has a slow click, and an INP to report.
3. Wait one second. Then either follow a link to another page, or switch to another tab and leave this one open.
4. Close the tab 5 s later.

Every tool goes on the page in a line or two. Here is Elastic's whole setup, as the course used it:

import { init } from "@elastic/apm-rum";
init({ serviceName: "shop", serviceVersion: "1.0.0", serverUrl: COLLECTOR + "/elastic" });

• Line 1 loads the init function from Elastic's package, @elastic/apm-rum.
• Line 2 starts the SDK. serviceName names the shop. serverUrl is the collector's address.
• COLLECTOR stands for the course's own server, filled in when the file was built.

Two lines. But what do they add to every page? When do they send? Does every report arrive? The five tools gave very different answers. One of them lost a report every time a visitor followed a link.

Dedicated performance RUM tools

Some tools do one thing, page speed, like a shop that sells only tea.

SpeedCurve is one. Its setup guide has you add a small script, lux.js, to every page. It says to put the script "as early in the page as possible, i.e. at the top of the HEAD element." It says this "ensures that we are able to collect as much data as possible for each page."

The same guide says: "The lux.js script is loaded asynchronously so it won't harm the performance of your site." What does that mean?

It means the script tag has the async attribute. MDN says such a script "will be fetched in parallel to parsing and evaluated as soon as it is available." So the browser keeps reading the HTML while the script downloads. But the script still downloads, and it still runs on the main thread, the same thread that answers clicks. Async lowers a script's cost, but it doesn't remove it.

SpeedCurve also runs synthetic tests: lab tests of your pages, on a schedule. A page label is a name for a kind of page, like "home" or "product". The guide says that matching labels in both "enables correlation between RUM and synthetic data". The lab numbers and the field numbers for one kind of page then sit side by side.

Akamai's mPulse is another speed specialist. It's built on boomerang, an open-source library. boomerang's README says: "boomerang is a JavaScript library that measures the page load time experienced by real users, commonly called RUM (Real User Measurement)."

Akamai's own version, the README says, "contains a few mPulse-specific plugins". A plugin is an add-on that gives a library one more job.

Both tools need an account with the vendor, so the course didn't install them. But you can test them the same way as the other five: weigh what they add to the page, and check that their last report arrives.

Observability platforms

If a speed specialist is a tea shop, an observability platform is a supermarket. Speed is one aisle of many.

Observability means being able to see what your whole system is doing: its errors, its slow server calls, its logs. Sentry and Datadog sell tools for all of that, and RUM is one part. Sentry's docs describe what it gives you as information "about errors and performance".

Datadog's billing docs list what its RUM collects. That's every page a visitor sees, the files each page loads, errors in the page, and long tasks.

What do you get for all that? A thread from the page to the server. Datadog's docs say that with its APM integration, "you can tie any XHR or Fetch request to its corresponding backend trace." APM, application performance monitoring, is its tool for servers. A trace is the record of one request's trip through your servers. So a slow click on the page can lead straight to the slow server call behind it.

What the SDK adds to every page

All that comes at a price, paid by every visitor. Each visitor's browser downloads and runs the SDK before it can measure anything. A big SDK is like a guest who brings three suitcases for one night: you have to make room for all of them.

To compare fairly, the course built each tool's smallest setup the same way. esbuild 0.25.11, a build tool, put each setup and everything it imports into one file. That's called bundling. It also minified the file, taking out spaces and shortening names. Then the course compressed it with brotli, as a server would. Here is Datadog's, next to web-vitals':

1// Each tool's smallest setup, bundled, minified and brotli-compressed, in bytes.2const webVitals = 3141;3const datadog = 73923;4 5console.log(`Datadog RUM adds ${datadog.toLocaleString("en-US")} compressed bytes to the page.`);6console.log(`That's ${(datadog / webVitals).toFixed(1)} times as much as web-vitals.`);

• Lines 2 and 3 are recorded: 3,141 bytes for web-vitals, and 73,923 bytes for Datadog RUM.
• Line 5 prints Datadog RUM adds 73,923 compressed bytes to the page.
• Line 6 divides one by the other. It prints That's 23.5 times as much as web-vitals.

Here are all five tools, measured the same way:

Bytes are half the cost; the code also has to run. So the course put a mark with performance.mark() on each side of the SDK's script tag. The tag was a plain one, without async. So the browser stopped reading the HTML until the script had downloaded and run. The gap between the marks is the whole cost.

Each tool had 10 visits. The median, the middle value of those visits, was 13.1 ms for web-vitals. It was 26.8 ms for Sentry, and 32.7 ms for Datadog. The script came from the same computer. On a real network, the download can take far longer.

Why it matters: a RUM tool slows down the page it measures. Before you pick one, build its smallest setup and weigh it, the way the course did.

Hosting-provider analytics

Your site lives on someone's servers: your hosting provider. Some hosts can measure your visitors too, in the same dashboard where you put your site live.

Vercel is one. Its docs say: "Vercel Speed Insights provides you with a detailed view of your website's performance metrics, based on Core Web Vitals". To start, you add one package to your site, @vercel/speed-insights. The numbers then appear in your Vercel project's dashboard.

Speed Insights comes in two tiers. Vercel's pricing docs list what each one shows:

• The free tier shows the Real Experience Score, or RES, only. Vercel says RES "uses real data points collected from your users' devices."
• Speed Insights Plus "adds all Core Web Vitals, detailed breakdowns, and Drains." Drains, Vercel says, "let you forward observability data from your applications to external services".

Cloudflare has one too. Its docs say: "Cloudflare Web Analytics provides free, privacy-first analytics for your website without changing your DNS or using Cloudflare's proxy." So your site can stay where it is. You need a Cloudflare account and a small script on your page. That script, the docs say, "collects metrics using the Performance API".

These are the quickest tools to start with. They're also the ones where the most is decided for you.

So before you rely on your host's analytics, look around the flat:

• What can I split by? On Vercel's free tier, the breakdowns are by path and route, with only the Great and Needs Improvement entries. Plus adds the Poor ones, countries, and element selectors.
• Can I get the raw data out? On Vercel, that's Drains, which come with Plus.
• What if I change host? Speed Insights lives in a Vercel project. Cloudflare's works without moving your site.

Why it matters: the quickest tool to turn on can be the hardest to leave. Check the walls before you move in.

Open-source and self-hosted options

Back to the meal kit. An open-source tool hands you the recipe and the ingredients: an SDK for the page, and a collector. You do the cooking: you run them on your own servers.

Grafana Faro is one. Grafana calls its SDK "a highly configurable open source JavaScript agent". It collects "performance metrics, logs, exceptions, events, and traces." And it sends them to "a collector that you can run yourself or that runs in Grafana Cloud."

Faro's README names the collector you'd run, Grafana Alloy. Grafana's page says the collector saves the data into Loki and Tempo, two of Grafana's storage tools, and you look at it in Grafana.

Elastic APM RUM is another. Elastic's docs say its agents "work in conjunction with the APM Server, Elasticsearch, and Kibana." Those are three more pieces for you to run. Elastic's page on the APM Server says it "receives performance data from APM agents" and turns it into Elasticsearch documents. Elasticsearch stores them, and you look at them in Kibana.

Every tool's code comes with a licence: the legal note that says what you may do with it. Faro's is Apache-2.0 and Elastic's agent's is MIT. Both are open-source licences, so you may use the code, change it and run it yourself.

Here's a surprise: every SDK the course used has an open-source licence. Sentry's is MIT, and Datadog's is Apache-2.0. With those two, what you pay for is the service behind the SDK: the collector, the storage and the dashboards.

Or you can cook at home: web-vitals, whose licence is Apache-2.0, and your own pipeline. This is the whole do-it-yourself setup the course bundled:

1import { onCLS, onFCP, onINP, onLCP, onTTFB } from "web-vitals";2const queue = [];3for (const on of [onCLS, onFCP, onINP, onLCP, onTTFB]) on((metric) => queue.push({ name: metric.name, value: metric.value, id: metric.id }));4addEventListener("visibilitychange", () => {5  if (document.visibilityState === "hidden" && queue.length) navigator.sendBeacon(COLLECTOR + "/diy", JSON.stringify(queue.splice(0)));6});

• Line 1 loads five functions from web-vitals, one per metric.
• Line 2 makes an empty queue: a list of reports waiting to be sent.
• Line 3 gives each function the same small callback. When a metric is ready, the callback adds its name, value and id to the queue.
• Lines 4 to 6 send the whole queue in one beacon, the moment the page becomes hidden. Line 5 checks that the queue isn't empty first.
• queue.splice(0) takes every report out of the queue, so none is sent twice.

It came to 9,183 bytes, and 3,141 with brotli. But free to use isn't free to run. A meal kit still needs your kitchen and your evening: you run the servers, keep them going, and pay for them. In return, the data never leaves your own servers.

What reached the collector

A tool is only as good as the reports that reach its collector. And the hardest report to get out is the last one.

Remember why. INP and CLS can change until the visitor leaves, so the web-vitals library hands over their final values as the page is hidden. That's the very moment the page may be about to disappear.

In the course's test, every visit clicked Add to cart. Each visit's own Event Timing entries showed that click taking at least 152 ms. So every visit had an INP to send. Did it arrive?

Try it first. Pick a tool, and how the visit ended. Each row is one visit, and each box is one report. A minus sign means the report arrived before the visit ended:

Here are the visits that followed a link, counted:

1// In how many of 5 visits that followed a link did each tool's INP arrive?2const inpArrived = {3  "web-vitals": 5,4  Sentry: 5,5  "Datadog RUM": 5,6  "Grafana Faro": 5,7  "Elastic APM RUM": 0,8};9 10for (const [tool, count] of Object.entries(inpArrived)) console.log(`${tool}: INP arrived in ${count} of 5 visits.`);

• Line 1 asks the question.
• Lines 3 to 7 are recorded: for each tool, in how many of its 5 visits an INP report reached the collector.
• Line 10 prints one sentence per tool. The first is web-vitals: INP arrived in 5 of 5 visits. Sentry, Datadog RUM and Grafana Faro print 5 of 5 too.
• The last is Elastic APM RUM: INP arrived in 0 of 5 visits.

Four tools got every INP out. Elastic got none. Its one report, with LCP, CLS, FCP and TTFB, had arrived about a second before the visit ended.

Switching tabs told a different story. Here are both endings, for every tool:

• Sentry's INP arrived in 3 of 5 visits.
• Datadog sent one report straight away, with TTFB and CLS. LCP and INP came only as the tab closed, 5 s later. A real visitor could leave a tab open far longer than that.
• Elastic's INP did arrive this time, in every visit. But it was late.

Why Elastic's INP was late, or lost

Elastic's source code shows what happens. When the page is hidden, Elastic builds its INP report and adds it to a queue. The queue doesn't always go out at once. Elastic's docs describe a timer:

"The agent maintains a single queue to record transaction and error events when they are added. This option sets the flush interval in milliseconds for the queue."

The option is flushInterval, and its default is 500. To flush a queue is to send everything in it. So a report can wait half a second before it leaves.

The recordings fit that timer:

• Following a link: the old page was gone before the timer could fire. So the report never left.
• Switching tabs: a hidden tab stays alive, so the timer does fire, but late. Chrome slows down timers in hidden pages. For a page hidden for less than 5 minutes, Chrome's blog says "The browser will check timers in this group once per second".

1// When Elastic's INP report arrived, in ms after the visitor switched tabs, in 5 visits.2const arrived = [1172, 1509, 836, 1131, 1479];3const flushInterval = 500; // Elastic's default wait, in ms.4 5const sorted = [...arrived].sort((a, b) => a - b);6console.log(`from ${sorted[0]} to ${sorted.at(-1)} ms, median ${sorted[2]} ms`);7console.log(`the shortest was ${sorted[0] - flushInterval} ms longer than the wait`);

• Line 2 is recorded: when each INP report arrived, in ms after the switch. Line 3 is Elastic's default wait.
• Line 5 sorts a copy of the times. Line 6 prints from 836 to 1509 ms, median 1172 ms.
• Line 7 prints the shortest was 336 ms longer than the wait.

No report beat its 500 ms timer. That fits a timer that's checked only once per second.

Elastic's source also compresses each report with CompressionStream, a browser API that gzips data. Every Elastic report the course caught was gzip-compressed. Only then does it send the report, with fetch() and keepalive. keepalive lets a request outlive its page, but only once the request has started.

Compare web-vitals' six lines. They send in the same moment the page is hidden, with sendBeacon(), so there's no timer to miss. All five metrics arrived from every visit, both ways.

Why it matters: a lost report doesn't look like an error. The dashboard just has fewer INPs, and the ones it has come from the visitors who didn't follow a link. So test any tool the way the course did: follow a link, switch tabs, and count what arrived.

Two tools, two answers

Suppose the team ran Elastic and web-vitals side by side. Would their INP agree? Not here. web-vitals heard every visit's INP, and Elastic heard only the visits that switched tabs. Each tool's p75 comes from the visits it heard about.

Tools even give the same metric different names. The course searched every report for each metric's name. For LCP, the search matched largest_contentful in Datadog's reports, lcp in Faro's, and largestContentful in Elastic's.

Field data from every browser found the same with browser engines: each one is its own thermometer. So ask each vendor how it counts, and compare a tool with its own earlier numbers.

Cost, sampling, and data ownership

A hosted tool is a restaurant: you pay for every plate. So the first question is what counts as a plate. Look for the unit on the pricing page before you look at the price:

• Datadog's billing docs say "Datadog charges per one thousand (1,000) sessions ingested" into its RUM. A session is one visitor's trip through your site. Datadog says it "usually includes multiple page views".
• Vercel counts events for Speed Insights, and prices Plus by the event.
• SpeedCurve sizes its RUM plans in page views. Its guide talks of "a RUM plan that covers a maximum of 10 million page views per month".

Prices change often, so read today's numbers on each vendor's own pricing page.

What a free tier leaves out

Vercel's free tier works like that tray. Its docs say:

• The free tier shows the Real Experience Score only. Each Core Web Vital on its own needs Plus.
• It includes a fixed number of events over the last 30 days, shared across the team.
• "When your team reaches the free tier limit, Vercel pauses collection for those projects for at least 14 days."
• "Your sites keep running, and you do not incur event overage charges for the free tier."

So taste a tool on its free tier, but check what it leaves out. A busy month can use up the free events, and then collection pauses.

Sampling: fewer visits, the same mix

To pay for fewer plates, send fewer visits. That's sampling: keep a random share of visits, and drop the rest. The share you keep is the sample rate. SpeedCurve's guide works out a rate for a plan:

1// SpeedCurve's example: 900 million page views a month, and a plan for 10 million.2const pageViews = 900_000_000;3const plan = 10_000_000;4 5const rate = Math.floor((plan / pageViews) * 100);6console.log(`sample rate: ${rate}%`);7console.log(`sampled page views: ${((pageViews * rate) / 100).toLocaleString("en-US")} a month`);

• Lines 2 and 3 are SpeedCurve's example: 900 million page views a month, and a plan for 10 million.
• Line 5 works out the rate as a percentage. Math.floor() rounds down, so the sample stays inside the plan.
• Line 6 prints sample rate: 1%. Line 7 prints sampled page views: 9,000,000 a month.

SpeedCurve's guide gives the same answer: "set the sample rate to 1%." And 9 million fits inside the plan.

Each tool also picks a sample rate for you, until you change it. Like the landlord's furniture, it's worth checking:

• SpeedCurve's guide: "The default sample rate is 100%."
• Datadog's sessionSampleRate "controls the percentage of overall sessions being tracked. It defaults to 100%, so every session is tracked by default."
• Sentry is the other way round. Sentry calls the timed work it records, like a page load, a transaction. Its docs say that until you set a rate, "no transactions will be sent to Sentry." The course set tracesSampleRate to 1, to send them all.

Does sampling bend the numbers? Not if it's random. Sentry says a uniform rate "Provides an even cross-section of transactions, no matter where in your app or under what circumstances they occur." The mix stays the same. There are just fewer visits in each group.

Who keeps the data

Last: where do the reports end up, and who can read them? With your own pipeline, or an open-source tool you run, they stay on your servers. With a hosted tool, the vendor keeps them.

Some tools let you send reports through your own server first. Here is the Datadog setup the course recorded:

1import { datadogRum } from "@datadog/browser-rum";2datadogRum.init({3  applicationId: "00000000-0000-0000-0000-000000000000", clientToken: "pub00000000000000000000000000000000", site: "datadoghq.com",4  service: "shop", proxy: COLLECTOR + "/datadog", sessionSampleRate: 100, sessionReplaySampleRate: 0, trackUserInteractions: true,5});

• Line 1 loads Datadog's SDK. Lines 2 to 5 start it.
• Line 3 holds the account details. The course used made-up ones, because its own server played the collector.
• Line 4 sets proxy to an address of your own, so every report goes there first. sessionSampleRate: 100 keeps every session, which is the default anyway.

A proxy is a server that passes requests on. Datadog's guide says where they go next: "The RUM data still needs to be forwarded to Datadog from the proxy." It even asks the proxy to add each visitor's IP address, in an X-Forwarded-For header, "for accurate geoIP": working out roughly where the visitor is. So a proxy changes the route, not the owner, like handing your letter to a neighbour to post.

Then there's what a tool learns about your visitors. Datadog's docs say it "uses cookies to stitch together the various steps of your users into a session." Cloudflare's say "Cloudflare Web Analytics does not collect or use your visitors' personal data." Where consent is needed, Sending performance data showed how to wait for it.

Here is everything to ask, in one place. Read each row across: the question, then how to answer it for any tool.

Which kind of tool is it?

Ask what each tool is mainly for. Is it built only for speed, part of a bigger platform, your host's own, or code you run yourself?

Common mix-ups

• "A RUM SDK is tiny." No. Datadog's smallest setup was 73,923 bytes with brotli, 23.5 times web-vitals.
• "An async script costs nothing." No. It doesn't stop the HTML while it downloads. But it still downloads, and it still runs on the main thread.
• "Every tool gets INP out when the visitor leaves." No. Elastic's INP never arrived from the visits that followed a link.
• "Two tools' numbers should match." No. They can count different visits at different moments, and even name metrics differently. Compare a tool with itself.
• "Sampling makes the numbers wrong." No. A random sample keeps the same mix of visits, just fewer of them.
• "A proxy means I own the data." No. Datadog's proxy still forwards it to Datadog.
• "Open source means free." No. The code costs nothing to use, but you run the servers, and you pay for them.

Practice exercises

Answer first, then use the hints.

// SpeedCurve's example: 900 million page views a month, and a plan for 50 million.
const pageViews = 900_000_000;
const plan = 50_000_000;

const rate = Math.floor((plan / pageViews) * 100);
console.log(`sample rate: ${rate}%`);
console.log(`sampled page views: ${((pageViews * rate) / 100).toLocaleString("en-US")} a month`);

// In how many of 5 visits that switched tabs did each tool's INP arrive before the tab closed?
const inpArrived = {
  "web-vitals": 5,
  Sentry: 3,
  "Datadog RUM": 0,
  "Grafana Faro": 5,
  "Elastic APM RUM": 5,
};

for (const [tool, count] of Object.entries(inpArrived)) console.log(`${tool}: INP arrived in ${count} of 5 visits.`);

Check your understanding

For each question, ask what the tool sends and when, what you pay for, and where the data ends up.

Key takeaways

• Every RUM tool measures, sends, stores and shows. Tools differ in who runs each job.
• Four kinds: speed specialists, observability platforms, hosting-provider analytics, and open-source tools you run.
• Weigh the SDK and time its script. Here, the smallest setups ran from 3,141 to 73,923 bytes with brotli.
• Follow a link and switch tabs, then count what arrived. A report sent on a timer after the page hides can be late, or lost.
• Two tools can disagree on the same visits. Compare a tool with its own earlier numbers.
• Read the unit before the price, check the defaults, and see what a free tier leaves out. A random sample keeps the same mix.
• Know where the data lives. A proxy changes the route, not the owner.

Sources

The facts on this page come from these docs and source files:

• Setup guide, SpeedCurve.
• <script>: The Script element, MDN.
• boomerang README, Akamai.
• Sampling for Browser JavaScript, Sentry.
• RUM & Session Replay billing, Datadog.
• Speed Insights overview, Vercel.
• Speed Insights limits and pricing, Vercel.
• Speed Insights metrics, Vercel.
• Working with Drains, Vercel.
• Cloudflare Web Analytics, Cloudflare.
• Grafana Faro, Grafana Labs.
• Faro Web SDK README, Grafana Labs.
• APM RUM JavaScript agent, Elastic.
• APM Server, Elastic.
• rum-core: page-visibility.js at 5.17.5, Elastic.
• rum-core: inp/report.js at 5.17.5, Elastic.
• APM RUM agent configuration, Elastic.
• Heavy throttling of chained JS timers beginning in Chrome 88, Chrome for Developers.
• rum-core: apm-server.js at 5.17.5, Elastic.
• Configure your setup for Browser RUM sampling, Datadog.
• Proxy your browser RUM data, Datadog.

Coming next: From field data to a fix. Your tool shows a slow group of visitors. Next, you'll use attribution to find the cause, reproduce it in the lab, fix it, and confirm the fix in the field.