Design an A/B testing system for the frontend

Requirements, and what makes experiments hard

An A/B test (also called an experiment) shows two or more versions of something to randomly chosen groups of users and measures which version does better. The current version is the control; each new version is a treatment or variant. An experimentation platform is everything that makes this routine: a place to set up tests, code that puts each user in a group, event logging, a statistics engine, and a results page.

This guide builds on other guides and does not repeat them. The feature-flag-sdk guide covers how a flag is evaluated in the browser, how a user is hashed into a stable bucket, how values reach the first frame without a flash (bootstrapping), and the basics of exposure events. The analytics-sdk guide covers how events get from the browser to the server: batching, sendBeacon, retries. The cookie-consent guide covers how consent is collected and read.

The ab-testing lesson covers the basic idea of a test and a p-value. Here we focus on what goes wrong when a company runs 50 experiments at once and needs every one of them to be right.

That focus matters because experiments fail silently. A broken button shows up in error tracking within minutes. A broken experiment shows up as a confident, wrong decision: the team ships a variant that "won by 3 percent" when the real effect was zero, or kills a good idea because a slow variant lost users before they were counted. Nobody notices, because the dashboard still shows a nice chart.

Questions to ask

Functional requirements

• Design an experiment: hypothesis, variants, a primary metric, guardrail metrics, the unit of randomization, targeting, traffic allocation, a layer, and a planned sample size.
• Assign every visitor to a variant deterministically, the same way on the server, at the edge, and in the browser, and keep that assignment stable when allocation changes or the user logs in.
• Show the variant on the first frame, with no flicker and no layout shift.
• Log an exposure when the user actually sees the variant, once per user per experiment.
• Analyze: join conversions to exposures, check data quality (sample ratio mismatch), and compute effects with confidence intervals.
• Show results that a product manager can read correctly: primary metric, guardrails, segments, and a clear decision.
• QA and debug: force a variant, share preview links, and see every assignment on the page.
• Govern: review before launch, detect collisions, stop and clean up finished experiments.

Non-functional requirements

Out of scope, and why

• Event transport: batching, retries, and offline queues belong to the analytics-sdk guide. We define which events to send and when.
• Flag evaluation internals: rule matching and the SDK's React hooks belong to the feature-flag-sdk guide. We use them.
• The warehouse statistics code: we explain what it computes and how the frontend shows it, not the SQL.
• Multi-armed bandits, which shift traffic toward the winner during the test. They answer a different question (earn the most now) than an A/B test (learn the true effect), and most platforms keep them separate.
• Native mobile apps: the ideas carry over, but release cycles and caching are different.

What makes experiments hard to get right

Showing version B to half the users is one line of code. The hard part is that an experiment is a measurement, and every step between "a product manager has an idea" and "we ship it" can bend the measurement without anyone noticing. Think of it as a chain: the result is only as good as its weakest link.

• Correctness is the product. For most features, a small bug is a small problem. Here a small bug, such as logging exposure 200 ms too early, can flip the decision on a feature worth millions. The platform's job is to make wrong results hard to produce and easy to detect.
• Randomization must survive the real web. Users clear cookies, switch devices, log in halfway through a visit, open links in new tabs, and arrive through cached pages. Each of these can move a person from one group to another or count them twice.
• The test can change what it measures. If the variant is applied by a script that runs after the page paints, the variant also flickers and loads later. The test then measures "new design plus a flicker", not the new design.
• Experiments overlap. With 50 tests live, the same user is in about 8 at once. Two tests on the same button can interact, and nobody sees it unless the platform tracks where each test runs.
• The readers are not statisticians. A product manager looks at the results every morning and wants to stop as soon as it turns green. Doing that turns a 5 percent false positive rate into 20 percent or more. The results page has to protect people from that habit.
• Experiments end. Each finished test leaves dead branches in the code. At 15 launches a week, that is hundreds of stale if statements a year unless cleanup is part of the process.

So the guide follows the chain. First the experiment's design in the platform, then how many experiments share traffic, then assignment, then how the variant is shown, then exposure logging, then the checks and statistics that decide whether the result can be trusted, and finally the results page, QA, and governance that keep it all honest over time.

Designing an experiment: hypothesis, metrics, unit, allocation, and layers

Most broken experiments are broken before the first user arrives. The team picks five metrics and decides later which one counts, randomizes per page view when the change lasts a whole visit, or launches with no idea how long the test must run. The setup screen is where the platform can prevent all of this, so we design it as a guided form, not a blank settings page.

Start with a hypothesis

The form opens with one required field shaped as a sentence: "Because we saw evidence, we believe that change for audience will move metric by at least amount." For example: "Because 38 percent of mobile users abandon on the shipping step, we believe that showing delivery dates on the product page for mobile users will raise checkout completion by at least 1 percent." Writing it down before launch stops the most common trap: running the test first and then searching the metrics for a story.

Primary, secondary, and guardrail metrics

Metrics are picked from a shared metric catalog, not typed as free text. The catalog holds one definition of each metric (which event, which filter, per user or per session), owned by the data team. If two teams each write their own "conversion", their results can't be compared and one of them is usually wrong. The platform also adds company guardrails automatically, such as error rate and page speed, so a team cannot forget them.

The unit of randomization

The unit of randomization is the thing we flip a coin for: a user, a device, a session, or a whole company account. Everything that shares a unit sees the same variant. Pick it badly and the same person sees both versions, which mixes the groups and shrinks every measured effect toward zero.

The hard case is the visitor who starts logged out and logs in at checkout. If the product page test was randomized by device id, it should stay on device id after login; switching to the user id mid-visit can move the person into the other variant on the very page where they convert. The figure shows the rule: an experiment keeps the unit it started with for its whole life, and identity stitching (covered in the assignment chapter) only links ids for analysis.

Allocation and traffic splits

Two numbers decide who is in the test. Traffic allocation is the share of eligible units that enter the experiment at all, such as 20 percent. The split divides those units between variants, such as 50/50.

Equal splits give the most statistical power for a given number of users. A 90/10 split looks safer, but the small arm makes the test need about 2.8 times as many users for the same precision. Use a small allocation for safety instead, and keep the split equal.

{
  "key": "pdp-delivery-dates",
  "hypothesis": "Showing delivery dates on mobile PDP raises checkout completion by 1%",
  "unit": "device_id",
  "layer": "pdp-main",
  "iteration": 3,
  "allocation": 0.2,
  "variants": [
    { "key": "control", "weight": 0.5 },
    { "key": "dates", "weight": 0.5 }
  ],
  "targeting": { "device": ["mobile"] },
  "exposeOn": "visible",
  "primaryMetric": "checkout_completion",
  "guardrails": ["revenue_per_visitor", "lcp_p75", "js_error_rate"],
  "plannedSampleSize": 307200,
  "salt": "pdp-delivery-dates:3f9a"
}

Running 50 experiments at once: layers, exclusion, and holdouts

Each experiment hashes the unit id with its own salt, so assignments in different experiments are independent: being in the treatment of one test tells you nothing about your group in another. For most pairs of tests this is exactly what we want. A test on search ranking and a test on the footer can share every user, and each one's random split balances the other's effect out.

It breaks when two tests touch the same thing. Team A tests a green "Buy now" button; team B tests the button text "Add to bag". A user in both treatments sees a combination nobody designed.

Worse, team C tests removing the button from the product page entirely, and team B's users in C's treatment never see B's change, so B's data is quietly diluted. These interactions are rare across a whole site, but they cluster on hot pages like the product page and checkout, which is where the important tests run.

Two hashes: one for the layer, one for the variant

Assignment inside a layer takes two steps. The first hash, salted with the layer's id, places the unit in one of 10,000 layer buckets; each experiment owns a range of them. The second hash, salted with the experiment's own salt, picks the variant.

Using a separate salt keeps the variant independent of the slot. Suppose we took the variant from the user's position inside the range instead: first half control, second half treatment. Widening the range from 1,000 to 2,000 buckets would then move every user in slots 500 to 999 from treatment to control in the middle of the test.

type Range = { experiment: string; from: number; to: number };

function assignInLayer(unitId: string, layer: Layer): Assignment | null {
  const slot = hashToBucket(`${layer.salt}:${unitId}`); // 0..9999
  const range = layer.ranges.find((r) => slot >= r.from && slot < r.to);
  if (!range) return null; // free buckets: in no experiment of this layer

  const exp = layer.experiments[range.experiment];
  if (!matchesTargeting(exp, unitId)) return null;

  const pick = hashToBucket(`${exp.salt}:${unitId}`) / 10000; // 0..1
  let edge = 0;
  for (const v of exp.variants) {
    edge += v.weight;
    if (pick < edge) return { experiment: exp.key, variant: v.key };
  }
  return null;
}

Holdouts: measuring everything we shipped

Each experiment measures one change over two to four weeks. Leadership asks a different question: did the 40 changes we shipped this quarter add up to the 12 percent gain the individual tests claim? Usually they don't, because of novelty effects, overlap between wins, and a few false positives.

A holdout answers it: a small slice of users, often 1 to 5 percent, is kept out of every experiment and every launch from a set of teams for a quarter. Comparing them with everyone else measures the real cumulative effect.

Holdouts have real costs, so the platform treats them as a first-class object with an owner and an end date. Held-out users miss improvements for months. Engineers must keep old code paths alive until the holdout ends. And the holdout must be carved out first, with its own salt, before any layer assigns, so that it is the same users across every layer.

Assignment: salts, sticky buckets, and identity stitching

The feature-flag-sdk guide covers the core: the SDK hashes a salted unit id into one of 10,000 buckets, so the same id always lands in the same bucket on the server, at the edge, and in the browser, with no network call. This chapter is about what experiments add on top: why the salt must change per experiment, what happens when someone edits a running test, and how one person with three ids stays one person.

A salt per experiment, and a new one per restart

Suppose every experiment hashed the bare device id. Device d-81 lands in bucket 3,120 for every test, and with 50/50 splits buckets 0 to 4,999 are always treatment. Half of our users would see every new idea and the other half none.

After a year the treatment group is a different population: used to change, maybe tired of it. That leftover influence is called carryover, and it biases every later test. A unique salt per experiment, such as pdp-delivery-dates:3f9a, reshuffles users for each test.

The same logic applies when a test is restarted after a bug. If the first run crashed the variant for two days, the users who saw the crash are still in the treatment group and still annoyed. Restarting with a new salt reshuffles everyone, so the damage is spread evenly over both groups. The platform does this automatically: "Restart" creates a new iteration with a fresh salt, and analysis only reads data from the current iteration.

Keeping users in place when the allocation changes

Teams like to start at 10 percent and ramp up to 50 percent once nothing breaks. If an experiment's bucket range only grows, ramping is safe: extending it from buckets 0 to 999 to buckets 0 to 4,999 only adds users, and everyone already in stays in their variant, because the variant comes from the second, independent hash. Two edits are not safe: changing the split (50/50 to 80/20) and ramping down. Both move users who were already exposed.

Identity stitching: one person, several ids

A shopper browses on their phone logged out (device d-81), logs in at checkout (user u-42), and the next day buys on a laptop (device d-93). Identity stitching records that these ids belong to one person, so that analysis can follow them. The client's job is small but exact: when login succeeds, send one identify event linking the device id to the user id. The warehouse builds the person graph from those events.

• Device-unit experiments ignore the login. They keep hashing d-81, so the checkout the user is on does not change variant at the moment they log in.
• User-unit experiments start at login. They were not eligible before. If one applies to the current page, its new variant takes effect on the next navigation, not mid-page, so nothing swaps under the user.
• Cross-device mixing is measured, not hidden. With stitching, analysis can see that person u-42 was control on d-81 and treatment on d-93. The results page reports this share (often a few percent) and warns if it grows large.
• Logout keeps the device id. Clearing it on logout would put the same device in new buckets and double-count it.

Where the variant is applied: browser, server, or edge

The current setup is the classic one: a vendor snippet in the page head downloads a list of changes made in a visual editor ("change this heading to X, hide that banner") and applies them to the DOM after the page arrives. It needs no engineers, which is why marketing teams love it. It also has two failure modes that bias results, and the fix for one makes the other worse.

• Flicker. If the snippet loads asynchronously, the browser paints the original page first, then the snippet rewrites it. Users see the control for a moment and then the variant. If the change adds or resizes something above the fold, content jumps, which adds layout shift (CLS) to the variant only.
• Hidden pages. To stop flicker, vendors either load the testing script synchronously in the head, which blocks rendering until it has run (Optimizely Web recommends this), or ship an anti-flicker snippet: a tiny inline script and style that hides the page (opacity: 0 or visibility: hidden) until the testing script has run, or until a timeout fires. Either way the page is blank for that time, for every user in every group, including users in no experiment at all.

The figure uses estimates for a mid-range phone on a 4G connection; real numbers depend on the vendor, the device, and how many changes are configured. A testing script of 40 to 100 KB that must be fetched from a third-party origin, parsed, and run often takes 200 to 600 ms before changes apply.

With anti-flicker hiding, nothing can paint until that script has run, so the reveal time becomes a floor under first contentful paint (FCP) and LCP for every visitor. In the figure, content that would show at 1.6 seconds shows at about 2.0. On a slower phone the same delay can push a page past the 2.5 second "good" LCP threshold.

Without hiding, the cost moves to layout shift. CLS scores each shift as the share of the viewport that moved times the distance it moved, as a share of the viewport's height. An above-the-fold change that pushes most of a 700-pixel-tall viewport down by 80 pixels scores about 0.9 x 0.11, or 0.1, on every variant view: the edge of the "good" range on its own.

Edge-side variants and cache keys

A cached product page has one URL and one cached copy. If the edge simply varied on the Cookie header, every visitor's unique cookie string would create its own cache entry and the hit rate would collapse to near zero. Instead, the edge function computes a short variant key from only the experiments that apply to this path, such as x7.b-x9.a, and uses it as part of the cache key. The cdn-caching lesson covers cache keys in general.

export async function handle(req: Request): Promise<Response> {
  const url = new URL(req.url);
  const known = readCookie(req, "cf_did");
  // No analytics consent: no id, no assignment, everyone gets control.
  const deviceId = hasAnalyticsConsent(req) ? (known ?? crypto.randomUUID()) : null;

  const exps = deviceId ? config.experimentsFor(url.pathname) : []; // config pushed to the edge
  const key = exps
    .map((e) => `${e.id}.${assign(e, deviceId!) ?? "off"}`)
    .sort()
    .join("-");

  const target = new URL(url.pathname + url.search, ORIGIN);
  target.searchParams.set("__v", key || "base"); // part of the cache key
  const cached = await fetch(target, { headers: req.headers });

  const res = new Response(cached.body, cached);
  res.headers.set("Cache-Control", "private, no-store"); // browser copy is per user
  if (deviceId && !known) res.headers.append("Set-Cookie", deviceCookie(deviceId));
  return res;
}

Two limits keep this safe. First, cardinality: three experiments with two variants each on one path means up to 8 cached copies, and five means 32. The platform caps cached-page experiments per path (we use 3), and the layer system helps, since experiments in one layer are exclusive and add copies instead of multiplying them.

Second, the edge only assigns. The page still carries the assignments to the browser SDK so hydration matches, and exposure is still logged by the browser when the variant is seen, as the next chapter explains.

Exposure logging: count people when they see the change

Being assigned to a variant and being exposed to it are different events. Assignment is a calculation: the server or SDK works out which variant this unit would get. Exposure is a fact: the unit was shown the experience that differs between variants. The analysis should include exactly the exposed units, from both groups, counted from the moment of exposure.

Why logging at assignment wastes most of the test

The delivery dates test changes a block that sits below the fold on mobile product pages. The feature-flag-sdk guide explains why counting people who never saw a change dilutes the result; here we size the damage. If we log exposure when the server assigns, every mobile product page view counts. Say 1,000,000 devices are assigned and only 120,000 scroll far enough to see the block.

If the change lifts conversion by 2 percent for people who see it, it lifts it by only about 0.24 percent across all assigned devices, because 88 percent of them had the same experience in both groups. Required sample size grows with one over the effect squared, so the diluted test needs about 70 times as many counted units. Only 12 of every 100 visitors would be counted under visibility logging anyway, so in traffic terms that is about 8 times more: a 2-week test becomes a 4-month test.

export function ExposureBoundary({ experiment, children }: Props) {
  const ref = useRef<HTMLDivElement>(null);
  const { variant, logExposure } = useExperiment(experiment);

  useEffect(() => {
    const el = ref.current;
    if (!el) return;
    let timer = 0;
    const io = new IntersectionObserver(([entry]) => {
      clearTimeout(timer);
      if (entry.isIntersecting) timer = window.setTimeout(() => {
        logExposure(); // deduped inside the client
        io.disconnect();
      }, 500);
    }, { threshold: 0.5 });
    io.observe(el);
    return () => { clearTimeout(timer); io.disconnect(); };
  }, [logExposure]);

  return <div ref={ref} data-experiment={experiment} data-variant={variant}>{children}</div>;
}

One source of truth, deduped twice

Assignment must be computed once per page and shared. The server or edge assigns, writes the assignments into the HTML bootstrap, and the browser's experiment client reads them from there instead of hashing again. Every exposure carries that exact assignment: experiment key, iteration, variant, unit type, unit id, and where it was assigned (server, edge, or client).

The warehouse never recomputes variants from ids. It can, however, recompute them as a nightly check: a mismatch rate above zero means a bug in some SDK.

Deduplication happens in two places. The client keeps a set of keys like pdp-delivery-dates:3:d-81 in memory and in sessionStorage, so re-renders and route changes send nothing new. That drops roughly 200 million raw exposure triggers a day to about 25 million events.

The warehouse then keeps the first exposure per unit, experiment, and iteration, because tabs, devices, and retries can still send duplicates. Events travel through the analytics SDK's queue, so batching, sendBeacon on pagehide, and retries are already handled.

Trusting the data: sample ratio mismatch

Before reading any metric, check the one number we know in advance: how many units each group should have. A 50/50 test should have about equal exposure counts. If it doesn't, something removed or added units in one group in a way that probably relates to the outcome, and every metric is suspect.

This is a sample ratio mismatch, or SRM. It is one of the most common reasons experiments turn out invalid. Microsoft researchers (Fabijan and colleagues, KDD 2019) reported SRM in about 6 percent of Microsoft's A/B tests, and LinkedIn once saw it in about 10 percent of its triggered analyses.

The check: a chi-square test on counts

Random assignment never gives exactly 50/50, so we need to know how far off is too far. A chi-square goodness-of-fit test compares observed counts with expected counts. Take 50,000 exposed units in control and 48,500 in treatment, so the expected count is 49,250 each.

The statistic sums (observed - expected)^2 / expected over both groups: about 11.4 + 11.4 = 22.8. With two groups (one degree of freedom) that gives a p-value near 0.000002. A split that looks harmless, 49.2 percent instead of 50, is almost impossible by chance at this size.

function srmCheck(counts: number[], weights: number[]) {
  const total = counts.reduce((a, b) => a + b, 0);
  const chi2 = counts.reduce((sum, observed, i) => {
    const expected = total * weights[i];
    return sum + (observed - expected) ** 2 / expected;
  }, 0);
  const df = counts.length - 1;
  const p = df === 1 ? erfc(Math.sqrt(chi2 / 2)) : chiSquareSurvival(chi2, df);
  return { chi2, p, mismatch: p < 0.001 };
}

Platforms use a strict threshold, because the check runs on every experiment at every refresh; at 0.05, one healthy experiment in twenty would raise a false alarm. Microsoft's platform uses p below 0.0005, GrowthBook defaults to 0.001, and we use 0.001.

When the check fires, the results page hides the decision and shows a blocking banner. That is a deliberate product choice: an SRM warning in small grey text gets ignored.

What causes SRM in a frontend

Try the first two causes in the lab. Its losses are realistic estimates, not measurements: about 1 percent of users leave before exposure for each extra 100 ms the variant needs, and a redirect loses about 3 percent. The users who leave would have converted at 1 percent, against an average of 4 percent, since impatient visitors buy less. Watch how a small leak becomes a clear SRM as the sample grows, and how logging before the redirect removes that cause.

When SRM fires, the platform helps find where. It reruns the check per segment: browser, device type, country, entry page, and new versus returning. A mismatch that appears only on Safari points to a browser-specific bug; one that appears only on the product page entry points to the redirect. The debug view lists these segment checks sorted by p-value.

Peeking, sample size, and the statistics the platform enforces

The ab-testing lesson explains p-values and significance. This chapter is about the ways real teams misuse them, and how the platform's design, not a training course, prevents it. The statistics run in the warehouse; the frontend's job is to collect the right inputs at setup and show outputs that can't be misread.

Plan the sample size before launch

A test needs enough units to detect the smallest effect worth acting on, the minimum detectable effect (MDE). A handy approximation for 80 percent power at a 5 percent significance level is n per group = 16 x variance / effect^2. Our delivery dates test has a 4 percent checkout rate and wants to detect a 5 percent relative lift, from 4.0 to 4.2 percent.

The variance of a 0 or 1 outcome is 0.04 x 0.96 = 0.0384, and the effect is 0.002. So n is about 16 x 0.0384 / 0.000004 = 153,600 per group, 307,200 in total.

The setup form does this math live as the team types the baseline and MDE, then converts it to days using the eligible traffic: about 29,000 exposed devices a day at 20 percent allocation gives about 11 days, rounded up to 14 so the test covers two full weeks. Weekday and weekend shoppers behave differently; a test that runs Monday to Thursday measures weekday shoppers only.

Peeking: why checking every day breaks a fixed test

The classic p-value assumes we look once, at the planned sample size. If we look every day and stop the first time p drops below 0.05, we get many chances to catch a random high point. The false positive rate climbs from 5 percent to about 22 percent over 14 daily looks, in line with the classic published figure of about 19 percent for 10 looks. Nothing in any single day's chart looks wrong.

Run the lab: it simulates 200 A/A tests, where both groups get the identical page, so every "win" is false. The batch is generated from a fixed seed, so the numbers match the text until you ask for a new batch.

CUPED: same answer, fewer users

Much of the noise in a metric like "orders per user" comes from people being different: some shop weekly, some yearly. CUPED (Controlled-experiment Using Pre-Experiment Data, from Microsoft researchers in 2013) uses each user's value of the same metric from before the test to predict part of that difference and subtract it out. Randomization means the pre-test values are balanced between groups, so the subtraction removes noise without adding bias.

The gain depends on how well the past predicts the present. If the correlation is r, the variance shrinks by a factor of about 1 - r^2: with r = 0.6 the variance drops by 36 percent, so the test needs about 36 percent fewer users for the same precision.

New visitors have no history, so CUPED helps little on metrics dominated by first-time users. The frontend's part is small but required: stable unit ids that let the warehouse find each unit's history, and a note on the results page saying the estimate is variance-reduced.

Novelty and change aversion

A new button gets clicked because it is new; after a week it is just a button. That is a novelty effect. The opposite, change aversion, makes regular users resist a new layout before they get used to it. Both make the first days unrepresentative.

The results page plots the effect by days since each user's first exposure. A line sloping toward zero signals novelty. A line rising from below zero signals change aversion. For big redesigns, also compare new users, who have no habits to break.

The results page: built for decisions, not for p-values

The old tool shows a table of metrics with a p-value and a green or red cell. Product managers read "p = 0.04, green" as "96 percent sure the variant is better", which is not what a p-value means, and the table says nothing about how big the effect is. A tiny, unimportant lift on a huge sample turns green; a large, promising lift on a small sample stays grey and gets thrown away.

The layout, top to bottom

The page reads in the order a decision should be made. First the health strip: SRM status, sample progress against the plan, days run, and data freshness ("updated 14 minutes ago"). If SRM fires, everything below is greyed out behind a banner.

Then the primary metric, large. Then guardrails as a pass or fail list. Then a decision box with the platform's suggestion: ship, don't ship, keep running, or inconclusive. Secondary metrics and segments come last, collapsed.

Segments and the multiple-comparisons trap

Everyone wants to slice: mobile vs desktop, new vs returning, by country. With 30 metrics and 10 segments, that is 300 comparisons, and if nothing really changed, a 5 percent error rate makes about 15 of them look significant by pure chance. Someone will find "treatment wins for returning Safari users in Germany" and ship on it.

The page handles this in three ways. Segment results are labelled exploratory. Significance in the segment table is adjusted for the number of comparisons, for example with the Benjamini-Hochberg method, which controls the expected share of false wins among the wins shown. And a counter above the table says how many comparisons are shown and how many false wins to expect.

The page reads precomputed results from one endpoint and does no statistics in the browser. A full results payload for 30 metrics, 10 segments, and 2 variants is about 600 rows, around 40 KB gzipped, so it renders in one pass without virtualization. The interval bars are SVG with a text equivalent in each row for screen readers, such as "plus 1.8 percent, interval 0.4 to 3.2".

Performance experiments: Core Web Vitals per arm

Some experiments change speed on purpose: a product page with 120 KB less JavaScript, server-rendered reviews instead of client-rendered ones, a different image format. These are the best way to prove that speed earns money, which is the argument the web-performance-overhaul guide needs to make. They also need two changes to the usual setup, because speed affects the page from its first byte.

• Exposure at assignment, on the server or edge. The change is the whole page load, so the moment of exposure is the request itself (exposeOn: "request"). Waiting for a component to render would lose exactly the users the slow arm drives away, which is the SRM from the earlier chapter.
• Web Vitals tagged with the arm. The real-user monitoring (RUM) beacon that already reports LCP, INP, and CLS adds the page's active assignments as a dimension, so the warehouse can compute each metric's 75th percentile per arm. The real-user-monitoring lesson covers collection; the core-web-vitals lesson covers the thresholds.

import { onCLS, onINP, onLCP, type Metric } from "web-vitals";

function report(metric: Metric) {
  rum.queue({
    name: metric.name, // "LCP" | "INP" | "CLS"
    value: metric.value,
    rating: metric.rating,
    page: location.pathname,
    // only experiments exposed on this page, e.g. "pdp-lite.b,search-v3.control"
    experiments: experiments.exposedOnThisPage().join(","),
  });
}

onLCP(report);
onINP(report);
onCLS(report);

Two traps are specific to speed. First, percentiles are not averages: the standard formulas for intervals do not apply to a 75th percentile. So the stats engine uses methods made for quantiles, such as the bootstrap, which resamples the data many times to see how much the percentile moves.

Second, missing samples. LCP is reported when the page is hidden or the user interacts, and users who leave a very slow page early may never send it. If one arm loses more samples, its percentile looks better than reality. The results page therefore shows "page views with an LCP sample" per arm next to the percentile, and treats a gap between arms like an SRM.

Architecture, data flow, and interfaces

The system has two frontends. The product app is the marketplace shoppers use; experiments run inside it. The experiment console is an internal web app where teams set up tests, QA them, and read results. Between them sit a config service, a shared assignment library, the analytics pipeline, and the stats engine.

Each boundary has a reason. The assignment library is one shared package because agreement between server, edge, and browser is the whole point; three implementations in three languages would drift. The config is pushed, not fetched per request, so assignment never waits on the network and keeps working with the last known config if the config service is down.

The statistics live in the warehouse, not the console, because they need billions of rows and must produce one agreed answer that every viewer sees. The console only renders.

The data flow for one shopper follows the chain from the first chapter. The request reaches the edge or app server, which assigns and renders, and the HTML carries the assignments to the browser. The browser logs exposures and conversions through the analytics SDK.

The warehouse joins them every hour, and the console reads the results. Every arrow in the figure below carries one specific thing, and each is a place where a bug can bend the result.

Data model and interfaces

Each piece of data lives in exactly one place that owns it, and everything else holds a copy with a known age. The table lists where each lives on the client side; the code below shows the shapes that cross the wire.

type UnitType = "device_id" | "user_id" | "account_id" | "session_id";

type Assignment = {
  experiment: string;     // "pdp-delivery-dates"
  iteration: number;      // bumps on restart, with a new salt
  variant: string;        // "dates"
  unitType: UnitType;
  source: "server" | "edge" | "client" | "override";
  exposeOn: "request" | "render" | "visible";
};

type ExposureEvent = Assignment & {
  type: "exposure";
  unitId: string;
  ts: number;             // client time; server adds received time
  page: string;
};

type IdentifyEvent = { type: "identify"; deviceId: string; userId: string; ts: number };

Overrides from QA are tagged source: "override" and the warehouse drops them, so testers never pollute results. The results API is read-only and returns precomputed rows; the console never asks for raw events.

GET /api/experiments/pdp-delivery-dates/results?iteration=3&segment=device:mobile

200 OK
Cache-Control: private, max-age=300
{
  "health": { "srm": { "p": 0.41, "ok": true }, "progress": 0.72, "updatedAt": "2026-10-21T09:00:00Z" },
  "method": "sequential",
  "metrics": [
    { "key": "checkout_completion", "role": "primary", "control": 0.0400, "treatment": 0.0407,
      "lift": 0.018, "ci": [0.004, 0.032], "significant": true },
    { "key": "lcp_p75", "role": "guardrail", "control": 2310, "treatment": 2290,
      "lift": -0.009, "ci": [-0.031, 0.013], "significant": false }
  ],
  "recommendation": "keep_running"
}

QA, debugging, and governance

A platform where 30 teams launch 15 tests a week needs two kinds of support. Engineers and designers need to see and force any variant before launch. And the company needs rules so that tests don't collide, risky tests get a second pair of eyes, and finished tests leave the codebase.

Forcing a variant and preview links

For QA, the console generates a preview link per variant, such as /p/123?exp_preview=eyJ.... The token names the experiment and variant and is signed by the server with an expiry of 24 hours, so a link pasted into a public chat can't expose an unreleased feature for long.

When the server sees a valid token it sets the cf_exp_force cookie and renders the forced variant on the first frame. Overrides are tagged source: "override" and excluded from analysis. Staff signed into the console can also force variants with a plain ?exp_force=key:variant parameter, which the server only honors for staff sessions.

The debug panel is a staff-only overlay, opened with a keyboard shortcut, that lists every experiment evaluated on the current page: its layer, assigned variant, where it was assigned, whether exposure was logged, and any override. Each row has buttons to switch variant, which set the override cookie and reload. A toggle outlines every exposure boundary on the page, which makes a missing or asymmetric boundary visible at a glance. The panel's code is loaded with a dynamic import() only for staff, so shoppers download none of it.

The experiment lifecycle

Every experiment moves through explicit states, and the console allows only the transitions in the figure. Launch requires a passing launch checklist: hypothesis written, primary metric and guardrails set, sample size planned, layer chosen, each variant opened through a preview link. Tests that touch checkout or revenue, or start above 50 percent allocation, also need approval from a data scientist. Editing a running test's split or salt is not a transition at all: it creates a new iteration.

Collisions between experiments

Layers prevent clashes only when teams pick the right layer. So each experiment also declares the surfaces it changes: route patterns plus named component slots, such as pdp:buy-box. When a test is submitted, the console checks running tests in other layers for overlapping surfaces and shows a warning with both owners' names. In development builds, the experiment client logs a console warning when two experiments from different layers render boundaries for the same slot on one page, which catches undeclared overlaps.

Cleaning up finished experiments

A decided experiment leaves two if branches in the code. The decision step in the console asks which variant wins and turns the experiment into a release flag serving that variant to everyone, so the change goes live without a deploy.

It then starts a 30-day clock for removing the losing branch. A CI check fails any pull request that adds a reference to a decided experiment, and a weekly report lists stale references per team. Some companies go further and generate the cleanup pull request automatically; Uber's open-source Piranha tool does this for stale flags.

Privacy and consent

In the EU, storing a device id and logging exposure for analysis generally needs the user's consent, and the cookie-consent guide covers how consent is collected and read. Our rule is simple and symmetric: users who have not granted analytics consent get no device id, are served control, and are not in any experiment. No exposures are logged for them.

If they grant consent later, they become eligible from the next page. Exposure events carry random ids, never emails or names. The results page states the population plainly, "users who accepted analytics cookies", since they may differ from everyone else.

Running it in the interview

This question is wide, so the risk is spending 40 minutes on hashing. The interviewer usually wants to see that you know an experiment is a measurement, and that you can name the places where a frontend bends it. Lead with that framing in the first two minutes.

Say the numbers out loud: "a 400 ms anti-flicker delay adds to LCP for everyone", "logging at assignment dilutes this test about eight times", "14 daily looks push false positives from 5 to about 22 percent". Numbers show you have reasoned about the system rather than memorized the words.