Benchmarking without fooling yourself

What a benchmark can and cannot say

A benchmark is a program that times some work. A microbenchmark is a very small benchmark. It times one small piece of work, such as one function.

A microbenchmark can answer one small, clear question. On its own, it cannot tell you whether a whole page feels fast.

1const start = performance.now();2let sum = 0;3for (let i = 0; i < 100000; i++) sum += i;4console.log(sum);

Line 1 reads a clock with performance.now(). It gives the time in milliseconds (thousandths of a second). Line 2 creates sum. Line 3 adds every number from 0 to 99,999. Line 4 prints the sum, 4999950000.

The lesson does not print how long the loop took. That time is different on every run, on every device, and in every browser.

This difference matters. The sum is always the same, so a test can check it. A time is different. It tells you about one computer at one moment. Treat a time as a clue to look into, not as a fact to remember.

A good benchmark starts with a clear question. Here is a good one: "After this change, is formatting the cart total slower, once the cart has loaded?" Here is a poor one: "Which JavaScript trick is fastest?" It is too broad. It does not say:

• how much data the code works on,
• which browser runs it,
• which phase you time (phases are explained next),
• which task the user is doing.

Small code can matter, but only in the right place. Picture a function that formats the cart total. If it runs once after the cart loads, a tiny speed difference hardly matters. If it runs on every key press while the user types, the same difference matters much more. So ask how often the code runs, what data it gets, and whether the user is waiting for it.

Start with a performance measurement question from a real page. Use a microbenchmark only when you can test one cause on its own. Other work must stay out of it: layout (the browser placing elements on the page), the network, user input, and framework code.

Warm-up and faster tiers

A JavaScript engine is the part of the browser that runs your code. It often starts a function in an interpreter. An interpreter runs code one step at a time. It can start at once, but it runs slowly.

The engine may see the same function run many times in the same way. Then it may compile the function: turn it into faster machine code. This happens while the program runs, so it is called JIT compilation (JIT means "just in time").

Each speed level is called a tier. Moving code from tier to tier is called tiered compilation. The code still gives the same results. Only its speed changes.

1function addCartPrices() {2  let total = 0;3  for (let i = 0; i < 3; i += 1) total += i;4  return total;5}6 7for (let run = 0; run < 2; run += 1) addCartPrices();8console.log(addCartPrices());

Lines 1 to 5 define addCartPrices. It always returns 3 (0 + 1 + 2). Line 7 calls it twice. These two calls are the warm-up: runs of the code before you start timing. Line 8 calls it once more and prints 3.

The output proves what the function returns. It says nothing about speed.

Do not call the later times "the speed of JavaScript". They are times after a warm-up that you chose. The tiered compilation lesson explains why engines move code to faster tiers, and why they sometimes move it back.

There are two phases you can time:

• Cold start: the first runs, before any warm-up. Time this when users run the code once, such as when a page loads.
• Steady state: the runs after warm-up, once the engine has settled. Time this for code that runs again and again, such as in a long-running editor or an animation.

Both can be the right choice. The mistake is to hide which one you timed.

JavaScript makes no promise about tiers. The engine can take a different path when any of these change:

• the kind of values the function gets,
• the properties the objects have,
• the browser version,
• the engine itself.

So a benchmark report writes down the browser and its version. One result on your computer is not a rule of the language.

A small, honest harness

A benchmark harness is a small program that runs the work, collects the times, and reports them. Each collected time is called a sample.

The harness itself can change the result by mistake. So keep it simple, and keep its phases easy to see.

1function measure(work, clock, warmupRuns, measuredRuns) {2  for (let run = 0; run < warmupRuns; run += 1) work();3  const samples = [];4  for (let run = 0; run < measuredRuns; run += 1) {5    const start = clock();6    work();7    samples.push(clock() - start);8  }9  return samples;10}11 12const clock = makeFakeClock([10, 13, 20, 24]);13console.log(measure(() => 6, clock, 1, 2).join(","));

The measure function does not read the real clock itself. It gets a clock as an argument, clock. So the lesson can hand it a fake clock: a function that returns numbers you choose, in order, instead of the real time.

• Line 2 runs the warm-up calls. It stores no samples.
• Line 3 creates the empty list samples.
• Lines 4 to 8 are the measured runs. Line 5 reads the clock. Line 6 runs the work. Line 7 reads the clock again and stores the difference.
• The fake clock on line 12 returns 10, 13, 20, and 24, in that order. So line 13 prints 3,4.

The fake clock is not a shortcut for a real benchmark. It lets this lesson test the harness exactly. A real harness is different in three ways:

• It reads a real clock that can measure tiny times, such as performance.now().
• It collects many more samples.
• It writes down the browser and the computer it ran on.

Setup is the work you do to get ready, before the work you want to time. Keep setup outside the timed part, unless the setup is part of your question. Here are some kinds of setup:

• reading test data,
• building a large array,
• opening a database,
• drawing a button.

Each of these can be important work. Time them only when the user's task includes them. Then say so in the name of the benchmark.

1function measure(work, clock, warmupRuns, measuredRuns) {Run2  for (let run = 0; run < warmupRuns; run += 1) work();3  const samples = [];4  for (let run = 0; run < measuredRuns; run += 1) {5    const start = clock();6    work();7    samples.push(clock() - start);8  }9  return samples;10}11 12const clock = makeFakeClock([10, 13, 20, 24]);13console.log(measure(() => 6, clock, 1, 2).join(","));

The replay never says that your browser took 3 ms or 4 ms. It only shows which runs are warm-up and which runs are measured. We can prove that part without timing anything.

When the engine skips unused work

An optimizer is the part of the engine that makes code faster. Dead code is code whose result nobody ever uses. Dead-code elimination means the optimizer removes that code. If removing it changes nothing you can see, the optimizer may skip it.

Usually this helps. In a benchmark it is a trap: you may time a loop that no longer does the work you care about.

1let total = 0;2for (let i = 0; i < 4; i += 1) {3  Math.sqrt(i); // answer thrown away4}5console.log(total);

Line 1 creates total with the value 0. Line 3 works out a square root with Math.sqrt(i), then throws the answer away. Line 5 prints 0, because no line changes total.

An optimizer may treat work whose answer is thrown away very differently from work whose answer is used. It may even skip it.

1let total = 0;2for (let i = 0; i < 4; i += 1) {3  total += Math.sqrt(i);4}5console.log(total.toFixed(3));

Line 3 now adds each square root to total. Line 5 prints 4.146. Now the result is something you can see. Removing the work would change what the program prints, so the engine must keep it. This proves the work is used. It still says nothing about time.

Sometimes a benchmark does not print its result. It stores it in a sink instead. A sink is a place that receives the result so the program really uses it, such as a variable that is read at the end.

The rule stays the same: use the result in a way the engine must keep. Do not trust a loop just because it looks like hard work in the source code.

Do not go too far the other way, either. Do not print inside every timed loop. Printing to the console is work too, and it gets timed. A good sink is quiet during the loop and is used after it. Here are some good sinks:

• one final total that you print,
• a checksum that the benchmark returns (one number worked out from all the results),
• a value that the calling code uses.

Many samples, not one lap

One time on its own can be luck. Many things can slow down one run:

• another program on the computer may be busy,
• the processor may speed up or slow down,
• the engine may do its own work, such as cleaning up memory that is no longer used.

So a benchmark collects many samples and shows all of them. It does not pick the one number that looks best.

1const samples = [10, 11, 10, 12, 50];2console.log(mean(samples));3console.log(median(samples));4console.log(spread(samples).toFixed(2));

• Line 1 stores five fixed lap times.
• Line 2 prints the mean, 18.6. The mean is the average: add up all the samples, then divide by how many there are.
• Line 3 prints the median, 11. The median is the middle value after you sort the samples.
• Line 4 prints the spread, 15.72. The spread says how far the samples sit from the mean, on average. (Its full name is the standard deviation.)

The one slow lap, 50, moves the mean much more than the median. A sample that sits far away from the others is called an outlier.

A report should not pick one of these numbers as the only answer. Each one answers its own question:

• The mean answers "What was the average of these samples?"
• The median answers "What was the middle sample, after sorting?"

A reader needs both answers, plus the slowest sample. Then they can see whether one long pause changed the result.

Each row of the next table names one number or phase. Read across to see what it shows and what it can hide.

1const withoutOutlier = [10, 11, 10, 12];Run2const withOutlier = [10, 11, 10, 12, 50];3const before = summarizeSamples(withoutOutlier);4const after = summarizeSamples(withOutlier);5console.log(before.mean, before.median);6console.log(after.mean, after.median);

The median is not magic. If many laps are slow, the median moves too. So show all of these:

• the count (how many samples),
• the minimum (the fastest sample),
• the maximum (the slowest sample),
• the median,
• the mean,
• the spread.

Then another developer can see both the normal case and the unusual ones.

Do not delete samples you do not like, unless you wrote down the rule for it before you started. A long pause might be noise: random slowness that has nothing to do with your code. Or it might be the exact problem your users feel.

Keep the full list when it is small. If you remove samples, say which ones and why. Look into surprising values instead of hiding them.

Read a fixed sample report

This playground changes one thing: whether the fixed 50 sample is in the list. It does not read your clock. So you can see exactly what one outlier does, and then reset it.

1const samples = [10, 11, 10, 12, 50];2console.log(mean(samples));3console.log(median(samples));4console.log(spread(samples).toFixed(2));

The playground compares your list with a second fixed list: 13, 14, 13, 15. It looks at the range of each list: everything from the smallest sample to the largest. When the two ranges overlap (share some values), this lesson's model says not sure.

The picture uses two short ranges from the quiz: 10 to 12 and 11 to 13. They share 11 to 12, so the answer is not sure.

not sure is not a failure. It is a useful answer. It tells you what to do next:

• collect more samples that match real use,
• reduce the noise, or
• use a profiler on the real page. A profiler is a tool that records where the time goes while the page runs.

The overlap rule is a simple rule for this lesson. It is not a proof from statistics. Ranges that do not overlap are not a proof either. So the model only says that one side was lower in this sample.

Real performance work may use stronger tools:

• confidence intervals (a range where the true value probably sits),
• many separate runs of the whole benchmark,
• computers set up carefully so that little else is running.

The habit to keep is simple. Describe the evidence you collected. Do not turn it into a winner for every case.

Speedometer and JetStream

Microbenchmarks are small on purpose. A benchmark suite is a collection of bigger tests. Each test in it is a workload: a realistic task, such as using a small to-do app.

Suites answer bigger questions than one microbenchmark. Their answers still have limits. They are useful when you know what each suite contains.

1const suite = {2  speedometer: "web app responsiveness",3  jetStream: "JavaScript and WebAssembly",4};5console.log(suite.speedometer);6console.log(suite.jetStream);

Lines 1 to 4 create an object with two labels. Line 5 prints web app responsiveness. Line 6 prints JavaScript and WebAssembly. These are just labels. They are not benchmark scores.

Speedometer 3 measures how quickly web apps respond to the user. It times pretend user actions, such as adding items to a to-do list, across many workloads. The teams behind the three major browser engines build it together. Its own guide says not to use it to rank frameworks.

JetStream tests JavaScript and WebAssembly. WebAssembly is a fast, low-level code format that browsers can run next to JavaScript. JetStream focuses on advanced web apps. It rewards code that starts quickly, runs quickly, and runs smoothly.

Both suites tell you about their own workloads, their own versions, and the computers they ran on. They are not laws that hold everywhere.

Pick the tool that fits your question:

• Use a suite for a broad signal about a browser. For example, check that a new browser version did not get slower.
• Use a profiler to find out why one of your own user actions is slow.
• Use a tiny benchmark only after a profiler has narrowed the question down to one small choice.

Benchmark a real web change

Say a checkout page feels slow after the user changes the number of items. First, record that real user action with the CPU profiler. It is a profiler that shows which functions used the processor's time. Find out where the time goes:

• your own JavaScript,
• rendering (the browser drawing the page),
• working on data,
• or waiting for the network.

Do this before you write any benchmark.

Then ask one small question. For example: after warm-up, do two ways of formatting a cart total that is already loaded take different times? To answer it honestly:

• keep the input data fixed,
• use the value the function returns,
• collect enough samples,
• keep the profiler result next to the benchmark.

Write down these details:

• the browser version,
• the operating system,
• the kind of device,
• the size of the input,
• the number of warm-up runs and measured runs,
• whether other tabs and programs were quiet.

These details do not make the result true everywhere. They let a teammate understand what your result describes.

After you change the code, try the real user action again. A tiny benchmark can get faster while the page gets slower. The new code might use more memory, cause more drawing work, or take a different path through the code. The real user action is the final check.

Keep the benchmark code in the code review, next to the change it led to. Future readers need the question, the inputs, and the limits as much as they need the result.

• Write down the question before you write the loop.
• Warm up, and say whether you time cold start or steady state.
• Make sure the program uses the result of the work.
• Report all the samples and the details of the computer, not one winning number.
• Check the real user action again before you merge a speed-up.

Common benchmark traps

• "The fastest single run wins." One run is one sample. It is not an answer.
• "Warm-up is cheating." You need it when your question is about steady state. Just say that you did it.
• "The loop clearly does work." If the answer is thrown away, the optimizer may skip the work.
• "The mean settles every argument." Outliers and overlapping ranges need more than one number.
• "A suite score is my page's score." A suite times its own workloads. Profile the page your users actually use.

Each row of the next table is a wrong belief. Read across to see why it is wrong and what to do instead.

Practice exercises

let total = 0;
for (let i = 0; i < 4; i += 1) total += i;
console.log(total);

const samples = [10, 11, 10, 12, 50];
console.log([...samples].sort((a, b) => a - b)[2]);

console.log(rangesOverlap([10, 12], [11, 13]) ? "not sure" : "faster");

Check your understanding

Pick answers that keep two things apart: output that is always the same, and times that need context.

Key takeaways

• Time one clearly named question, not "the speed of JavaScript".
• Warm up when your question is about steady state, and say that you did.
• Make the program use the result, so the engine cannot skip the work.
• Show many samples: the median, the spread, and how sure you are.
• Treat Speedometer and JetStream as facts about their own workloads, not as rankings for everything.
• Profile the real user action before and after you speed it up.

Coming next: The HTML event loop processing model. It shows how the browser decides which piece of work runs next, and when it draws the page.