On-stack replacement

Switch while a call is already running

The engine is the program that runs your JavaScript. Chrome and Node use an engine called V8. V8 can run the same function in two ways, called tiers:

• The interpreter runs your code right away. It starts fast, but each step is slow.
• Optimized code is fast code that the computer's processor runs directly. The engine takes time to build it, so it only builds it for code that runs a lot.

Code that runs a lot is called hot. Code that has barely run is called cold. Usually, the engine switches a function to optimized code the next time the function is called.

But some functions are called only once, and then spend a long time in one loop. A data import, an image filter, or a report builder can work like this. If the engine waited for the next call, the work might finish before the fast code ever ran.

The name comes from two words:

• The stack is the engine's list of function calls that are running right now.
• A frame is one call's entry in that list. It holds the call's variables and where to go back to when the call ends.

OSR replaces the code that a frame is running, while the frame stays on the stack. The function does not start again.

1function sumOnce(limit) {2  let sum = 0;3  for (let i = 1; i <= limit; i += 1) {4    sum += i;5  }6  return "sum=" + sum;7}8 9console.log(sumOnce(5));

Line 9 calls sumOnce one time. Lines 3 and 4 are the loop. One pass through a loop is called an iteration. Here the loop runs 5 iterations and prints sum=15. With a much bigger number, the loop could run millions of iterations inside that one call.

The lesson Tiered compilation explains how the engine counts how hot code is. The previous lesson, What optimizing compilers do, explains what the fast code does differently. This lesson is only about the switch in the middle of a loop.

Hot loops in cold functions

A function and a loop inside it can be hot or cold on their own:

• A function is cold if it has been called only a few times.
• A loop is hot if it has run many iterations.

So one call to a cold function can hold a very hot loop. Counting calls is not enough to spot it. V8 also counts how often each loop goes round.

The jump from the end of a loop back to its top is called the back edge. V8 counts at the back edge. Each time the loop goes round, V8 can ask: "Is this loop hot enough to switch to faster code?"

1function longOsrLoop(limit) {2  let sum = 0;3  for (let i = 0; i < limit; i += 1) {4    sum += i & 7;5  }6  return sum;7}8 9longOsrLoop(10000000);

Line 9 calls longOsrLoop once. The loop on lines 3 and 4 runs ten million iterations. On line 4, i & 7 turns i into a small number from 0 to 7. It just gives the loop some cheap work to do.

Node can print a note each time it does OSR. You turn this on with the flag --trace-osr. A flag is an extra word you type after node to change how it runs.

These notes come from V8, not from the JavaScript language. Your app should never depend on when OSR happens, or on it happening at all.

OSR enters at the top of the loop

Optimized code usually expects to start at the top of the function. The function is called, its variables are set up, and then line 1 runs. OSR can't do that, because the function is already halfway through.

So the engine builds the optimized code with a second way in, called an OSR entry. It is placed at the loop header: the spot at the top of the loop where the next iteration begins.

Before V8 runs your code, it turns it into bytecode: a simple list of small steps that the interpreter runs one by one. You can ask Node to print that list.

1$ node --print-bytecode --print-bytecode-filter=longOsrLoop probe.js2[generated bytecode for function: longOsrLoop]3... JumpLoop ..., [0], ...

Line 1 is the command. The flag --print-bytecode-filter=longOsrLoop prints the bytecode for that one function only. Line 3 shows the step that matters, JumpLoop. It is the back edge: the jump from the end of the loop back to its top. The ... stands for numbers that change from run to run, so the test does not check them.

Each time JumpLoop runs, the loop starts its next iteration, so it is a good place to ask "switch to faster code now?" The name JumpLoop belongs to V8 12.4. It is not part of JavaScript, and a future V8 could rename it.

A function can be entered in three ways. Read each row as: "this way in starts here, and is used when this happens."

Carry the variables across

Spotting a hot loop is the easy part. The hard part is the switch itself. The fast code must start with the exact values the variables have right now. Say the interpreter has reached i = 5 and sum = 10. The fast code must start with those same two values. If it didn't, the function would give a wrong answer.

The replay below is a small model written in plain JavaScript. It does not look inside a real engine. Instead, a variable called tier says which kind of code is running, and osrBudget says at which iteration the model switches.

1const osrBudget = 5;Run2function sumWithOsrModel(limit) {3  let tier = "interpreter";4  let sum = 0;5  for (let i = 1; i <= limit; i += 1) {6    if (tier === "interpreter" && i === osrBudget) {7      tier = "optimized";8      console.log("OSR carries i=" + i + ", sum=" + sum);9    }10    sum += i;11  }12  return { tier, sum };13}14 15const report = sumWithOsrModel(8);16console.log(report.tier);17console.log(report.sum);

Step through it:

• Line 15 calls the function once.
• Iterations 1 to 4 run in the interpreter and add up to sum = 10.
• Line 6 is the check at the top of the loop. At i = 5, it says "switch now".
• Line 7 switches tier to "optimized".
• Line 8 prints the values that are carried across: i=5 and sum=10.
• Iterations 5 to 8 finish in optimized code, and line 17 prints 36.

1function runTeachingOsr(iterations, budget) {2  let tier = "interpreter";3  let sum = 0;4  let switchIteration = null;5  for (let i = 1; i <= iterations; i += 1) {6    if (tier === "interpreter" && i === budget) {7      switchIteration = i;8      tier = "optimized";9    }10    sum += i;11  }12  return { switchIteration, tier, sum };13}

Try other numbers. With a budget of 6 and 12 iterations, the interpreter runs 5 iterations, then optimized code runs 7. If the budget is bigger than the number of iterations, the loop never switches.

Going back to slower code in the middle of a loop

OSR moves a running loop from the interpreter to optimized code. The engine can also move the other way. That is called deoptimization, or deopt for short.

Optimized code is fast because it makes guesses, such as "every value here is a number". Before it uses a value, it runs a quick check called a guard. If the guard fails, the guess was wrong. The engine then goes back to the interpreter, at the exact step where it stopped, with the current values. The lesson Speculation & deoptimization covers guesses and deopt in full. Here we only look at the switch in the middle of a loop.

1function sumWithDeoptModel(values) {2  let mode = "optimized-number-loop";3  let total = 0;4  for (let i = 0; i < values.length; i += 1) {5    const value = values[i];6    if (mode === "optimized-number-loop" && typeof value !== "number") {7      mode = "interpreter";8      console.log("deopt carries i=" + i + ", total=" + total);9    }10    total += value;11  }12  return total;13}14 15console.log(sumWithDeoptModel([2, 3, null, 5]));Run

This model starts in optimized code that only handles numbers:

• The values 2 and 3 pass, so total is 5.
• On line 6, the guard sees null, which is not a number, so the check fails.
• Line 7 switches back to the interpreter.
• Line 8 prints the values that are carried back: i=2 and total=5.
• Line 10 goes on with JavaScript's normal adding rules. 5 + null is 5, because null counts as 0 here. Then 5 more gives 10.

You can see a real deopt in Node too. The code below uses two test-only tools:

• %OptimizeFunctionOnNextCall(sumNumbers) on line 11 tells V8 to build optimized code for sumNumbers now. It only works when Node runs with the flag --allow-natives-syntax.
• The flag --trace-deopt prints a note each time V8 deoptimizes.

1function sumNumbers(values) {2  let total = 0;3  for (let i = 0; i < values.length; i += 1) {4    total += values[i];5  }6  return total;7}8 9const numbers = [1, 2, 3, 4];10for (let i = 0; i < 100000; i += 1) sumNumbers(numbers);11%OptimizeFunctionOnNextCall(sumNumbers);12sumNumbers(numbers);13sumNumbers([1, "2", 3, 4]);

Line 10 calls sumNumbers 100,000 times with numbers only. Line 11 asks for optimized code. Line 13 passes the text "2" instead of a number. A check inside the loop fails, and V8 deoptimizes.

1[bailout (kind: deopt-eager, reason: ...): begin. deoptimizing ... <JSFunction sumNumbers ...>, ... bytecode offset ...]

The real note is long and noisy. The test only checks three parts of it:

• deoptimizing: V8 is going back to slower code.
• sumNumbers: the function it happened in.
• bytecode offset: the position in the bytecode list where the interpreter picks up again.

Memory addresses, ID numbers, and timing change from run to run, so the test ignores them. The wording may also change in later V8 versions.

What the V8 tests really show

The tests for this lesson run on Node 22, which uses V8 12.4. They check that process.version starts with v22. and that process.versions.v8 starts with 12.4.. They don't check exact version numbers, memory addresses, bytecode positions, or times, because those change from run to run.

These are facts about V8 12.4, not rules of the JavaScript language. Other engines, like the ones in Firefox and Safari, switch code in their own ways. The JavaScript rules only promise one thing: the same code gives the same result.

Why timing a small loop can fool you

A benchmark is code that runs a piece of work many times to measure how fast it is. A small benchmark that times one loop often measures the engine's switching instead of your code. Here is what can go wrong:

• The first iterations run in the slow interpreter. Later ones may run in optimized code after OSR.
• If nothing uses the result, the optimized code may skip or change the work.
• If the input never changes, the work may be easier than it is in the real app.
• Garbage collection, the engine pausing to free memory that is no longer used, can add random delays.
• Timers can't measure very short times well.

1function cartTotal(items) {2  let total = 0;3  for (let i = 0; i < items.length; i += 1) {4    total += items[i];5  }6  return total;7}8 9const basket = [10, 20, 30];10for (let sample = 0; sample < 1000; sample += 1) {11  cartTotal(basket);12}13console.log("bad benchmark kept result:", false);

Line 9 makes one basket of prices. Lines 10 to 12 call cartTotal 1,000 times. The bug is on line 11: the total that cartTotal returns is thrown away. Line 13 prints false to say so. The example prints this instead of a time, because times differ on every computer, so a test can't check them.

1function cartTotal(items) {2  let total = 0;3  for (let i = 0; i < items.length; i += 1) {4    total += items[i];5  }6  return total;7}8 9const baskets = [[10, 20, 30], [5, 15, 25], [7, 11, 13]];10let sink = 0;11for (let warmup = 0; warmup < 100; warmup += 1) {12  sink += cartTotal(baskets[warmup % baskets.length]);13}14for (let sample = 0; sample < 1000; sample += 1) {15  sink += cartTotal(baskets[sample % baskets.length]);16}17console.log("better benchmark kept result:", sink > 0);18console.log("sample total:", cartTotal(baskets[1]));

This version fixes the bad one:

• Line 9 makes three different baskets, so the input changes.
• Line 10 makes sink, a variable that collects every result. Because the results are used, the engine must really do the work.
• Lines 11 and 12 warm up: they run the work 100 times first, so the switching is over before you measure.
• Lines 14 and 15 are the runs you would time.
• Line 17 prints true, and line 18 prints 45, the total of the second basket.

This is still only an outline. A real benchmark also needs a timer, many runs, and the middle value of those runs. For the full steps, see Measuring performance.

Each row of this table is one trap. Read it as: "this goes wrong, so do this instead."

What to do with this in real work

You almost never need to cause OSR or check for it. What helps is knowing three things:

• Why benchmarks need a warm-up.
• Why one long loop can speed up while it is running.
• Why you should trust a measurement more than a rule of thumb.

If a loop really is slow, keep the code easy to read first. Then measure it with real data. Use this checklist when you time code:

• Warm up with the same kinds of values the measured runs will use.
• Use the result, so the engine must really do the work you want to measure.
• Change the inputs so they look like real data, such as different sizes and shapes.
• Run many times and report the middle value, not one surprising number.
• Don't ship V8 flags, test-only commands like %OptimizeFunctionOnNextCall, or code that counts on when OSR happens.

Measure what the user actually waits for first. Knowing how the engine works helps you explain a measurement. It is not a list of rules for how to write code.

Common misconceptions

• "OSR is a JavaScript feature." It is something the engine does on its own. Your code can't call it.
• "A function must be called many times before it gets faster." A loop can get hot inside one call. That is exactly when OSR helps.
• "OSR starts the loop again." The loop goes on from its top, with the variables keeping their current values.
• "A deopt is a crash." It is a safe way back to slower code, and the answer stays right.
• "Timing a small loop shows how fast my app is." It may mostly show warm-up, OSR, timer limits, or skipped work.

The table below compares four words that are easy to mix up. Read each row as: "this word means this, and it is not that."

Practice exercises

function sumOnce(limit) {
  let sum = 0;
  for (let i = 1; i <= limit; i += 1) sum += i;
  return sum;
}
console.log(sumOnce(6));

function countTiered(iterations, switchAt) {
  let tier = "interpreter";
  let optimized = 0;
  for (let i = 1; i <= iterations; i += 1) {
    if (i === switchAt) tier = "optimized";
    if (tier === "optimized") optimized += 1;
  }
  return optimized;
}
console.log(countTiered(8, 5));

function countTiered(iterations, switchAt) {
  let tier = "interpreter";
  let optimized = 0;
  for (let i = 1; i <= iterations; i += 1) {
    if (i === switchAt) tier = "optimized";
    if (tier === "optimized") optimized += 1;
  }
  return optimized;
}
console.log(countTiered(8, 5));

Check your understanding

Key takeaways

• OSR lets a loop that is already running switch to optimized code, without waiting for the next call.
• The switch happens at the top of the loop. Variables such as i and sum keep their current values.
• A deopt goes the other way. When a guess fails in the middle of a loop, the engine goes back to the interpreter at the same step.
• Timing a small loop can fool you. Warm-up, OSR, unused results, inputs that never change, garbage collection, and timers can all get in the way.
• Use what you know about OSR to explain measurements. Don't write code that counts on one engine's timing.

Up next: Tagged values, Smis & heap numbers. You'll see how engines store JavaScript values in the computer's memory.