Speculation & deoptimization

Make the usual case fast, keep a way back

The engine is the program that runs your JavaScript. Chrome and Node use one called V8. When a function runs many times, the engine calls it hot. It then tries to turn the hot function into fast machine code: instructions the computer's processor runs directly.

There is a problem. In JavaScript, the same variable can hold a number now and a string later. Fast machine code works best when it knows the kind of value in advance. So the engine makes a bet.

• It looks at the values a function has seen so far.
• It makes fast code for those values only.
• It adds a quick check, in case a different value shows up later.

This bet is called speculation. If the bet turns out wrong, the engine throws the fast code away and goes back to its slower, general code. That step back is called deoptimization, or deopt for short.

Two earlier lessons explain how the engine collects its notes about values: Type feedback and Inline caches. The lesson before this one, Optimizing compilers, explains the part that makes the fast code. This lesson is about the checks that keep the fast code safe, and what happens when one fails.

Speculative optimization: a fast path with a check

A speculative plan has three parts:

• Feedback: notes about which values reached one operation, like one +. The place where the engine keeps the notes for one operation is called a feedback slot.
• Fast code made for those values. This is the fast path.
• A guard: a quick check at the start of the fast code. It asks, "Do these new values still fit?" If not, the engine takes the slow path: its general code, which works for any value.

V8 has a short name for small whole numbers, such as 10 or -3: Smis, short for "small integers". Fast code for Smis can add two numbers in one quick processor step.

1const feedback = createFeedback("total + fee");Run2observe(feedback, 10, 2);3const optimized = compileWithGuards(feedback);4optimized(12, 3);5optimized("12", "3");

This replay is a small model written in JavaScript. It does not look inside your browser's engine. Press Step to go line by line:

1. Line 1 makes an empty feedback slot for the total + fee operation.
2. Line 2 adds 10 and 2. The notes now say SignedSmall: V8's name for "only Smis so far".
3. Line 3 makes fast code from those notes, with a guard in front.
4. Line 4 calls the fast code with 12 and 3. The guard passes, and the result is 15.
5. Line 5 calls it with the strings "12" and "3". The guard fails. The engine leaves the fast code, and the slow path joins the two strings into "123".

The answer "123" is exactly what JavaScript always gives for "12" + "3". A deopt changes which code the engine runs. It never changes what your code means.

1function totalCents(price, tax) {2  return price + tax;3}4 5console.log(totalCents(100, 8));6console.log(totalCents(250, 20));

Line 2 has the + we care about. Lines 5 and 6 call totalCents with two numbers each time, so this + always adds numbers. The code prints 108, then 270.

This does not mean "never use strings". It means: try not to make one busy + switch between adding numbers and joining strings, unless your program really needs both.

The feedback lattice: notes only get wider

The engine sorts its notes for one + into a few states. The states form a ladder, from narrow to wide. A narrow state fits only a few kinds of values. A wide state fits more. This ladder is called a lattice. When a new kind of value arrives, the state moves up. It does not move back down.

V8 calls the notes for an operation with two sides, like a + b, BinaryOp feedback. In the V8 version inside Node 22 (V8 12.4), its states for + include None, SignedSmall, Number, NumberOrOddball, and Any. V8 calls undefined, null, true, and false oddballs. The real compiler splits some cases further, but this ladder is a safe way to picture it.

Read each row of the table from left to right: the state's name, the values the + has seen, and the fast code the engine can make.

The next code runs only in Node, not in the browser editor. It needs the flag --allow-natives-syntax. That flag turns on V8's own test functions, whose names start with %. %DebugPrint prints what V8 knows about a function, including its feedback.

1function onlySmi(a, b) { return a + b; }2for (let i = 0; i < 30; i += 1) onlySmi(i, i + 1);3console.log("ONLY_SMI");4%DebugPrint(onlySmi);5 6function smiThenDouble(a, b) { return a + b; }7for (let i = 0; i < 30; i += 1) smiThenDouble(i, i + 1);8smiThenDouble(1.5, 2.25);9console.log("SMI_DOUBLE");10%DebugPrint(smiThenDouble);11 12function numberOddball(a, b) { return a + b; }13for (let i = 0; i < 30; i += 1) numberOddball(undefined, i);14console.log("ODDBALL");15%DebugPrint(numberOddball);

1. Lines 1 to 4: onlySmi adds small integers 30 times. V8 prints BinaryOp:SignedSmall.
2. Lines 6 to 10: smiThenDouble also gets 1.5 and 2.25 on line 8. V8 prints BinaryOp:Number.
3. Lines 12 to 15: numberOddball adds undefined to a number. V8 prints BinaryOp:NumberOrOddball.

1const slot = createSlot("BinaryOp slot #0");2feed(slot, 1, 2);          // SignedSmall3feed(slot, 1.5, 2.25);     // Number4feed(slot, undefined, 1);  // NumberOrOddball5feed(slot, "7", "8");     // Any

Press "Run the next value" to make seven calls in a set order, or press one call button at a time. Then press Reset and try strings first. A + that has only ever joined strings gets its own state, String. But once the same + has seen both numbers and strings, this model moves it to Any.

Guards, checks, and watchpoints

Fast code can rely on two kinds of facts. Some facts are about the values in this call. Others are about things outside the function.

Checks inside the fast code test the values of this call, right where they are used. Three common ones:

• A Smi check asks, "Is this value still a small integer?"
• A map check asks, "Does this object still have the same layout?" V8 gives every object a hidden map: a record of which properties it has, in which order.
• A bounds check asks, "Is this index still inside the array?"

A watchpoint works differently. It is a note the engine keeps about a fact outside the function, such as "this prototype won't change". (A prototype is the object that other objects borrow shared properties from.) The fast code does not check the fact on every call. Instead, if some other code changes that fact, the engine marks every piece of fast code that relied on it as out of date. "Watchpoint" is a word several engines use. V8's own messages call these code dependencies. The exact details differ from engine to engine.

Read each row of the table as: "this kind of check sits here, asks this, and this happens if it fails."

Eager and lazy deopts

A deopt can happen at two different times:

• An eager deopt happens right at the failing guard. The fast code is in the middle of running, and the value it has is the wrong kind. It cannot go on, so the engine leaves at once.
• A lazy deopt happens later. First, other code changes a fact the fast code relied on, and the engine marks the fast code out of date. The code is left afterwards, at a safe point: a moment the engine picks where it can switch safely.

Leaving fast code in the middle of a call takes some work. Each running call has a frame: a record of its variables and the line it has reached. The engine keeps extra notes about where each variable sits inside the fast code. It uses those notes to rebuild the frame for its slower code, with every variable in place.

1const menu = makeMenu({ taxRate: 0.05 });Run2const plan = compilePriceReader(menu);3plan({ subtotal: 100 });4changeTaxRate(menu, 0.12);5plan({ subtotal: 100 });

This replay is a small model too. Press Step to go line by line:

1. Line 1 makes a menu with a tax rate of 0.05.
2. Line 2 makes fast code that relies on that tax rate. A watchpoint now protects the fact.
3. Line 3 runs the fast code. It prints 105, which is 100 × 1.05.
4. Line 4 changes the tax rate to 0.12. It does not call the fast code. It only marks it out of date.
5. Line 5 calls again. The model sees the mark and uses the slow path. It prints 112.00000000000001: about 112, with a tiny rounding error from how computers store decimals.

Real engines choose their own safe point. A V8 message about a code dependency proves the fast code was marked. It does not tell you the exact moment the code was left.

Read each row of the table as: "this kind of deopt starts like this, shows this in the trace, and often comes from these causes." A trace is a log the engine can print about what it is doing.

Deopt loops

A deopt loop is when the same code goes round and round. The engine makes fast code for one kind of value. Soon another kind arrives, and the engine deopts. It collects new notes, makes fast code again, and the same thing happens. All that work is wasted. Engines count these rounds, and after a few they may wait longer or stop making fast code for that function.

1function addOne(value) {2  return value + 1;3}4 5const values = [1, "1", 2, "2"];6console.log(values.map(addOne).join("|"));

Line 2 has the +. Line 5 makes a list that switches between numbers and strings. Line 6 sends every item through addOne, then joins the results with |. It prints 2|11|3|21: 1 + 1 is 2, but "1" + 1 joins into "11".

Four calls are far too few for the engine to make fast code. But if a hot + kept switching like this thousands of times, the circle in the picture could happen. Your fix is usually simple: keep the values going into a hot operation the same kind, or split different kinds into different functions.

• Convert data from a server into the right kind once, before it reaches a hot loop.
• Don't send many different kinds of values or object layouts through one hot operation.
• Move rare cases into their own function when that makes the code clearer and your timings show it matters.
• Keep code easy to read. Don't twist code to silence a trace line on code that is not hot.

Seeing deopts in Node

Node can print V8's deopts if you start it with special flags (options typed after node). The examples below run only in Node, so they can't run in the browser editor. The flags they use:

• --trace-deopt prints a line each time V8 deopts.
• --trace-opt prints a line each time V8 makes fast code.
• --allow-natives-syntax turns on V8's test functions that start with %.
• --no-concurrent-recompilation makes V8 build fast code right away, so the output comes in a steady order.

The lesson's test runs these examples in Node and checks only facts that stay the same between runs: Node's version starts with v22., V8's starts with 12.4., an eager deopt line has a kind and a reason, and an out-of-date line mentions code dependencies.

1function add(a, b) {2  return a + b;3}4 5%PrepareFunctionForOptimization(add);6for (let i = 0; i < 100000; i += 1) {7  add(i, i + 1);8}9%OptimizeFunctionOnNextCall(add);10add(1, 2);11add("x", "y");

1. Line 5: %PrepareFunctionForOptimization tells V8 to start keeping notes for add.
2. Lines 6 to 8 call add 100,000 times with small integers.
3. Line 9: %OptimizeFunctionOnNextCall tells V8 to make fast code on the next call.
4. Line 10 runs the fast code with 1 and 2. The guard passes.
5. Line 11 sends the strings "x" and "y". The Smi guard fails, and V8 deopts.

$ node --allow-natives-syntax --no-concurrent-recompilation --trace-deopt probe.js
[bailout (kind: deopt-eager, reason: not a Smi): begin. deoptimizing ...]

bailout means "left the fast code". kind: deopt-eager says it happened right at a guard. reason: not a Smi says which check failed. The ... stands for details that change from run to run.

1const proto = { taxRate: 0.05 };2const menu = Object.create(proto);3menu.subtotal = 100;4function price(row) {5  return row.subtotal * (1 + row.taxRate);6}7%PrepareFunctionForOptimization(price);8for (let i = 0; i < 100000; i += 1) price(menu);9%OptimizeFunctionOnNextCall(price);10price(menu);11proto.taxRate = 0.12;12price(menu);

1. Lines 1 to 3 make proto with a tax rate, and a menu that borrows from it. Object.create(proto) makes a new object whose prototype is proto.
2. Lines 4 to 6: price reads row.taxRate, which it finds on the prototype.
3. Lines 7 to 10 warm up price, make fast code, and run it once. V8 assumes the prototype's tax rate won't change.
4. Line 11 changes proto.taxRate. V8 marks the fast price code as out of date.
5. Line 12 calls price again, after the mark.

$ node --allow-natives-syntax --trace-deopt probe.js
[marking dependent code ... for deoptimization, reason: code dependencies]

The clue is marking dependent code ... for deoptimization with reason: code dependencies: fast code that depended on the prototype was marked.

1function add(a, b) { return a + b; }2%PrepareFunctionForOptimization(add);3add(1, 2);4%OptimizeFunctionOnNextCall(add);5add(3, 4);6const status = %GetOptimizationStatus(add);7console.log("status is number", typeof status === "number");

Lines 2 to 5 prepare add, call it, ask for fast code, and call it again. Line 6 asks V8 for a status number with %GetOptimizationStatus. Line 7 only checks that it is a number and prints status is number true. The exact number changes between V8 versions, so the test doesn't rely on it.

Habits for real code

Most frontend developers don't need to chase every deopt. This knowledge helps when you have measured a slow, hot path: a parser, the code that adds up chart data, a function that formats every cell of a big table, or an animation loop. In that code, keep the values plain and predictable.

Read each row of the table as: "this habit means doing this, and it helps because of this."

1function toCents(value) {2  return Number(value);3}4const inputs = ["100", "50", "25"];5console.log(inputs.map(toCents).reduce((sum, value) => sum + value, 0));

Lines 1 to 3 make toCents, which turns a value into a number with Number(value). Line 4 is data from a server, where the prices arrive as text. Line 5 first turns every price into a number with toCents, then adds them up with reduce. So the + inside reduce only ever adds numbers. It prints 175. If speed matters, time the real page before and after the change.

Common mix-ups

• "A deopt means my code is wrong." No. It means the engine's bet stopped fitting. Your code still gives the right answer.
• "The notes go back to narrow after a tidy call." Usually not. Notes get wider, so the next fast code stays safe for every value seen.
• "A watchpoint runs my JavaScript function." No. It is a note inside the engine. It marks fast code out of date and never runs your code.
• "Every trace line must be fixed." No. Fix hot paths you have measured. Ignore lines from code that rarely runs.

The table below puts four terms side by side. Read each row as: "this term means this, and it does not mean that."

Practice exercises

The first exercises ask what code prints. The later ones ask you to read a trace and fix real code. Try a guess first, then use the hints if you get stuck.

function totalCents(price, tax) {
  return price + tax;
}

console.log(totalCents(100, 8));
console.log(totalCents(250, 20));

function totalCents(price, tax) {
  return price + tax;
}

console.log(totalCents(100, 8));
console.log(totalCents(250, 20));

let state = "None";
function merge(previous, observed) {
  if (previous === "None") return observed;
  if (previous === observed) return previous;
  if (previous === "SignedSmall" && observed === "Number") return "Number";
  if (previous === "Number" && observed === "NumberOrOddball") return "NumberOrOddball";
  return "Any";
}
state = merge(state, "SignedSmall");
state = merge(state, "Number");
state = merge(state, "NumberOrOddball");
console.log(state);

let state = "None";
function merge(previous, observed) {
  if (previous === "None") return observed;
  if (previous === observed) return previous;
  if (previous === "SignedSmall" && observed === "Number") return "Number";
  if (previous === "Number" && observed === "NumberOrOddball") return "NumberOrOddball";
  return "Any";
}
state = merge(state, "SignedSmall");
state = merge(state, "Number");
state = merge(state, "NumberOrOddball");
console.log(state);

function addFee(value) {
  return value + 1;
}

console.log(addFee(4));
console.log(addFee("4"));

function addFee(value) {
  return value + 1;
}

console.log(addFee(4));
console.log(addFee("4"));

function normalizePrice(value) {
  // return a number here
}

const raw = ["100", "50", "25"];

function toCents(value) {
  return Number(value);
}
const inputs = ["100", "50", "25"];
console.log(inputs.map(toCents).reduce((sum, value) => sum + value, 0));

Quiz

Key takeaways

• The engine makes fast code for the values it has seen, and puts guards in front to check every new value.
• The notes for an operation only get wider. One new kind of value can move an operation up the ladder for good.
• A failing guard causes an eager deopt, right away. A change elsewhere marks fast code out of date, and it is left later: a lazy deopt.
• Use deopt traces to study hot paths you have measured. Keep the kinds of values and object layouts steady where it really helps.

One-line summary: speculation makes the usual case fast, and deoptimization keeps the unusual case correct.

Next, What optimizing compilers do shows the changes the compiler makes to your code once it trusts its bets, such as copying a small function into its caller (inlining) and working out sums ahead of time (constant folding).