Scanning source text

Text becomes a list of tokens

Your JavaScript file is just a long line of characters. Before the engine can run it, it must split that text into small pieces. Each piece is one word or symbol of the language, such as let, total, or =.

The part of the engine that does this splitting is called the scanner. People also call it a lexer or a tokenizer. All three names mean the same thing. It is the very first step, before anything else in the engine happens.

The tree the parser builds is called an AST, short for "abstract syntax tree". You will meet it in the next lesson.

This lesson uses ideas from a few earlier lessons:

• Code structure, for automatic semicolon insertion (ASI).
• Unicode & strings, for UTF-16.
• Text encoding & Base64, for bytes.
• Regex basics, for regex literals like /ok/u.

It also follows Meet the engines, and it leads into Parsing & abstract syntax trees.

A byte is a small number from 0 to 255. Files and network responses are made of bytes. So a browser that downloads a script gets bytes, and a program that reads a file from disk gets bytes too.

Bytes are not text yet. Something must turn them into characters first. This is called decoding, and the rule it follows is called an encoding, such as UTF-8. The browser or runtime does this before the scanner starts. Which encoding it uses depends on the script:

• Modules are always decoded as UTF-8.
• Older classic scripts can follow the page's encoding or a <script charset> attribute.
• A special mark at the very start, the byte order mark (BOM), can name the encoding. The decoder then removes it.

The step-through below makes UTF-8 bytes with TextEncoder, then turns them back into text with TextDecoder.

1const bytes = new TextEncoder().encode('\uFEFFconst café = "😀";\n');Run2const decoded = new TextDecoder("utf-8").decode(bytes);3const codeUnits = Array.from(4  { length: decoded.length },5  (_, index) => decoded.charCodeAt(index).toString(16)6);7console.log(decoded);8console.log(codeUnits.join(" "));

After decoding, nobody counts in bytes any more. Error messages, browser developer tools, and source maps point at places in the decoded text. They count in the same units that JavaScript strings use. You'll see those units in the next section.

The table below keeps four ideas apart. Read each row as: "this is the thing, this is what one of it is, here is a small example, and here is why it matters."

Engines read the text as UTF-16

JavaScript strings store text in small pieces called UTF-16 code units. Most characters, like a or é, take one code unit. The engine reads your source in these same pieces.

The V8 scanner article explains how V8, the engine in Chrome and Node.js, does this. Chrome may hand V8 text stored in different ways. V8 wraps all of them in one reader called UTF16CharacterStream, so the scanner always sees UTF-16. Its method Utf16CharacterStream::Advance() gives the next code unit, or -1 when the text has ended.

Some characters need two code units, like the emoji 😀. Such a pair is called a surrogate pair. You met these in Unicode & strings. V8's article says the scanner only joins a pair back into one character when it must check whether the character may appear in a name. Inside strings and comments, it just copies the code units as they are.

The scanner keeps a cursor: a number that says where it is in the text. It only moves forward. At each spot it does four things:

1. It skips trivia: spaces, tabs, line breaks, and comments. These don't become tokens.
2. It takes the longest token that fits. For example, === is one token, not three = tokens.
3. It writes down where the token starts and ends.
4. It hands the token to the parser and moves the cursor past it.

The kinds of tokens it can make include:

• names, like total, and keywords, like if
• symbols, like = and =>, which the spec calls punctuators
• numbers, including ones with _ separators like 1_000 and BigInts ending in n
• strings, with their escapes, and pieces of template literals
• regex literals like /ok/u, but only where the grammar allows one

The step-through below runs this lesson's own small tokenizer on the line let total = price / 2;.

1const source = "let total = price / 2;";Run2const tokens = tokenizeSubset(source);3console.log(tokens.map((token) => token.type + ":" + token.value).join(" | "));

The table below lists each job a JavaScript scanner has. Read each row as: "for this kind of text, the scanner does this."

One odd case is very old. In classic browser scripts, a line that starts with <!-- is treated as a comment. This rule is in Annex B, the part of the spec that keeps old websites working. Don't use it in new code, and never in modules.

1<!-- an old HTML-style comment, allowed only in classic scripts2console.log("after an Annex B comment");3// Old rule. Don't use it in modules or in new code.

Line 1 is the old HTML-style comment, so the scanner skips it. Line 2 is normal code. Line 3 is a normal comment with a warning.

A name in code, like total, is called an identifier. Here are the rules for which characters it may use:

• The first character must be a letter, $, or _. Unicode calls these letters ID_Start characters.
• Later characters may also be digits and a few other marks. Unicode calls these ID_Continue characters.

You may also write a letter as an escape: \u followed by the letter's number in hexadecimal. For example, \u0061 is the letter a. The scanner turns the escape into its letter before the parser sees the name. So \u0061bc is just the name abc.

1const \u0061bc = 1;2console.log(abc);3 4try {5  new Function("const \\u0069f = 1;");6} catch (error) {7  console.log(error.name + ": " + error.message);8}

Line 1 makes a variable whose name is written \u0061bc. Line 2 reads it as abc and prints 1.

Lines 4 to 8 try something that must fail. new Function(...) turns a string into code at run time, so the error can be caught instead of breaking the whole page. The string spells the keyword if as \u0069f. V8, used by Node 22 and Chrome, prints SyntaxError: Keyword must not contain escaped characters. Acorn, a parser written in JavaScript, also reports a syntax error with its own message. So an escape can't hide a keyword.

Some words are only special in certain places. They are called contextual keywords. let, yield, await, async, of, get, set, and static are examples. For instance, async is special before function, but it can also be a normal variable name. The scanner only hands over the word. The parser decides whether it acts as a keyword in that place.

A slash can mean several things in JavaScript:

• division, as in b / c
• the start of a regex literal, as in /c/
• the start of a comment, as in // or /*
• part of /=, which divides and stores

Comments and /= are easy, because the next character tells them apart. But division and a regex can look the same. The scanner can't decide from the slash alone.

So the parser helps. The spec gives the scanner two modes, called goals. InputElementDiv means "a slash here is division". InputElementRegExp means "a slash here starts a regex". The parser knows where it is in the grammar, so it tells the scanner which mode to use for the next token.

1let a, b = 20, c = 4, d = 2;2a = b / c / d;3console.log(a);4 5try {6  new Function('let a, b = true, d = "abc"; a = b\n/c/.test(d);');7} catch (error) {8  console.log(error.name + ": " + error.message);9}10 11try {12  let x, y = 10;13  x = y / 2 / g;14} catch (error) {15  console.log(error.name + ": " + error.message);16}

Here is what each puzzle prints:

• Line 2 is normal division: 20 / 4 / 2. Line 3 prints 2.5.
• Line 6 puts a line break before /c/. It still reads as division, because a = b may go on with an operator on the next line. So the code means a = b / c / .test(d), and the dot is a mistake. V8 prints SyntaxError: Unexpected token '.'.
• Line 13 looks like it ends with a g flag, but both slashes are division. JavaScript looks for a variable named g, finds none, and throws a ReferenceError.

The panel below runs five slash cases through two tokenizers. "lesson" is this lesson's small tokenizer. "acorn" is the real Acorn tokenizer.

The lesson tokenizer uses a shortcut: it only looks at the token just before the slash. Look at the if (ready) case. The lesson tokenizer sees ) and guesses division. That is wrong. Acorn knows that if (ready) is finished and a new statement starts. So Acorn reads /c/ as one regex token, which is right.

Looking at the token before the slash is a good clue for a lesson or a simple color highlighter. But it can't know everything the parser knows. For tools that change, check, or shrink code, use a tokenizer that comes with a real parser.

This lesson's tokenizer is small, but it is real code. It can:

• skip spaces and comments
• read names, keywords, numbers, BigInts, strings, and template literals
• read symbols such as = and =>

To choose between a regex and division, it only looks at the token before the slash. This shortcut is on purpose, and it is sometimes wrong. The lab shows Acorn next to it, so you can see exactly where the shortcut and real JavaScript disagree.

1function tokenizeSubset(source) {2  // The full lesson code also reads comments, strings, numbers, and names.3  // For a slash, it only looks at the token before it, not the grammar.4  const tokens = [];5  let cursor = 0;6  let previous = null;7 8  while (cursor < source.length) {9    const next = scanOneToken(source, cursor, previous);10    if (next) {11      tokens.push(next.token);12      previous = next.token;13      cursor = next.end;14    } else {15      cursor += 1;16    }17  }18 19  return tokens;20}

Line 4 starts an empty list of tokens. Line 8 repeats until the cursor reaches the end of the text. Line 9 reads one token, and line 10 checks that it found one. Line 11 adds the token to the list. Line 12 remembers it as previous, and line 13 moves the cursor past it.

1let café = 1_000n;2const ok = /ok/u.test(name);3if (ready) /c/.test(name);4x = y / 2 /g;

A fast scanner makes pages start faster

No code can run until it has been scanned. So engines work hard to make scanning fast. The V8 team reported these speed-ups from its scanner work:

• Reading a single token: about 1.4× faster.
• Reading strings: about 1.3× faster.
• Reading multi-line comments: about 2.1× faster.
• Reading names: about 1.2–1.5× faster, depending on how long the name is.

The table below lists the tricks behind those numbers. Read each row as: "V8 does this, and this is why it helps."

You don't need to change how you write code for this. Write clear code, and let your build tool shrink it for production. Names made only of ASCII characters (a–z, A–Z, digits, $, and _) take V8's fastest path, so only use other letters in names when you have a real reason. This work is about how fast pages and tools start, not about tuning one variable name by hand.

Where you meet tokens in real tools

Many tools you use every day split code into tokens first:

• A syntax highlighter colors code. It needs to know that /ok/u is a regex.
• A minifier makes code smaller. It removes comments, but it must keep line breaks that change where semicolons go.
• A linter finds mistakes. It needs the exact start and end of an escaped keyword to point at it.
• Formatters, bundlers, source maps, and browser error messages also use token positions.

1const total = 1_000n;2const word = "token";3const okay = /tok(en)?/u.test(word);4console.log(total, word, okay);

Line 1 has a keyword, a name, =, and the BigInt 1_000n. Line 2 has a string. Line 3 has a regex, /tok(en)?/u. Line 4 prints 1000n token true.

The next lesson, Parsing & abstract syntax trees, takes these tokens and builds a tree from them: declarations, expressions, statements, and early errors.

Common misconceptions

The table below lists code that looks like one thing but is another. Read each row as: "this looks like X, but it is really Y, and here is a safe way to think about it."

Key takeaways

• Code arrives as bytes. The browser or runtime decodes them into text before scanning starts.
• Token positions count UTF-16 code units, the same units that JavaScript strings use.
• The scanner skips spaces and comments, takes the longest token that fits, and remembers line breaks for ASI.
• An escape in a name becomes its letter, but a keyword written with an escape is a SyntaxError.
• The parser decides whether a slash is a regex or division. The token before it is only a clue.
• Scanning comes before everything else, so a fast scanner makes code start sooner.

In one line: the scanner turns your text into tokens, and the parser decides what they mean together.

Next, Parsing & abstract syntax trees shows how these tokens become a tree the engine can compile.