Scanning source text

Raw text becomes a token stream

Before JavaScript can build an AST, compile bytecode, or optimize a hot function, it has to answer a smaller question: where does one meaningful piece of source text end and the next one begin? The scanner, often called a lexer or tokenizer, performs that first pass over decoded source text.

This lesson builds on Code structure for automatic semicolon insertion, Unicode & strings for UTF-16, Text encoding & Base64 for bytes, and Regex basics for regex literals. It also points back to Meet the engines and ahead to Parsing & abstract syntax trees.

A browser usually receives JavaScript as network bytes. A runtime reading from disk also starts with bytes. The host chooses a character encoding before the engine scans syntax: modules are decoded as UTF-8 by default, classic scripts still have legacy charset rules such as a document encoding or <script charset>, and a byte order mark can identify or be stripped from the beginning of the stream.

1const bytes = new TextEncoder().encode('\uFEFFconst café = "😀";\n');Run2const decoded = new TextDecoder("utf-8").decode(bytes);3const codeUnits = Array.from(4  { length: decoded.length },5  (_, index) => decoded.charCodeAt(index).toString(16)6);7console.log(decoded);8console.log(codeUnits.join(" "));

Once decoding finishes, the original byte offsets are no longer the scanner's main coordinate system. Parser locations, syntax errors, devtools ranges, and source maps usually talk in line/column pairs or UTF-16 offsets because that is how JavaScript strings index text.

Engines scan a UTF-16 view

The V8 scanner article describes a UTF16CharacterStream abstraction. Chrome may hand V8 Latin-1, UTF-8, or UTF-16-backed data, and the stream presents a UTF-16 view to the scanner. The blog names the low-level method Utf16CharacterStream::Advance(): it returns the next UTF-16 code unit or -1 at end of input.

That does not mean every Unicode character is one unit. Astral characters use surrogate pairs, just as you saw in Unicode & strings. V8's post explains that the scanner avoids combining surrogate pairs until it needs Unicode identifier checks, because strings and comments can often be copied as code units.

A scanner walks forward. At each cursor position it skips trivia, chooses the longest token that fits, records a source range, and hands the token to the parser. It recognizes identifiers, keywords, punctuators, numeric literals with separators or n, string escapes, template chunks, and regex literals when the grammar allows them.

1const source = "let total = price / 2;";Run2const tokens = tokenizeSubset(source);3console.log(tokens.map((token) => token.type + ":" + token.value).join(" | "));

In classic browser scripts, Annex B also permits HTML-like comments such as a line starting with <!--. Treat that as legacy compatibility, not modern style, and do not rely on it in modules.

1<!-- valid as an HTML-like comment in classic scripts2console.log("after an Annex B comment");3// Modules and non-browser tooling should not rely on this legacy form.

Identifier starts use Unicode's ID_Start property, plus $ and _. Later characters use ID_Continue, which also admits digits and a few join controls. Escapes are cooked before the parser sees the name, so \u0061bc is the identifier abc.

1const \u0061bc = 1;2console.log(abc);3 4try {5  new Function("const \\u0069f = 1;");6} catch (error) {7  console.log(error.name + ": " + error.message);8}

The second half proves the engine-specific V8 message used by Node 22 and Chrome: SyntaxError: Keyword must not contain escaped characters. Acorn reports the same source as a syntax error too, with its own wording. Either way, an escape cannot hide a reserved keyword.

Some words are contextual. let, yield, await, async, of, get, set, and static gain special meaning only in particular grammar positions. The parser decides those contexts after the scanner has delivered the token text.

A slash can begin a regex literal, divide two expressions, start a comment, or combine with =. The scanner cannot decide from the slash alone. The ECMAScript lexical grammar has goal symbols such as InputElementDiv and InputElementRegExp; in practice, the parser drives the scanner by asking for the kind of input element that is valid in the current grammar position.

1let a, b = 20, c = 4, d = 2;2a = b / c / d;3console.log(a);4 5try {6  new Function('let a, b = true, d = "abc"; a = b\n/c/.test(d);');7} catch (error) {8  console.log(error.name + ": " + error.message);9}10 11try {12  let x, y = 10;13  x = y / 2 / g;14} catch (error) {15  console.log(error.name + ": " + error.message);16}

Line 2 is ordinary division. The newline puzzle still fails as division because a = b can continue with a binary operator on the next line; V8 reports SyntaxError: Unexpected token '.'. The /g puzzle is also division, so runtime looks for a variable named g and throws ReferenceError.

The lesson's tokenizer is intentionally small but real: it skips comments and whitespace, recognizes identifiers, keywords, numbers, BigInts, strings, templates, punctuators, and chooses regex versus division from the previous token. It is not a full JavaScript implementation; Acorn is here so you can see exactly where a production parser differs.

1function tokenizeSubset(source) {2  // Real lesson code also handles comments, strings, numbers, identifiers,3  // and a previous-token decision for slash. This excerpt shows the loop shape.4  const tokens = [];5  let cursor = 0;6  let previous = null;7 8  while (cursor < source.length) {9    const next = scanOneToken(source, cursor, previous);10    if (next) {11      tokens.push(next.token);12      previous = next.token;13      cursor = next.end;14    } else {15      cursor += 1;16    }17  }18 19  return tokens;20}

1let café = 1_000n;2const ok = /ok/u.test(name);3x = y / 2 /g;

Scanner performance is parser performance

Scanning happens before code can run, so engines spend real effort making it fast. V8 reports that its scanner improvements made single-token scanning roughly 1.4× faster, string scanning 1.3× faster, multiline comment scanning 2.1× faster, and identifier scanning 1.2–1.5× faster depending on identifier length.

The developer-facing advice is modest: write clear code, let your build pipeline minify production bundles, and avoid non-ASCII identifiers when there is no product reason for them. The engine work is about startup cost and tooling scale, not micro-optimizing one variable name by hand.

Where this shows up in real tools

Syntax highlighters, formatters, linters, minifiers, bundlers, source-map tools, and browser error messages all depend on token boundaries. A highlighter wants to know that /ok/u is a regex literal; a minifier wants to remove comments without accidentally changing ASI; a linter wants to report the exact range of an escaped keyword error.

1const total = 1_000n;2const word = "token";3const okay = /tok(en)?/u.test(word);4console.log(total, word, okay);

The next lesson, Parsing & abstract syntax trees, takes these tokens and builds nested structure: declarations, expressions, statements, and early errors.

Common misconceptions

Key takeaways

• Bytes are decoded before scanning; JavaScript source positions are based on decoded text, usually UTF-16 code units.
• The scanner skips trivia, recognizes the longest valid token, and keeps line-break facts for ASI.
• Identifier escapes are cooked, but escaped keywords still produce syntax errors.
• Slash is decided by grammar context: regex in regex-allowed positions, division after completed expressions.
• Scanner performance matters because no parser, compiler, or optimizer can run before tokenization starts.

Next, Parsing & abstract syntax trees shows how these tokens become a tree the engine can compile.