Parsing & abstract syntax trees

The parser's job

The scanner turns source text into tokens. The parser consumes those tokens, checks the grammar, reports syntax and early errors, and builds a tree. Later phases can compile that tree to bytecode, while tools can inspect a similar tree to lint, format, transform, or bundle your code.

This lesson builds on code structure, the precedence table from other operators, the Acorn module graph in bundlers, and the AST transform pipeline in transpilers & polyfills. We will also connect to linting, destructuring, arrow functions, and strict mode.

A recursive-descent parser is organized as functions that match grammar rules. A parser for declarations calls a parser for statements. A parser for statements calls a parser for expressions. Parentheses, blocks, function bodies, and nested calls naturally become recursive calls.

Expressions need one extra trick: precedence. The mini parser in this lesson uses a Pratt parser, also called precedence climbing. Each binary operator has a binding power. The parser reads a left operand, then keeps consuming operators whose binding power is high enough for the current context.

1const input = "a + b * c(d)";2const tokens = tokenize(input);3let cursor = 0;4 5function parseExpression(minBP = 0) {6  let left = parsePostfix();7  while (isBinary(peek()) && bindingPower(peek()) >= minBP) {8    const operator = consume();9    const right = parseExpression(nextMinBP(operator));10    left = binary(left, operator, right);11  }12  return left;13}14 15function parsePostfix() {16  let expression = parsePrimary();17  while (peek() === "(" || peek() === ".") {18    expression = parseCallOrMember(expression);19  }20  return expression;21}22 23function parsePrimary() {24  return tokenToNode(consume());25}26 27function tokenize(source) {28  return source.match(/[A-Za-z_$][A-Za-z0-9_$]*|[0-9]+|[*][*]|[()+*/.,-]/g).concat("<eof>");29}30function peek() { return tokens[cursor]; }31function consume() { return tokens[cursor++]; }32function isBinary(token) { return ["+", "-", "*", "/", "**"].includes(token); }33function bindingPower(token) {34  return token === "+" || token === "-" ? 10 : token === "*" || token === "/" ? 20 : token === "**" ? 30 : -1;35}36function nextMinBP(operator) { return operator === "**" ? bindingPower(operator) : bindingPower(operator) + 1; }37function tokenToNode(token) { return /^[A-Za-z_$]/.test(token) ? { type: "Identifier", name: token } : { type: "Literal", value: Number(token), raw: token }; }38function parseCallOrMember(expression) {39  if (peek() === ".") return { type: "MemberExpression", object: expression, property: tokenToNode((consume(), consume())) };40  consume();41  const args = peek() === ")" ? [] : [parseExpression(0)];42  consume();43  return { type: "CallExpression", callee: expression, arguments: args };44}45function binary(left, operator, right) { return { type: "BinaryExpression", left, operator, right }; }46 47console.log(parseExpression(0).type);Run

Step through line by line. The source has the same shape a production parser uses: one function reads expressions, one reads postfix operators such as calls and member access, and one reads primary tokens. The recording is from the lesson's real parser on a + b * c(d).

1const input = "a + b * c(d)";2const tokens = tokenize(input);3let cursor = 0;4 5function parseExpression(minBP = 0) {6  let left = parsePostfix();7  while (isBinary(peek()) && bindingPower(peek()) >= minBP) {8    const operator = consume();9    const right = parseExpression(nextMinBP(operator));10    left = binary(left, operator, right);11  }12  return left;13}14 15function parsePostfix() {16  let expression = parsePrimary();17  while (peek() === "(" || peek() === ".") {18    expression = parseCallOrMember(expression);19  }20  return expression;21}22 23function parsePrimary() {24  return tokenToNode(consume());25}26 27function tokenize(source) {28  return source.match(/[A-Za-z_$][A-Za-z0-9_$]*|[0-9]+|[*][*]|[()+*/.,-]/g).concat("<eof>");29}30function peek() { return tokens[cursor]; }31function consume() { return tokens[cursor++]; }32function isBinary(token) { return ["+", "-", "*", "/", "**"].includes(token); }33function bindingPower(token) {34  return token === "+" || token === "-" ? 10 : token === "*" || token === "/" ? 20 : token === "**" ? 30 : -1;35}36function nextMinBP(operator) { return operator === "**" ? bindingPower(operator) : bindingPower(operator) + 1; }37function tokenToNode(token) { return /^[A-Za-z_$]/.test(token) ? { type: "Identifier", name: token } : { type: "Literal", value: Number(token), raw: token }; }38function parseCallOrMember(expression) {39  if (peek() === ".") return { type: "MemberExpression", object: expression, property: tokenToNode((consume(), consume())) };40  consume();41  const args = peek() === ")" ? [] : [parseExpression(0)];42  consume();43  return { type: "CallExpression", callee: expression, arguments: args };44}45function binary(left, operator, right) { return { type: "BinaryExpression", left, operator, right }; }46 47console.log(parseExpression(0).type);Run

The second replay proves why config.get(user.name).theme is one chain before binary operators get a chance to group anything. Calls and member access are postfix operations: they attach directly to the expression already built.

A concrete syntax tree, or parse tree, mirrors grammar productions. It can include layers for parentheses, comma lists, and punctuation that matter while parsing but are noisy for most tools. An AST is more abstract: it keeps the meaning-bearing nodes.

ESTree is the community AST shape used by many JavaScript tools. Acorn, Espree, AST Explorer, ESLint rules, and many Babel-compatible workflows expose ESTree or ESTree-like nodes. V8 does build an AST internally, and the V8 scanner post says that AST is compiled to Ignition bytecode, but V8's internal node classes are not ESTree.

{
  "type": "Program",
  "body": [
    {
      "type": "ExpressionStatement",
      "expression": {
        "type": "BinaryExpression",
        "left": {
          "type": "Identifier",
          "name": "a"
        },
        "operator": "+",
        "right": {
          "type": "BinaryExpression",
          "left": {
            "type": "Identifier",
            "name": "b"
          },
          "operator": "*",
          "right": {
            "type": "CallExpression",
            "callee": {
              "type": "Identifier",
              "name": "c"
            },
            "arguments": [
              {
                "type": "Identifier",
                "name": "d"
              }
            ],
            "optional": false
          }
        }
      }
    }
  ],
  "sourceType": "script"
}

{
  "type": "Program",
  "body": [
    {
      "type": "ExpressionStatement",
      "expression": {
        "type": "BinaryExpression",
        "left": {
          "type": "Identifier",
          "name": "a"
        },
        "operator": "+",
        "right": {
          "type": "BinaryExpression",
          "left": {
            "type": "Identifier",
            "name": "b"
          },
          "operator": "*",
          "right": {
            "type": "CallExpression",
            "callee": {
              "type": "Identifier",
              "name": "c"
            },
            "arguments": [
              {
                "type": "Identifier",
                "name": "d"
              }
            ],
            "optional": false
          }
        }
      }
    }
  ],
  "sourceType": "script"
}

The operator lesson introduced precedence. A parser turns that table into structure. Multiplication nests inside addition, member access nests before multiplication, and exponentiation is right-associative.

1console.log(2 ** 3 ** 2);2 3try {4  new Function("return -2 ** 2;");5  console.log("parsed");6} catch (error) {7  console.log(error.name + ": " + error.message);8}

Line 1 prints 512, because JavaScript groups 2 ** 3 ** 2 as 2 ** (3 ** 2). Line 4 proves the special grammar restriction: a unary expression cannot appear directly on the left of **. Use -(2 ** 2) or (-2) ** 2 when you mean one of those shapes.

The ECMAScript specification defines early errors: errors detected and reported before evaluating the script, module, or eval code that contains them. This is stronger than “the engine could guess it will fail.” The spec says which static checks are early errors.

1const early = [];2try {3  eval('early.push("before"); let item = 1; let item = 2; early.push("after");');4} catch (error) {5  early.push(error.name);6}7console.log(early.join(" -> "));8 9const runtime = [];10try {11  eval('runtime.push("before"); missingFunction(); runtime.push("after");');12} catch (error) {13  runtime.push(error.name);14}15console.log(runtime.join(" -> "));

The first eval string has a duplicate let. It records only SyntaxError; the earlier early.push("before") never runs. The second eval string parses, pushes before, then throws when execution reaches missingFunction().

1// These are fragments for discussion, not code to run together.2let total = 1;3let total = 2;       // duplicate lexical declaration4 5return 1;            // outside a function61 = 2;               // invalid assignment target7 8"use strict";9010;                 // legacy octal literal in strict code

1const cases = [2  ["duplicate let", 'let count; let count;'],3  ["return outside function", 'return 1;'],4  ["invalid assignment target", '1 = 2;'],5  ["strict octal literal", '"use strict"; 010;'],6  ["strict duplicate parameters", '"use strict"; function f(a, a) {}'],7  ["await outside async", 'await 1;'],8];9 10for (const [label, source] of cases) {11  try {12    eval(source);13    console.log(label + ": ran");14  } catch (error) {15    console.log(label + ": " + error.name);16  }17}

Some JavaScript token sequences are ambiguous until a later token appears. The spec uses cover grammars: broad grammar shapes that temporarily cover more than one final meaning. The classic example is (a, b). Without =>, it is a parenthesized comma expression. With =>, it becomes an arrow parameter list.

1const samples = [2  ["parenthesized expression", "let a = 1, b = 2; (a, b);"],3  ["arrow parameters", "const fn = (a, b) => a + b;"],4  ["object assignment pattern", "let a; const b = { a: 2 }; ({ a } = b);"],5  ["object literal expression", "const value = ({ a: 1 });"],6];7 8for (const [label, source] of samples) {9  try {10    new Function(source);11    console.log(label + ": parses");12  } catch (error) {13    console.log(label + ": " + error.name);14  }15}

Object syntax has the same issue. In ({ a } = b), the object-looking shape is reinterpreted as a destructuring assignment pattern. In a normal object literal, { a = 1 } is not a valid property initializer. The spec's cover grammar rules explain why the parser can accept the broad shape first and apply early-error rules only after the final meaning is known.

The fastest way to learn ASTs is to change code and inspect nodes. AST Explorer is the standard web tool. Choose a parser, turn on options, and watch how the tree changes as source changes. In Acorn, Espree, and Babel-style parsers, the options you will reach for first are ecmaVersion, sourceType, locations, and ranges or tokens when a tool needs exact source positions.

const total = prices
  .map((price) => price * tax)
  .reduce((sum, value) => sum + value, 0);
console.log(total);

Try deleting a closing parenthesis, changing sourceType-sensitive code such as import, or clicking a CallExpression. The playground parses your source with Acorn and highlights character ranges; it never evaluates learner input.

ASTs matter because tools need syntax-aware facts. A linter should flag a real console.log() call without touching the string "console.log". A formatter must understand nesting before it chooses line breaks. A codemod should rename a function call without rewriting comments.

1import { parse } from "acorn";2import { ancestor, full } from "acorn-walk";3 4const source = `const total = prices.map((price) => price * tax);5console.log(total);`;6const ast = parse(source, { ecmaVersion: 2024, locations: true });7 8let identifiers = 0;9full(ast, (node) => {10  if (node.type === "Identifier") identifiers += 1;11});12 13const consoleLogs = [];14ancestor(ast, {15  CallExpression(node) {16    if (17      node.callee.type === "MemberExpression" &&18      node.callee.object.type === "Identifier" &&19      node.callee.object.name === "console" &&20      node.callee.property.type === "Identifier" &&21      node.callee.property.name === "log"22    ) {23      consoleLogs.push(node.loc.start.line);24    }25  }26});27 28console.log(identifiers);29console.log(consoleLogs.join(","));

This is the same reason the bundlers lesson used Acorn to find import declarations, and the transpilers lesson used Acorn positions to rewrite ??. Once source is a tree, tools can make precise decisions that plain text matching cannot.

Common misconceptions

• “Grammar and syntax are the same word.” Grammar is the rule system; syntax is the actual source shape being checked against those rules.
• “An AST contains every token.” A parse tree can mirror every grammar detail. An AST keeps a smaller, useful structure.
• “ESTree is what the engine runs.” ESTree is for tools. Engines use their own internal ASTs and compile to bytecode or machine code.
• “An early error happens after earlier lines run.” Early errors prevent evaluation of the entire script, module, or eval code that contains them.
• “Cover grammars mean the parser guesses at runtime.” Cover grammars are a parse-time technique. No JavaScript code executes while the parser reinterprets a shape.

Key takeaways

• The parser consumes tokens, checks grammar, reports syntax and early errors, and builds a tree.
• Recursive descent uses functions for grammar rules; Pratt parsing adds binding power for expressions.
• ESTree is a community AST format for tools, not V8's internal AST.
• Early errors prevent any evaluation of the script, module, or eval code that contains them.
• Cover grammars let the parser accept broad shapes and reinterpret them when later context arrives.
• ASTs power linting, formatting, transforms, codemods, bundling, and clearer syntax error messages.

Up next: Lazy parsing & preparsing, where engines decide how much of that work to do during startup and how much to defer.