Design a video call web app like Google Meet

Requirements: the questions that shape a video call

A video call app looks simple: a grid of faces and a row of buttons. Behind it are three hard systems at once. A real-time media pipeline sends your camera and microphone to other people within about 200 ms. A layout engine decides which of 100 people get a tile and how big.

The third system is a control channel that keeps everyone in agreement about who is in the call, who is muted, and who is sharing a screen. In 45 minutes you can't go deep on all of it, so the first five minutes decide where to spend the time.

Ask about size first (how many people in one call), then about the people (which devices and networks), then about features (screen share, chat, recording), and last about hard constraints (encryption, accessibility, enterprise firewalls). The table lists the answers this guide assumes and what each one changes.

Functional requirements

• Open a meeting link and land on a pre-join screen: see your own camera preview, pick a camera, microphone, and speaker, see a live microphone level, and set your name. Join with camera or microphone already off if you want.
• Join the call and see other people in a grid of tiles that adapts to the window size and the number of people. Hear everyone who speaks.
• See who is talking: the speaking tile gets a highlight, and in a speaker layout the active speaker gets the big tile.
• Pin a person so their tile stays big for you. A host can spotlight a person so they are big for everyone.
• Mute and unmute the microphone, turn the camera on and off, and switch devices during the call.
• Share a screen, a window, or a tab. Viewers see the shared content large, with the presenter's face small next to it.
• Send chat messages, emoji reactions, and raise a hand. See a list of participants with their mute and hand state.
• See captions when they are on, and see clearly when the call is being recorded.
• Leave the call, and rejoin by reloading or by clicking Rejoin.

Non-functional requirements

• Latency: mouth to ear under 200 ms on a good network, and under 400 ms in the worst case we accept. Above that, people start talking over each other.
• Join time: from clicking Join to hearing other people in under 3 seconds on a normal network.
• Audio first: when the network gets worse, video quality drops before audio does. Audio must keep working down to about 50 kbps.
• Bandwidth: a receiver with 2 Mbps of download must still have a usable call with 25 visible tiles.
• CPU: a mid-range laptop must hold a 25-tile call without the fan running at full speed or the video freezing. Phones must last a 1-hour call.
• Resilience: a Wi-Fi switch or a 5-second network drop must not end the call. The client reconnects on its own.
• Accessibility: mute and camera can be toggled from the keyboard, joins and leaves are announced to screen readers without flooding them, and captions are available.
• Privacy: the camera and microphone are on only when the person wants them on, and the browser's recording indicator matches what the app shows.

Out of scope, and why

• Webinars and live streams to thousands: that is a broadcast problem with a few speakers and a delay of seconds. It uses different servers and players. The live streaming guide covers it.
• Native apps and phone dial-in: they talk to the same servers but are separate clients.
• The media server's internals: we treat it as a black box with a clear API: it accepts our streams, forwards the ones we ask for, and tells us about bandwidth. We discuss what the client needs from it.
• Scheduling and calendar: creating a meeting and inviting people is a normal web form backed by a REST API.

What makes a video call hard

Most web apps move small JSON messages and can wait a second for them. A video call moves a steady river of media, and a second is far too long. Five properties of the problem drive every later decision. Name them early in the interview, because each one rules out a simple design.

1. Late is the same as lost

A chat message that arrives 2 seconds late is still useful. An audio packet that arrives 2 seconds late is useless: the conversation has moved on. So media doesn't use TCP-style "retry until it arrives". It travels over UDP with RTP (the Real-time Transport Protocol), and the receiver keeps a small jitter buffer, typically tens of milliseconds, to smooth out packets that arrive unevenly.

Lost audio is hidden by guessing (packet loss concealment). Lost video is either re-sent quickly once (a NACK, "please resend packet 4012") or skipped until the next full frame (a key frame, which can be decoded on its own). The client's job is to keep this pipeline fed and tuned, not to make every packet arrive.

2. Every person is a sender and a receiver

In a YouTube player, one server sends and many people watch. In a call, each of 25 people sends a camera stream and receives up to 24. If every browser sent its video straight to every other browser, a 720p stream at 1.5 Mbps would cost each person 36 Mbps of upload in a 25-person call. Home upload is often 10 to 20 Mbps, and a phone on 4G may have 5.

The shape of the network (who sends to whom) is the first architecture decision, and the webrtc lesson has the bandwidth math for mesh, MCU, and SFU in detail. Here we use its result: beyond about 4 people, we need a media server that forwards streams.

3. Every receiver is different

In the same call, one person sits on office fiber with a 27-inch screen showing the speaker full size, while another is on a phone on a train with 800 kbps showing 4 small tiles. Sending both the same stream is wrong for both: too heavy for the phone, too blurry for the big screen.

The sender must produce video that can be cut down per receiver, and each receiver must ask for what it can show and afford. Simulcast (sending several sizes of the same video at once) and layer selection solve this, and they are the heart of the frontend's media job.

4. Hardware you don't control

The app depends on cameras, microphones, speakers, headsets that get plugged in mid-call, Bluetooth earbuds that switch profiles, and permission prompts that people dismiss. It also depends on the CPU: encoding one 720p stream and decoding 25 small ones is real work, and a laptop on battery may slow down its CPU to save power. The browser tells us when this happens (through events and statistics), and the app must react, for example by sending smaller video when the encoder is CPU-limited.

5. Shared state that must agree

Besides media, the call has state everyone must see the same way: who is in the call, who is muted, who has a raised hand, who is sharing, who the host spotlighted, whether recording is on. This state changes often and comes from many people at once. It travels over a separate signaling channel (usually a WebSocket) with the server as the single source of truth. The media and the signaling can fail separately, and the client must handle each.

The pre-join screen: permissions, devices, and a preview

Most failed calls fail before they start: the camera is blocked, the wrong microphone is picked, or the headset isn't selected as the speaker. The pre-join screen exists to catch these while the person is alone, not in front of 20 colleagues. It is the first thing people see, so it must also be fast: a preview within about a second of the page loading.

The naive approach: ask for everything on page load

The simplest code calls navigator.mediaDevices.getUserMedia({ video: true, audio: true }) as soon as the page loads. The browser shows its permission prompt before the person has seen anything of the app. Many people dismiss prompts that appear with no context. And when it fails, one call that asked for both camera and microphone fails as a whole: if another app is using the camera, the person also loses their microphone and ends up with a black tile and silence, and the app has no idea which part went wrong.

Reading the errors

getUserMedia rejects with an error (almost always a DOMException) whose name tells you what went wrong. Each name needs its own message, because the fix is different. Ask for audio and video separately when the combined request fails, so one broken device doesn't take the other down.

async function openDevices(prefs: DevicePrefs) {
  const audio = { deviceId: prefs.micId ? { ideal: prefs.micId } : undefined,
    echoCancellation: true, noiseSuppression: true, autoGainControl: true };
  const video = { deviceId: prefs.camId ? { ideal: prefs.camId } : undefined,
    width: { ideal: 1280 }, height: { ideal: 720 }, frameRate: { ideal: 30 } };
  try {
    return { stream: await navigator.mediaDevices.getUserMedia({ audio, video }), errors: {} };
  } catch (both) {
    const [a, v] = await Promise.allSettled([
      navigator.mediaDevices.getUserMedia({ audio }),
      navigator.mediaDevices.getUserMedia({ video }),
    ]);
    const stream = new MediaStream();
    if (a.status === "fulfilled") a.value.getTracks().forEach((t) => stream.addTrack(t));
    if (v.status === "fulfilled") v.value.getTracks().forEach((t) => stream.addTrack(t));
    return { stream, errors: {
      mic: a.status === "rejected" ? (a.reason as DOMException).name : undefined,
      cam: v.status === "rejected" ? (v.reason as DOMException).name : undefined,
    } };
  }
}

Notice ideal around the saved deviceId, not exact. With ideal, the browser uses the saved camera if it is still plugged in and the default camera otherwise. With exact, an unplugged camera gives an OverconstrainedError and a blank preview.

Device pickers that stay true

navigator.mediaDevices.enumerateDevices() lists the devices as MediaDeviceInfo objects with a deviceId, a kind (audioinput, videoinput, or audiooutput), a label like "Logitech C920", and a groupId that is shared by the parts of one physical device, such as a headset's microphone and speaker. Two details catch people out. First, before permission is granted, browsers hide the labels and most of the list, so enumerate again after getUserMedia succeeds. Second, devices come and go: listen to the devicechange event on navigator.mediaDevices and enumerate again each time.

• When a new headset appears, offer it ("Use AirPods?") rather than switching silently. Switching the microphone mid-sentence surprises people.
• When the device in use disappears, its track ends. Switch to the default device at once and show a short notice, because silence is worse than a different microphone.
• Remember the chosen deviceIds in localStorage. A device id usually stays the same for a site until the person clears its data, so the same camera is picked next time.
• For speakers, call setSinkId(deviceId) on the audio elements. Chrome, Edge, Firefox 116 and later, and Safari 18.4 and later support it; feature-detect it and hide the speaker picker where it is missing, so the system default output is used.

The preview, the level meter, and the network check

The self preview is a muted video element with playsinline (so iOS doesn't open it full screen), mirrored with a CSS transform because people expect to see themselves like in a mirror. The mirroring is display only: the stream sent to others is not flipped, so text on a shirt reads correctly for them.

The microphone level meter is the most useful thing on the screen: it proves the right microphone hears you. Connect the audio track to a Web Audio AnalyserNode, read the samples each animation frame, and draw the loudness. Stop it when the page is hidden. A speaker test button plays a short sound through the chosen output.

A light network check runs in parallel: open the signaling WebSocket early, measure a few round trips, and try to gather ICE candidates (the network addresses a connection could use, covered in the next chapter) against the TURN server to learn whether UDP gets out. If only TCP works, or round trips are above about 300 ms, show a gentle warning ("Your connection may cause choppy video"). Never block joining on it.

Joining the call: signaling, the SFU, and the relay fallback

When the person clicks Join, two connections start. A signaling connection carries control messages: who is in the room, the technical offer and answer that set up media, mute changes, chat. A media connection carries audio and video. WebRTC defines the media part but leaves signaling to the app, so we design it ourselves.

Who sends media to whom

The naive design is a mesh: one RTCPeerConnection from each browser to each other browser. It needs no media server and works well for a 1-to-1 call. At 6 people each browser encodes and uploads 5 copies of its camera and decodes 5 streams, and at 25 people it collapses. The webrtc lesson works through the bandwidth math for the three shapes; here is the short version for our numbers.

Signaling over a WebSocket

The signaling channel is a WebSocket to our room server (see the websockets lesson for heartbeats and reconnects). The first message is join with a short-lived meeting token. The server replies with a room snapshot: the participant list with each person's mute, camera, and hand state, the current presenter, the room settings, and a version number.

The snapshot also carries the media setup: the SFU's address and a list of ICE servers with temporary TURN credentials. After the snapshot, the server sends small events (participant-joined, mute-changed) each with the next version number, so the client can spot a gap and ask for a fresh snapshot with a resync command.

Then the media setup. The client creates an RTCPeerConnection with the ICE servers, adds its microphone and camera tracks as transceivers (a transceiver is one send and receive slot for a single track), and creates an offer: an SDP (Session Description Protocol) text that lists the codecs it supports, the streams it will send, and its encryption fingerprint. It sends the offer over the WebSocket, and the SFU answers with its own SDP.

Both sides then gather ICE candidates (addresses where they can be reached) and try pairs until one works. Then DTLS sets up the keys, and SRTP (encrypted RTP) media starts to flow. All audio and video share one transport (BUNDLE), so there is one network path to keep alive, not one per stream.

When UDP is blocked: the TURN ladder

An SFU usually has a public address, so most clients reach it directly over UDP. Some networks block outgoing UDP, and some corporate proxies allow only HTTPS on port 443. For them, a TURN server relays media. The client tries the options in order and uses the first that works:

1. UDP straight to the SFU: the fastest path, used by most people.
2. TURN over UDP: for networks that allow UDP only to some addresses.
3. TURN over TCP: for networks that block UDP entirely. It brings back the head-of-line blocking from the "What makes a video call hard" chapter, so quality drops under loss.
4. TURN over TLS on port 443: looks like HTTPS traffic to the firewall. The last resort, and the one that saves enterprise customers.

ICE gathers all these candidates at once and checks them in parallel, so the fallback costs a second or two at most, not a sequence of timeouts. Log which candidate type won (host for a local address, srflx for the public address a STUN server saw, or relay for TURN, plus the relay protocol) for every call. If 8% of calls in one company run over TURN TLS, that explains their complaints about quality better than any guess.

The join time budget

Sending and receiving video: simulcast, layers, and bandwidth

The SFU forwards packets without decoding them, so it can't shrink a video for a weak receiver. If the sender uploads one 720p stream at 1.5 Mbps, every receiver gets 1.5 Mbps for that person, whether they show them full screen or in a thumbnail the size of a stamp. A phone with 800 kbps showing 4 people can't even receive one of them. If the sender uploads only 360p instead, the person on the big screen watching the speaker sees mush.

Setting up the layers

Simulcast is set when the camera track is added. Each entry in sendEncodings is one layer, with an id (rid), how much to scale it down, and a bitrate cap. The bitrates below are example values, typical for VP8 camera video; real products tune them per codec and content.

const camera = pc.addTransceiver(cameraTrack, {
  direction: "sendonly",
  streams: [localStream],
  sendEncodings: [
    { rid: "q", scaleResolutionDownBy: 4, maxBitrate: 150_000 },   // 320x180
    { rid: "h", scaleResolutionDownBy: 2, maxBitrate: 500_000 },   // 640x360
    { rid: "f", scaleResolutionDownBy: 1, maxBitrate: 1_500_000 }, // 1280x720
  ],
});
cameraTrack.contentHint = "motion";

If nobody in the call is watching your full layer (everyone has you in a small tile), uploading it wastes about 1.5 Mbps and real CPU. A good SFU tells the sender which layers have subscribers, and the client turns the others off with setParameters, setting active: false on those encodings. In a 50-person call where you never speak, your laptop may encode only 180p.

Choosing a layer per tile

The receiver knows something the SFU doesn't: how big each tile is on screen. So the client tells the SFU, for every person it shows, the largest size it needs.

A simple rule: take the tile's height in CSS pixels, multiply by devicePixelRatio capped at about 1.5 (sharper than that is rarely visible in a video), and pick the smallest layer at least that tall. A tile 170 px tall needs 180p. A tile 330 px tall on a laptop with a pixel ratio of 2 needs about 495 physical pixels (330 times the capped 1.5), so it gets 720p.

Recompute when a ResizeObserver reports a tile size change, and send updates at most a few times a second, because every layer switch costs a key frame.

Then bandwidth decides what is actually possible. WebRTC estimates the available bandwidth all the time from packet timing feedback. In Chrome and Safari, which share Google's WebRTC library, the algorithm is Google Congestion Control (GCC), fed by per-packet feedback from the receiver; the details differ between browsers and versions. On the sending side the browser lowers bitrates and drops the top layer when the uplink shrinks.

On the receiving side the SFU estimates each receiver's downlink and fits the requested layers into it. When the budget is too small, it lowers the least important tiles first: thumbnails before the pinned person, the pinned person before the screen share, and video before any audio. When even the smallest layer doesn't fit, it pauses that person's video and the tile shows their avatar. Try it below.

The grid: layout, pages, visible-only video, and the active speaker

The grid is what people look at for the whole call. It has to answer four questions every moment: how big is each tile, which people get a tile at all, whose video do we actually download, and who is talking. The naive answer is a CSS grid with one video element per participant. At 100 people that means 100 decoders, 100 downloads, tiles the size of a fingernail, and a grid that reshuffles every time someone joins.

Sizing N tiles in a window

Camera video is 16:9. Given a stage of width W and height H and N people, we want the biggest tiles that fit while keeping the 16:9 shape. CSS alone can't do this, because the best number of columns depends on both N and the window's shape: 6 people fit best as 3 by 2 in a wide window but 2 by 3 in a tall one. The algorithm is short: try every column count, compute the tile size each one allows, and keep the biggest.

function bestGrid(n: number, w: number, h: number, gap = 8, ratio = 16 / 9) {
  let best = { cols: 1, rows: n, tileW: 0, tileH: 0 };
  for (let cols = 1; cols <= n; cols++) {
    const rows = Math.ceil(n / cols);
    const maxW = (w - gap * (cols - 1)) / cols;
    const maxH = (h - gap * (rows - 1)) / rows;
    const tileW = Math.min(maxW, maxH * ratio);
    if (tileW > best.tileW) best = { cols, rows, tileW, tileH: tileW / ratio };
  }
  return best;
}

Worked example, ignoring the gap: 7 people on a 1280 by 720 stage. With 3 columns there are 3 rows; each tile can be 426 wide (1280 / 3) and 240 tall (720 / 3), and 240 tall at 16:9 is 426 wide, so tiles are 426 by 240. With 4 columns there are 2 rows; tiles can be 320 wide and 360 tall, but 320 wide at 16:9 is only 180 tall, so tiles are 320 by 180. Three columns win.

The last row has one tile, which we center. Run this on every resize of the stage (a ResizeObserver), which is cheap: at most a few dozen loop steps.

Tiles crop with object-fit: cover so faces fill the box, which matters for phones sending portrait video. Screen shares use object-fit: contain instead, because cropping a slide cuts off text.

Who gets a tile: caps, order, and pages

The lab shows why a cap is needed: past about 49 tiles on a laptop screen, faces become too small to read, and the receive budget breaks. Big products cap the grid at a number the person can choose: Google Meet's tiled layout, for example, shows 16 tiles by default and up to 49 on desktop. Phones show far fewer (4 to 9 is common). Everyone else goes to the next page, or the person scrolls the participant list.

Order matters more than the cap. A grid sorted by join order hides the person speaking on page 3. A grid re-sorted every time someone speaks jumps around constantly, and people lose track of faces. A stable order with a few rules works:

1. Pinned people and the screen share come first, always.
2. Then everyone already on the first page keeps their position.
3. When someone who is not on the first page starts speaking, they replace the visible person who spoke least recently, in that person's slot. Nobody else moves.
4. People with camera on are preferred over camera off when filling empty slots.

Download only the video people can see

A tile on page 2, scrolled out of view, or hidden behind a full-screen share needs no video at all. The client tracks visibility per tile (an IntersectionObserver on tiles in a scrolling list, the page number for paged grids) and sends the SFU an updated subscription: video for the visible people at the size they need, nothing for the rest. Audio is not affected: you still hear people on page 3.

When the whole tab is hidden (document.visibilityState is hidden), unsubscribe from all camera video, keep audio, and resubscribe when the tab returns. This one rule can cut a hidden tab's download from several Mbps to under 200 kbps.

Detecting the active speaker

Every audio packet can carry a tiny header extension with its loudness, the audio level (RFC 6464), so the SFU can see who is loud without decoding audio. The SFU smooths these levels and sends a dominant-speaker event. For the small "speaking" ring on each tile, the client reads levels itself: receiver.getSynchronizationSources() returns the level of the latest packets cheaply, and the audioLevel field of each audio inbound-rtp entry in getStats gives the same signal.

Two speeds are needed. The speaking ring should react within about 200 ms, or it looks broken, while the big speaker tile must react slowly, or a cough, a laugh, or a door slam steals the screen. So the big tile uses hysteresis (a change must last a while before the display follows it, so short blips don't flip it): a new person must be the loudest for about 1 second before they take over, and after a switch, no other switch for about 2 seconds.

People who are muted are never the active speaker, whatever their level.

Pin and spotlight

Pin is a personal choice: "keep Maria big on my screen". It lives only in this client's view state and changes this client's layout and subscriptions; nobody else knows. Spotlight is a host's choice ("everyone sees Maria big"), so it is room state: sent to the server, broadcast to everyone, and shown in each client's layout unless they pinned someone else. Mixing these up is a common interview mistake: pin is local UI state, spotlight is shared state.

Screen sharing: capture, encoding for text, and the presenter view

Screen sharing looks like "one more video", but it differs from a camera in every way that matters. A camera shows a face that moves all the time, where a bit of blur is fine. A screen shows small text that barely moves, where blur makes it unreadable, and where a frame rate of 5 is fine until someone scrolls. It also has its own permissions flow, its own way to stop, and changes the layout for everyone.

Capturing with getDisplayMedia

navigator.mediaDevices.getDisplayMedia() opens the browser's own picker, where the person chooses a whole screen, a window, or a browser tab. The app can't choose for them or skip the picker, and it must call this from a click handler: browsers require a recent user action (transient activation) and otherwise reject with InvalidStateError. If the person cancels the picker, the promise rejects with NotAllowedError, which is not an error to show: they changed their mind.

async function startShare() {
  const stream = await navigator.mediaDevices.getDisplayMedia({
    video: { frameRate: { ideal: 15, max: 30 } },
    audio: true,                    // tab audio, where the browser supports it
    selfBrowserSurface: "exclude",  // don't offer the call's own tab
    surfaceSwitching: "include",    // allow "share this tab instead"
  } as DisplayMediaStreamOptions);
  const [video] = stream.getVideoTracks();
  video.contentHint = "detail";     // keep text sharp, drop frames first
  video.addEventListener("ended", () => stopShare("browser-ui"));
  return stream;
}

• Audio: sharing a tab with its sound (a video in a tab) works in desktop Chromium-based browsers, which can also share system audio with a whole screen on Windows and ChromeOS. Firefox and Safari return video only. Check stream.getAudioTracks().length and tell the presenter if the sound isn't shared, rather than letting them discover it from confused viewers.
• Stopping: the browser shows its own "Stop sharing" bar. When the person uses it, the track fires ended. The app must listen for this and clean up exactly as if the app's own Stop button was clicked.
• Content hint: contentHint = "detail" (or "text" for slides and documents) tells the encoder to keep the resolution and lower the frame rate when bandwidth is short. A camera uses "motion", which does the opposite. Chromium and Safari support the hint; Firefox ignores it. Without it, a browser may treat the share like camera video, and a congested share becomes blurry text at a smooth frame rate: the worst of both.
• Focus: when a person shares another tab, the browser may switch to it. Chromium offers a CaptureController with setFocusBehavior to keep focus on the call; treat it as an optional improvement where it exists.

Encoding a screen

A shared 1080p screen of a code editor needs about 1 to 2.5 Mbps to look sharp, but only at 5 to 15 frames per second, and almost nothing when the screen is still, because the encoder only sends what changed. Send it on its own transceiver, separate from the camera, so the SFU and receivers can treat it as a different stream with a higher priority. Many apps send the share as one high-resolution layer plus one small layer for phones, rather than three layers.

Add the screen transceiver when joining, with no track, and use sender.replaceTrack(screenTrack) when sharing starts. replaceTrack swaps the source without a new offer and answer, so the share starts faster and the call doesn't renegotiate in the middle.

One presenter, and the presenter view

Who is presenting is room state, owned by the server. The client asks with start-share; the server grants it if nobody else is sharing (or if the room allows taking over) and broadcasts share-started to everyone. Only then does the client attach the track. This avoids two people both believing they are the presenter after clicking at the same moment.

Viewers switch to a presentation layout: the share on the main stage with object-fit: contain, and the faces in a filmstrip on the side or bottom, at the smallest layer. The presenter sees something different: not their own share at full size (sharing a screen that shows the share creates an endless tunnel of copies), but a banner like "You are presenting your screen" with a Stop button and a small preview.

Architecture: who owns what on the client

We now know the pieces: a pre-join flow, a signaling socket, a peer connection to the SFU, a grid that decides subscriptions, and screen share. The architecture question is where each piece lives and who is allowed to change what.

One fact shapes everything: media objects (MediaStreamTrack, RTCPeerConnection, MediaStream) are live browser objects with their own lifecycles. They can't be copied, serialized, or compared like JSON. Putting them into a React state store invites leaks (a track nobody stops keeps the camera light on) and needless re-renders.

So the client splits into three layers. A media engine outside React owns every live object. Stores hold plain, serializable state: ids, flags, versions, sizes. Components read the stores and, for the one thing that needs a live object (a video element's srcObject), ask the engine for the track by id.

Why these boundaries? The room store has exactly one writer, the signaling client, so the UI can never drift from the server: a mute icon changes because the server said so.

The view store is the opposite: purely local, never sent anywhere except through the subscription manager. The subscription manager sits between them because subscriptions depend on both: who exists and is sharing (room) and what is visible at what size (view). And the media engine is the only place that touches WebRTC, so it can be tested with a fake, and swapped for a vendor SDK if the company buys one.

How data moves through it

Two rivers run side by side. The control river is small JSON over the WebSocket: room events in, user commands out. The media river is RTP packets over the peer connection: our tracks out, remote tracks in. They meet in two places: the subscription manager (view state becomes subscribe messages that change which media arrives) and the tiles (a participant from the room store is drawn with a track from the media engine).

Follow one action through both rivers: the person clicks Mute. The device manager sets track.enabled = false on the microphone at once: from that instant the browser sends silence, so no word leaks while the network catches up. The local media store flips to muted, and the button updates in the same frame.

Then the signaling client sends set-mute. The server updates the room and broadcasts mute-changed with a new version. Every client, including ours, applies it to the room store, and the muted icon appears on our tile everywhere. The figure in the interface chapter shows this flow, plus the failure path when a host has locked microphones.

Data model: plain state on one side, live media on the other

The data model follows the split from the architecture: serializable entities in stores, and live media objects referenced only by id. Most of the state is short-lived. When the call ends, nearly all of it is thrown away; only device choices and a few preferences survive in localStorage.

type Participant = {
  id: string; name: string; avatarUrl?: string;
  role: "host" | "cohost" | "guest";
  audioMuted: boolean; videoOff: boolean; handRaisedAt?: number;
  audioTrackId?: string; videoTrackId?: string; screenTrackId?: string;
  joinedAt: number; lastSpokeAt?: number;
};
type Room = {
  id: string; version: number; participants: Record<string, Participant>;
  presenterId?: string; spotlightId?: string;
  recording: boolean; captionsOn: boolean; e2ee: boolean;
};
type ViewState = {
  layout: "grid" | "speaker" | "presentation"; pinnedId?: string;
  page: number; tileSizes: Record<string, { w: number; h: number }>;
  panel?: "chat" | "people";
};
type Subscription = { participantId: string; video: boolean; maxHeight: 0 | 180 | 360 | 720; screen: boolean };
type LocalMedia = {
  micId?: string; camId?: string; speakerId?: string;
  micMuted: boolean; camOff: boolean; blur: boolean;
  errors: { mic?: string; cam?: string };
};
type ChatMessage = { id: string; clientId: string; fromId: string; text: string; sentAt: number; status: "sending" | "sent" | "failed" };

Two details deserve a sentence each in an interview. First, version on the room: every server event carries the next number, so a client that receives version 42 after 40 knows it missed one and asks for a snapshot instead of drifting. Second, lastSpokeAt on the participant drives the stable grid order from the grid chapter: when a speaker from page 2 needs a slot, the visible person with the oldest lastSpokeAt gives up theirs.

Interface: the signaling protocol and the component contracts

The client talks to three servers: a small REST API for the meeting itself (look up a meeting, get a join token), the room server over a WebSocket for signaling and room state, and the SFU for media. The SFU's media setup (offers, answers, candidates) is carried inside the WebSocket too, so the client has one control channel to secure, monitor, and reconnect.

REST: before the call

GET /api/meetings/abc-defg-hij
200 { "id": "abc-defg-hij", "title": "Design review", "lobby": false,
      "e2ee": false, "maxParticipants": 100 }

POST /api/meetings/abc-defg-hij/join-tokens
{ "displayName": "Maria" }
200 { "token": "eyJhbGciOi...", "expiresIn": 120,
      "signalingUrl": "wss://rtc-eu1.example.com/rooms/abc-defg-hij" }

WebSocket: one envelope, two kinds of messages

Every message has a type. Commands from the client carry a reqId so the reply can be matched to the request, like a tiny RPC. Events from the server carry the room version. The client never assumes a command worked until the reply or the matching event arrives.

type ClientCommand = { reqId: string } & (
  | { type: "join"; token: string; muted: boolean; camOff: boolean; caps: { svc: boolean; codecs: string[] } }
  | { type: "resume"; sessionId: string } | { type: "resync" }
  | { type: "offer"; sdp: string } | { type: "ice"; candidate: RTCIceCandidateInit }
  | { type: "subscribe"; subs: { participantId: string; video: boolean; maxHeight: number; screen: boolean }[] }
  | { type: "set-mute"; audio?: boolean; video?: boolean }
  | { type: "start-share" } | { type: "stop-share" }
  | { type: "raise-hand"; up: boolean } | { type: "react"; emoji: string }
  | { type: "chat"; clientId: string; text: string }
  | { type: "host-action"; action: "mute" | "remove" | "spotlight"; targetId: string });

type ServerEvent =
  | { type: "reply"; reqId: string; ok: true; data?: unknown }
  | { type: "reply"; reqId: string; ok: false; error: { code: string; message: string; retry: boolean } }
  | { type: "snapshot"; version: number; room: Room; iceServers: RTCIceServer[] }
  | { type: "answer"; sdp: string } | { type: "ice"; candidate: RTCIceCandidateInit }
  | { type: "participant-joined" | "participant-left" | "participant-updated"; version: number; participant: Participant }
  | { type: "mute-changed"; version: number; participantId: string; audio: boolean; video: boolean }
  | { type: "share-started" | "share-stopped"; version: number; participantId: string }
  | { type: "dominant-speaker"; participantId: string }
  | { type: "layers-wanted"; camera: ("q" | "h" | "f")[] }
  | { type: "chat"; message: ChatMessage } | { type: "reaction"; fromId: string; emoji: string }
  | { type: "captions"; speakerId: string; text: string; final: boolean };

• Errors come back as a reply with a code like not-host, share-in-progress, room-full, or token-expired, and a retry flag that tells the client whether trying again can help.
• subscribe sends the full desired set, not a diff, so a lost or reordered message can't leave the SFU with a wrong picture. The client sends it only when the set changed, at most a few times per second.
• dominant-speaker and reaction carry no version. They are moments, not state: if one is lost, nothing needs repairing.
• layers-wanted tells the sender which of its simulcast layers anyone is watching, so it can turn the others off.

Component contracts

type VideoTileProps = {
  participantId: string;
  trackId?: string;             // looked up in the media engine
  kind: "camera" | "screen";
  fit: "cover" | "contain";
  name: string;
  audioMuted: boolean;
  speaking: boolean;            // fast ring, from audio levels
  handRaised: boolean;
  quality?: "good" | "fair" | "poor";
  pinned: boolean;
  onPinToggle(id: string): void;
  onResize(id: string, size: { w: number; h: number }): void;
};

In-call features: chat, reactions, raised hands, captions, and recording

Around the video sit features that look small but each has a trap. They differ in one key way: some are state that must be correct for everyone (a raised hand, recording on), some are messages that must arrive once (chat), and some are moments that are worthless a few seconds later (reactions). Treat them differently and each becomes simple.

Chat

Reactions

Reactions are moments. At a 100-person all-hands, a joke can produce 300 reactions in 2 seconds. Each one rendered as a floating emoji animation would cost the main thread hundreds of DOM nodes and animations, right when the video needs the CPU.

So the server limits each person to a few reactions per second and drops the rest; the client never retries a failed reaction; and the renderer caps the number of floating emojis on screen (say 20), merging the rest into a counter like "thumbs up x 37". Animations use transform and opacity only, so they stay on the compositor, and when prefers-reduced-motion is set they become a short static badge on the sender's tile instead.

Raised hands

A raised hand is state: it must survive until lowered and be the same for everyone. Store it as handRaisedAt (a server timestamp) on the participant, so every client can show the queue in the same order: first raised, first asked. The host can lower anyone's hand, and the hand lowers on its own when that person starts speaking for a while, if the product wants that rule. Late joiners see raised hands in the snapshot.

Captions and transcripts

Captions come from speech recognition running on the server side, fed with the audio the SFU already receives. Browsers do have a Web Speech API, but its support, quality, and privacy (some engines send audio to a cloud service) differ widely, and it listens to the local microphone, so each client could caption only its own speech and would have to send the text to everyone. Server captions hear every speaker's stream and know who is talking, so they can say "Maria: let's ship it".

The client receives captions events of two kinds. Interim results (final: false) change as the recognizer hears more, so they replace the current line in place. Final results are appended to a short rolling caption area (2 or 3 lines on screen) and to a transcript in the side panel. Words keep arriving every few hundred milliseconds, so the caption area must not be an aria-live region; the accessibility chapter explains why.

Recording

Record on the server, not in a browser. A client recording with MediaRecorder captures one person's view, uses that laptop's CPU for the whole call, must upload gigabytes afterwards, and is lost if the tab crashes.

A server recorder (often a hidden participant that subscribes to everything, or a compositor next to the SFU) records every stream and has no tab to close. The client's job is the trust part: a recording indicator that is always visible, present in the snapshot so late joiners see it, announced when it starts, and a notice before joining a call that is already being recorded.

Network resilience: reconnecting, degrading, and telling people

Networks fail in calls all the time: Wi-Fi drops for 3 seconds in the kitchen, a laptop moves from Wi-Fi to a phone hotspot, a train enters a tunnel, a VPN reconnects. The naive client treats any drop as the end: it shows "You left the meeting" or reloads and rejoins.

A full rejoin takes 3 or more seconds, restarts the camera, re-runs the pre-join steps, and makes the person reappear in everyone's grid as a new tile. Most of these drops can be fixed within a second or two, with nothing worse than a short freeze.

Two connections, two kinds of failure

The signaling WebSocket and the media path fail separately. A WebSocket drop with healthy media is common (a proxy times out an idle socket): media keeps flowing, so people still hear each other, and the client quietly reconnects the socket with backoff, sends resume with its session id, and applies a fresh snapshot. A media drop with a healthy socket is the case that hurts, because people stop hearing each other. The socket stays useful during a media repair: it carries the new offer and answer.

Don't wait for ICE to notice a network change. When the browser fires online, or the Network Information API (Chromium only) reports a connection change, restart ICE right away. The old path is almost certainly dead, and waiting for timeouts costs seconds. Not every switch fires these events, so the ladder above still catches the rest.

Degrade video before audio

People forgive a frozen face. They don't forgive missing words. So when bandwidth or CPU runs short, the client and SFU give up quality in a fixed order, cheapest loss first, and climb back up in reverse order when things improve, slowly, to avoid flapping.

Help the network help audio: where the browser supports it, set a high priority (and networkPriority) on the audio sender's encoding so its packets are favored. Opus in-band forward error correction (FEC) hides some packet loss without retransmission, and discontinuous transmission (DTX) sends almost nothing during silence, which saves bandwidth when 99 people are quiet.

Showing connection quality

Every 2 seconds the media engine reads getStats and turns round-trip time, packet loss, jitter, and any active degradation into one level: good, fair, or poor. Show our own level near the controls, and a small icon on other people's tiles so a listener knows the freeze is Ana's network, not theirs.

Use hysteresis here too: drop a level after about 5 seconds of bad numbers, and rise only after about 10 good ones, or the icon flickers. When the client degrades something, say so in plain words: "Your connection is unstable. Incoming video is paused to keep audio clear." A silent black tile looks like a bug; an explained one looks like care.

Audio processing, background blur, and the CPU budget

A call keeps the CPU busy for an hour: encoding our camera in three sizes, decoding up to dozens of streams, mixing audio, rendering tiles, and maybe blurring our background 30 times a second. On a desktop that's fine. On a 4-year-old laptop on battery, the browser slows the encoder, frames drop, and the fan spins. This chapter is about spending that budget well.

Audio processing: mostly the browser's job

Three audio constraints do most of the work, and browsers usually turn them on for getUserMedia audio: echoCancellation (removes the other people's voices coming out of your speakers and back into your microphone), noiseSuppression (reduces steady background noise like a fan), and autoGainControl (evens out loud and quiet speakers). Ask for them explicitly anyway, so the intent is in the code.

• Don't turn echo cancellation off for normal calls. Without it, one person on laptop speakers sends everyone's voices back into the call as an echo.
• Offer a "music mode" for teachers and musicians: turn off noise suppression and gain control (they treat a violin as noise), raise the Opus bitrate, and warn that headphones are needed.
• Stronger noise removal (a dog barking, keyboard clicks) needs a machine learning model. It runs as an AudioWorklet, often compiled to WebAssembly, between the microphone and the sender. It costs CPU for the whole call, so make it a setting, not a default on weak devices.

Background blur

Blur needs, for every frame, a mask saying which pixels are the person. A small segmentation model produces that mask; then the background pixels are blurred and the person drawn sharp on top. At 30 frames per second that is 30 model runs and 30 blurs per second, so where this runs matters a lot.

Three tricks keep blur affordable. Run the model on a small copy of the frame (for example 256 by 144) and scale the mask up; the edges barely change. Run it at 15 frames per second and reuse the last mask in between; people move slowly. And blur on the GPU with a shader, not pixel by pixel in JavaScript.

If the encoder still reports it is CPU-limited, turn blur down before turning video quality down, and tell the person.

The CPU and battery budget

The browser tells us when we are over budget. The outbound video stats include qualityLimitationReason, which says cpu when the encoder can't keep up and bandwidth when the network can't. Watch it every few seconds. The runtime-performance lesson covers measuring the main thread itself with long task timings.

Security and end-to-end encryption

WebRTC media is always encrypted on the wire: DTLS sets up keys and SRTP encrypts every packet. But in an SFU call, the encryption runs from your browser to the SFU, and again from the SFU to each receiver. The SFU holds the keys for both hops, so the company running it could, in principle, see and hear the call.

For most meetings that is acceptable and it is what makes server recording and captions possible. For sensitive calls, products offer end-to-end encryption (E2EE): media the SFU can forward but not read.

How E2EE works with an SFU

The trick is to encrypt the content of each encoded frame, but leave the RTP headers readable. The SFU still sees which stream a packet belongs to, its layer, and its timing, so it can route and pick layers as before. It just can't decode the picture.

The browser hook for this is the encoded transform API: RTCRtpScriptTransform hands each encoded frame to a worker before it is packetized (on the sender) and after it is reassembled (on the receiver). Safari has it since 15.4, Firefox since 117, and Chrome since 141; older Chromium versions only have an earlier form, createEncodedStreams (Insertable Streams). Feature-detect both, and turn E2EE off where neither exists.

The frame format is standardized as SFrame (RFC 9605): a small header with a key id and a counter, the encrypted frame, and an authentication tag. The webrtc lesson introduces it. The hard part for the app is the keys.

Each sender has a key; everyone in the call needs everyone's key; and when someone leaves, keys must change so they can't decrypt what comes next. Key exchange goes over the signaling channel, encrypted to each participant, and the room server never sees the keys. Standards like MLS (Messaging Layer Security) exist for exactly this group key problem.

The rest of the security list

• Who can join: the meeting link is not a secret, because it gets forwarded. Join tokens are short-lived, and the host can turn on a waiting room (lobby) where people wait until admitted, and lock the meeting.
• Host powers are checked on the server: the client hides the Remove button for guests, but the room server must still reject host-action from anyone who isn't a host.
• Chat is text, not HTML: render messages as text and turn URLs into links safely, as in the chat-app guide. A chat message is the easiest XSS route into a page that holds a camera.
• Embedding: if the call runs inside an iframe in another product, the frame needs allow="camera; microphone; display-capture", and the call page should list which sites may embed it in its frame-ancestors CSP rule.
• TURN and SFU credentials: short-lived and per person, from the snapshot.

Accessibility: keyboard control, announcements, and captions

A video call is a conversation, and some people join it with a keyboard only, a screen reader, low vision, or hearing loss. The goal is not a perfect score on a checklist; it is that each of them can mute, unmute, know who is in the call, and follow what is said. Three areas need real design: controls, announcements, and captions.

Keyboard control

• Shortcuts with a modifier for the two actions people need most, in a hurry: Google Meet uses Ctrl+D (Cmd+D on a Mac) for the microphone and Ctrl+E for the camera. Single-letter shortcuts clash with screen reader keys and with typing, so if offered at all they need a way to turn them off (WCAG 2.1.4, Character Key Shortcuts).
• Push to talk: holding the Space bar while muted unmutes until release, but only when focus is not in a text field like chat.
• The control bar is a toolbar: one tab stop, arrow keys between buttons (see the keyboard-navigation lesson for roving tabindex). Each toggle is a button with a fixed name like "Microphone" and aria-pressed, rather than a label that flips between "Mute" and "Unmute", which leaves people unsure whether the label is the state or the action.
• Panels manage focus: opening chat moves focus into the message box; closing it returns focus to the Chat button. Escape closes the top panel or menu.
• Tiles are reachable: each tile is a group named like "Maria, microphone muted, hand raised", with a menu button for Pin and host actions. Focus is visible on top of video, which needs a focus ring that contrasts with any picture.

Announcing without flooding

Screen reader users can't see 25 tiles change, so important changes are announced through a polite aria-live region. The danger is the opposite of silence: at 9:00 in a 100-person all-hands, 60 people join in 2 minutes. Announcing each one buries everything else, and a screen reader reads its queue in order, so the person hears "Ana joined" long after Ana spoke. The screen-readers lesson covers live regions; here are the rules for a call:

1. Batch joins and leaves: collect them for about 5 seconds, then announce once: "Ana, Ben, and 4 others joined."
2. In calls above a size (say 10 people), stop announcing joins by default; offer a setting to turn them back on.
3. Always announce what affects the person directly: "You are muted" after a shortcut, "The host muted you", "Recording started", "Your hand was lowered", "Maria is presenting".
4. Never announce the active speaker or every reaction. They change too often to be useful as speech.
5. Use role="alert" (an assertive announcement) only for things that need action now, like "Your microphone stopped working".

Captions and the transcript

Captions are the main accessibility feature for deaf and hard-of-hearing people, and helpful for everyone in a noisy room. Show them in a fixed area that doesn't cover faces or the shared screen, with the speaker's name, large readable text, and a setting for size. Don't make the caption area a live region: a screen reader would try to read every interim update, several per second. Screen reader users get the transcript panel instead, which they can read at their own pace, plus the option to hear the call itself.

Visual details matter too. The speaking indicator should not rely on color alone: use a thicker border or an icon as well. Mute and hand icons need 3:1 contrast against video, which is unpredictable, so put them on a solid badge. And respect prefers-reduced-motion for reactions and tile animations.

Observability: knowing how calls really went

"The call was bad" is the most common complaint about any meeting product, and the least useful. Was it the person's Wi-Fi, a TURN relay in the wrong region, a CPU-limited laptop, or the SFU? Without data from the client, every answer is a guess. The client has the best view of quality, because it is where audio is played and video is shown, and WebRTC exposes almost everything through one call: pc.getStats().

What to read from getStats

getStats returns a report of typed entries. Read it every 2 to 5 seconds, compute differences between readings (most counters only grow), and keep a few numbers per stream:

From numbers to a quality score

Engineers want the raw numbers; dashboards want one score. Voice quality has a long-standing model for this, the ITU-T E-model (G.107), which estimates a rating from delay, loss, and codec. That rating maps to a MOS (mean opinion score), the 1 (bad) to 5 (excellent) scale from listening tests; the E-model tops out at about 4.5. Products often use simplified versions of it for WebRTC.

Treat such a score as an estimate, not a measurement: it can't hear an echo or a robotic voice. Calibrate it with the 1 to 5 star survey shown to a small sample of people after calls, and look at the cases where the two disagree.

Events and summaries, not raw streams

Sending every stats reading from every client would flood the backend: 100 people, each with dozens of streams, every 2 seconds. Instead, the client sends a few kinds of data, all tagged with the call id, a session id, the browser, and the SFU region (the real-user-monitoring lesson covers sampling and beacons):

• Milestones: page load to preview, Join click to first audio heard (our 3-second target), the ICE candidate type that won.
• Incidents: every reconnect with its cause and how long media was gone, every device error by name, every time a degradation step kicked in.
• A summary per minute: average and worst round trip, loss, frames per second, time spent CPU-limited, number of visible tiles.
• An end-of-call summary, sent with navigator.sendBeacon on Leave and on pagehide, so it survives the tab closing.

Running it in the interview

This question has more depth than any 45 minutes can hold. The interviewer is watching whether you find the core quickly (media topology and what each client sends and receives), whether you reason with numbers, and whether you know which parts are frontend work and which belong to the media server. Here is one way to spend the time.

Follow-up questions and strong answers

Interviewers use follow-ups to push on a decision you made and see whether you know its cost. The best answers name the trade, give a number, and say what you'd measure before changing course. Three common ones:

Common mistakes

• Designing a mesh for 100 people, or not knowing why it fails.
• Subscribing to 720p for every tile, so a 25-tile grid downloads over 30 Mbps.
• Putting MediaStream objects in a global store and leaking tracks (the camera light stays on after leaving).
• Switching the big tile on every sound, without hysteresis.
• Treating a dropped connection as the end of the call instead of restarting ICE.
• Promising E2EE together with server recording and captions.
• Announcing every join in a live region.