WebRTC

The problem: a conversation can't wait

You are adding a "Call" button to a team chat app. Maya clicks it, Leo answers, and they talk with cameras on.

For that to feel like a normal conversation, Maya's voice has to reach Leo's speakers in well under 300 ms, measured from her mouth to his ear. Past about 300 ms, people start talking over each other. Past 500 ms, the call feels broken.

The chat app already has a WebSocket to its server. The first idea is to reuse it: record 20 ms chunks of audio and small video frames, send them up the socket, and let the server forward them to Leo. This works in a demo on office Wi-Fi. It falls apart on a real network.

Why TCP hurts live media

A WebSocket runs on TCP, and TCP promises that every byte arrives, in order. When one packet is lost, TCP holds back every later packet until the lost one is sent again. That retry takes at least one round trip, often 100 to 300 ms on a mobile network. Meanwhile, Leo hears silence, and then a burst of stale audio.

For a file, waiting is right: a missing byte ruins the file. For a call, waiting is wrong.

A 20 ms piece of audio that arrives 200 ms late is useless, because the moment has passed. It's better to skip it, smooth over the gap, and play the next piece on time. Live media wants a transport that is allowed to lose things.

There is a second cost: the detour. If Maya and Leo sit in the same city and the server is on another continent, every packet travels there and back. A direct path between their two devices could be 10 to 30 ms, while the relay adds 100 ms or more each way. And the server now pays for every byte of every call, in both directions.

What WebRTC gives you

WebRTC (Web Real-Time Communication) is a set of browser APIs and network protocols, standardized by the W3C and the IETF, that lets two browsers send audio, video, and arbitrary data straight to each other. It usually runs over UDP, a transport that sends packets without waiting for lost ones. Everything is encrypted, always. On a good network, the delay from one person's microphone to the other's speaker usually stays under about 200 ms.

It ships in every modern browser (Chrome, Edge, Firefox, Safari, and their mobile versions), so there is no plugin. Google Meet, Discord voice in the browser, Microsoft Teams on the web, Whereby, and most "join a call from a link" products are built on it. Screen sharing in a browser is WebRTC too.

The surprise for most engineers is how many servers a "peer to peer" technology needs. Peers can't find each other on their own, they often can't reach each other through home routers, and in a group call they can't afford to send video to everyone. This lesson walks through each of those servers, why it exists, and what it means for a frontend design.

The mental model: a direct line, set up by servers

Think of two people who want to talk on walkie-talkies but don't know each other's channel. They first exchange a note through a friend: "I'm on channel 7, my code is 4410". Once both have the note, they talk directly, and the friend is no longer involved. If a wall blocks the signal, a repeater on a hill can pass it along.

That is WebRTC. The friend passing notes is the signaling server. The note is the session description: which codecs each side supports, the encryption fingerprint, and the network addresses where it can be reached.

The direct talk is the peer connection. The repeater on the hill is a TURN server. Keep this picture: setup goes through servers, media goes directly whenever it can.

The pieces, from bottom to top

• Media capture: navigator.mediaDevices.getUserMedia() asks for the camera and microphone and returns a MediaStream made of MediaStreamTracks. getDisplayMedia() does the same for screen sharing.
• The peer connection: RTCPeerConnection is the object that owns one link to one other peer. You add tracks to it, it negotiates codecs, finds a network path, encrypts, sends, and adapts to bandwidth.
• Data channels: RTCDataChannel, created from a peer connection, sends text or binary messages to the other peer, with reliability you choose per channel.
• Signaling: your own channel (usually the WebSocket you already have) that carries the setup messages between the two peers. WebRTC does not define it on purpose.
• STUN and TURN: servers on the public internet that help a peer learn its public address (STUN) or relay its traffic when no direct path works (TURN).
• Media servers: for calls with more than a few people, a server called an SFU (Selective Forwarding Unit) receives each person's stream once and forwards it to the others.

What the browser does for you, and what it leaves to you

The browser handles the hard real-time work: echo cancellation, noise suppression, jitter buffers (a small queue that smooths out packets arriving at uneven times), packet loss concealment, codec choice, bandwidth estimation, and encryption. You never see an RTP packet (Real-time Transport Protocol, the packet format for live media) unless you go looking for one.

Your app owns everything around that: who is allowed to call whom, how the setup messages travel, which STUN and TURN servers to use and how to give out credentials for them, whether group calls go through a media server, and the whole user experience of permissions, devices, mute, and reconnects. In an interview, that boundary is the most useful thing to draw.

How a call connects, step by step

Let's follow Maya calling Leo, from the click to the first frame of video. The whole setup usually takes 0.5 to 2 seconds. Most of that time goes into waiting for the other person's browser and into network checks, not into the code you write.

Offer and answer

1. Maya's app calls getUserMedia({ audio: true, video: true }). The browser shows the permission prompt; once she allows it, the camera light turns on and the app gets a stream.
2. The app creates an RTCPeerConnection with a list of STUN and TURN servers, and adds the audio and video tracks with addTrack().
3. It calls createOffer(). The browser writes an offer: a text document in SDP (Session Description Protocol) listing the tracks, the codecs Maya's browser supports, and a fingerprint of its encryption certificate.
4. It calls setLocalDescription(offer), then sends the offer to Leo over the signaling channel: a WebSocket message like { type: "offer", to: "leo", sdp }.
5. Leo's app shows "Maya is calling". When he answers, his app gets its own media, creates its own peer connection, and calls setRemoteDescription(offer).
6. Leo's browser calls createAnswer(), which picks the codecs both sides support, and setLocalDescription(answer). The answer goes back to Maya over signaling, and she applies it with setRemoteDescription(answer).

This offer and answer exchange is the core of WebRTC. Both browsers now agree on what to send and how to encrypt it. What they don't know yet is how to reach each other. That is the job of ICE.

ICE candidates, trickled

As soon as setLocalDescription() runs, each browser starts gathering ICE candidates: possible addresses where it might be reachable. ICE (Interactive Connectivity Establishment) is the procedure that finds a working path. A candidate looks like "UDP, 192.168.1.20, port 54321" or "UDP, 203.0.113.7, port 61002, learned from STUN".

Each candidate fires an icecandidate event. The app sends it to the other peer over signaling right away, and the other side adds it with addIceCandidate(). Sending candidates one at a time as they appear is called trickle ICE. It lets the connection checks start while slower candidates (like a TURN allocation) are still being gathered, which saves hundreds of milliseconds.

The two browsers then pair up local and remote candidates and send small STUN check packets on each pair, both directions. The first pairs that get answers become usable, and ICE picks the best one (direct beats relayed). The ICE part of the connection is now connected.

Encryption, then media

On the chosen path, the browsers run a DTLS handshake (TLS adapted for UDP). Each side checks that the certificate it sees matches the fingerprint from the SDP it received over signaling. The handshake produces keys for SRTP (Secure RTP), which encrypts the audio and video packets.

Then media flows. Leo's app already got a track event when it applied Maya's offer and put her stream in a video element; now frames start to play.

const pc = new RTCPeerConnection({ iceServers });
const media = await navigator.mediaDevices.getUserMedia({ audio: true, video: true });
media.getTracks().forEach((track) => pc.addTrack(track, media));

pc.onicecandidate = ({ candidate }) => {
  if (candidate) signal.send({ type: "candidate", to: peerId, candidate });
};
pc.ontrack = ({ streams }) => {
  remoteVideo.srcObject = streams[0];
};

// Caller side
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
signal.send({ type: "offer", to: peerId, sdp: pc.localDescription });

signal.on("answer", (msg) => pc.setRemoteDescription(msg.sdp));
signal.on("candidate", (msg) => pc.addIceCandidate(msg.candidate));

Reading an SDP

You rarely edit SDP, but you will read it while debugging. It's plain text, one setting per line. A few lines carry most of the meaning:

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 9 0 8
a=mid:0
a=sendrecv
a=rtpmap:111 opus/48000/2
a=fingerprint:sha-256 4A:1F:...:C2
a=setup:actpass
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 45
a=mid:1
a=rtpmap:96 VP8/90000
a=rtpmap:98 VP9/90000
a=rtpmap:45 AV1/90000
a=candidate:1 1 udp 2122260223 192.168.1.20 54321 typ host

Getting through NAT: ICE, STUN, and TURN

Why can't Maya's browser just send packets to Leo's IP address? Because neither of them has one that the other can reach. Almost every device sits behind a NAT (Network Address Translation): the home router, the office firewall, or the mobile carrier gives the device a private address like 192.168.1.20 and shares one public address among many devices.

A NAT lets traffic out freely and lets replies back in. It drops anything unsolicited from outside, because it has no idea which device inside should get it. So Leo can't simply "call" Maya's laptop. ICE gets around this with three kinds of candidates, tried from best to worst.

Three kinds of candidates

STUN (Session Traversal Utilities for NAT) is a tiny service: the browser sends a packet, and the STUN server replies "I saw you as 203.0.113.7:61002". That is the hole the NAT opened. If both peers send to each other's reflexive addresses at about the same time, both NATs see outgoing traffic first and let the replies in. This trick is called hole punching, and it works for most home networks and many mobile ones.

It fails in two common cases. Some NATs (called symmetric NATs, common in corporate and some carrier networks) give a different public port for every destination, so the address STUN saw is useless for talking to Leo. And some firewalls block all UDP, or everything except ports 80 and 443.

TURN (Traversal Using Relays around NAT) is the fallback. The browser asks the TURN server for an allocation, and the server forwards packets between the two peers. TURN can run over UDP, TCP, or TLS on port 443, which looks like normal HTTPS to most strict firewalls. The media stays encrypted end to end with SRTP; the TURN server only moves packets it can't read.

How often calls need TURN

Published figures are mostly old and vary a lot by audience. The number most often quoted by WebRTC vendors and in older large-scale measurements is roughly 10 to 20 percent of calls relayed, lower for consumer home users and much higher on corporate networks. Plan for a meaningful share, not an edge case.

That share matters because TURN is where the bandwidth bill lives. A 1.5 Mbps video call relayed for 30 minutes moves about 340 MB through your server per direction. Teams run TURN servers in several regions (coturn is the common open-source one) or buy it from a provider, and they put the TURN server close to users, because a relay on another continent adds the same detour we tried to avoid.

// Fetched from your API right before the call; credentials expire in minutes.
const { iceServers } = await fetch("/api/rtc/ice-servers").then((r) => r.json());
// iceServers looks like:
// [
//   { urls: "stun:stun.example.com:3478" },
//   { urls: ["turn:turn.example.com:3478?transport=udp",
//            "turns:turn.example.com:443?transport=tcp"],
//     username: "1760900000:maya", credential: "hmac-of-username" }
// ]
const pc = new RTCPeerConnection({ iceServers });

Media tracks and data channels

Once connected, a peer connection carries two kinds of traffic. Media tracks carry audio and video over SRTP, tuned for live playback. Data channels carry your own messages over SCTP (a message protocol) inside the same encrypted DTLS link. Both share one network path and one set of ports.

Media tracks and bandwidth

Each track you add gets an RTCRtpSender on your side and an RTCRtpReceiver on the other. The browser picks the codec from the negotiated list: Opus for audio, and VP8, VP9, H.264, or AV1 for video, depending on the browsers and the hardware. You can reorder the preference with transceiver.setCodecPreferences() before creating the offer.

Typical bitrates give you a feel for the numbers you'll use in the bandwidth math later. These are rough and depend on codec, motion, and settings:

The browser runs congestion control on the media: it watches delay and loss reported back by the receiver, estimates how much bandwidth the path has, and lowers the bitrate, frame rate, or resolution when the path gets worse. You can cap it with sender.setParameters() (for example maxBitrate on an encoding) and read what is happening with pc.getStats().

Data channels and reliability

pc.createDataChannel("game", options) opens a channel to the peer. The API looks like a WebSocket: send(), an onmessage handler, open and close events. The difference is that you choose, per channel, whether messages must arrive and whether they must arrive in order.

The unreliable modes are the reason game developers care about data channels. A player's position is sent 30 times a second. If one update is lost, the next one, 33 ms later, replaces it. Waiting to resend the old one would only make every later update late too, which is exactly the TCP problem from chapter one.

Data channel traps

• Message size. Keep messages small (about 16 KB is the long-standing safe size across browsers) and split files into chunks. Very large messages can block other messages on the same connection.
• Backpressure. send() never blocks; it queues. For file transfer, watch bufferedAmount and wait for the bufferedamountlow event before sending more, or you'll fill memory.
• One per purpose. Use a reliable channel for chat and a separate unreliable one for positions. Mixing them on one ordered channel makes the cursor wait behind a big chat message.
• It's peer to peer. A data channel goes to one peer. With an SFU, data usually goes through the SFU or through your WebSocket instead; check what your media server supports.

Group calls: mesh, SFU, and MCU

A peer connection links exactly two browsers. With three or more people, you have to decide how media gets from everyone to everyone. There are three shapes, and the choice comes down to one thing: whose upload bandwidth and CPU pays for each extra person.

The bandwidth math

Take a call with N people, each sending video at b = 1 Mbps. Upload is the scarce resource: home connections often upload at a fraction of their download speed, and mobile upload drops sharply with signal.

• Mesh: everyone connects to everyone. Each browser keeps N minus 1 peer connections and encodes and uploads its video N minus 1 times. With 6 people: 5 Mbps up and 5 Mbps down per person, and 15 links in total. Doubling to 12 people means 11 Mbps up each, which most laptops can neither send nor encode.
• SFU (Selective Forwarding Unit): everyone sends one stream to a server, and the server forwards copies to the others without decoding them. Upload stays at 1 Mbps no matter how big the call is. Download is still up to N minus 1 streams, but the SFU can send smaller versions of most of them.
• MCU (Multipoint Control Unit): the server decodes every stream, composes them into one picture, re-encodes it, and sends each person one stream. 1 Mbps up, about 1 Mbps down, for any N. The server pays for decoding and encoding everything, adds delay for that work, and forces one fixed layout on everyone.

The lab below uses 1.5 Mbps, a typical 720p stream, so its numbers are 1.5 times the ones above. Move the slider and watch one person's upload against a modest 5 Mbps home uplink.

Simulcast: one sender, several qualities

In a 10-person grid, Leo sees Maya as a small 180p tile. Priya, on a big screen with Maya as the active speaker, wants 720p. Raj is on a train with 500 kbps of download.

One stream can't suit all three. With simulcast, Maya's browser encodes her camera three times at different sizes, and the SFU forwards whichever layer each receiver should get.

pc.addTransceiver(cameraTrack, {
  direction: "sendonly",
  sendEncodings: [
    { rid: "q", scaleResolutionDownBy: 4, maxBitrate: 150_000 },
    { rid: "h", scaleResolutionDownBy: 2, maxBitrate: 500_000 },
    { rid: "f", maxBitrate: 1_500_000 },
  ],
});

Simulcast costs the sender a little more upload (here about 2.15 Mbps instead of 1.5) and some CPU, in exchange for every receiver getting a fitting stream. SVC (Scalable Video Coding, supported by VP9 and AV1) gets a similar result with one stream built from layers, and the SFU drops the layers a receiver doesn't need.

The frontend has a real part to play here. The client tells the SFU which tiles are visible and how big they are, so the SFU sends 720p only for the large tile and nothing at all for participants scrolled off screen. In a 50-person call with nine tiles on screen, that is about 3 Mbps of download (one 720p stream and eight 180p tiles), instead of about 25 Mbps for all 49 streams at 360p.

What breaks in production

A WebRTC demo works on the first try. A WebRTC product spends most of its engineering on the cases below. Each one has a symptom users will report in their own words, so it helps to know what they mean.

Connection states and ICE restarts

RTCPeerConnection reports its health through connectionState, with a connectionstatechange event on every change. The states are new, connecting, connected, disconnected, failed, and closed. Two of them need different handling, and mixing them up is a classic bug.

• disconnected means checks on the current path stopped getting answers, for example in a Wi-Fi blip or a tunnel. It often recovers on its own within a few seconds. Show "Reconnecting" and wait; don't tear the call down.
• failed means ICE gave up on every candidate pair. It won't recover by itself. Call pc.restartIce(), which gathers fresh candidates and triggers a new offer and answer over signaling. If that fails too, rebuild the peer connection.

A network change, like walking out of the office from Wi-Fi to 4G, changes the device's addresses. The old path dies, and the call needs an ICE restart to find a new one. Good apps listen to online events and the connection state and restart early, so the user hears a one-second gap instead of a dropped call.

Glare: both sides offer at once

Renegotiation happens whenever tracks change: turning on the camera, starting a screen share, an ICE restart. The browser fires negotiationneeded, and the app sends a new offer. If both peers do this at the same moment, each receives an offer while holding its own, and both get stuck. This is called glare.

The standard fix is the perfect negotiation pattern from the WebRTC spec examples. One peer is "polite" and one is "impolite" (decided by, say, comparing user ids). On a collision, the impolite peer ignores the incoming offer; the polite peer rolls back its own and accepts the other. Use it from day one; adding it later is painful.

Devices and permission errors

Devices also change mid-call: headphones plugged in, a Bluetooth headset connecting. Listen to navigator.mediaDevices devicechange, refresh the device list with enumerateDevices(), and swap tracks with sender.replaceTrack(), which switches the camera or microphone without renegotiating.

Quiet failures

• One-way audio. Usually a track never added on one side, a muted MediaStreamTrack, or a video element that never started playing. Log both sides' getStats(): if bytesSent rises on one side and bytesReceived stays flat on the other, the problem is in the path or the receiver.
• Remote video stays black on iPhones. iOS Safari needs playsinline and autoplay on the video element, and the join click must happen before media plays so autoplay rules allow sound.
• Signaling drops mid-call. The media keeps flowing, because it doesn't go through signaling. Reconnect the WebSocket in the background (with backoff) and only worry when you need to renegotiate.
• Quality slowly falls apart. Collect getStats() every few seconds: packetsLost, jitter, roundTripTime, framesPerSecond, and the selected candidate type. Send a summary to your backend so you can see that, say, relayed calls in one region are worse.

Security, privacy, and the permission experience

Always encrypted, but check who holds the keys

WebRTC has no unencrypted mode. The standards make DTLS-SRTP mandatory: media goes over SRTP with keys from a DTLS handshake, and data channels run inside DTLS. A coffee-shop Wi-Fi snooper or a TURN server in the middle sees only encrypted packets.

The weak point is signaling. Each peer trusts the certificate fingerprint it received in the other's SDP. Someone who can change signaling messages can swap in their own fingerprint and sit in the middle of the call. So signaling must run over wss://, be authenticated, and the server must check that Maya is allowed to call Leo or join that room before passing anything along.

With an SFU, encryption is hop by hop: Maya to SFU, then SFU to Leo. The SFU ends each encrypted hop and holds its keys, so the server operator could in principle see the media. For end-to-end encryption through an SFU, products encrypt each encoded frame again in the browser with keys the server never gets, using encoded transforms (RTCRtpScriptTransform, with an older Chrome-only form called Insertable Streams). SFrame (RFC 9605) standardizes that frame format.

IP address privacy

ICE candidates contain addresses. Early WebRTC leaked a user's private LAN address to any page; browsers now replace host candidates with random .local names (mDNS, a naming scheme for the local network) unless the page already has camera or microphone permission. The public address in srflx candidates is still shared with the other peer.

If peers must not learn each other's IP (a marketplace call between strangers, a support call), set iceTransportPolicy: "relay". All traffic then goes through TURN, which costs bandwidth but reveals only the relay's address. An SFU design gets the same benefit for free, since peers only ever talk to the server.

The permission prompt is part of your design

getUserMedia() works only in a secure context (HTTPS or localhost); on plain HTTP, navigator.mediaDevices doesn't even exist. It shows a browser prompt unless the user already decided, and how long that decision is remembered differs by browser. If the user clicks Block in Chrome, the page can't ask again; the user has to find the site settings themselves. So the moment and framing of that first prompt decide whether people can join at all.

• Don't prompt on page load. Show a pre-join screen first, explain why you need the camera, and request access when the user clicks "Turn on camera". getUserMedia() doesn't require a click, but the spec does require one for getDisplayMedia(), so start screen sharing from a button.
• Use the Permissions API (navigator.permissions.query({ name: "camera" })) where supported to know the state ahead of time and show the right message, not a prompt that won't appear.
• Show a live preview, a microphone level meter, and device pickers before joining. Most "they can't hear me" support tickets are the wrong microphone.
• Always let people join with camera off, or audio only, or listen only. A denied camera should never block the call.
• Stop tracks with track.stop() when leaving. Otherwise the camera light stays on, which users read as being watched.

Accessibility

• Mute and camera buttons are toggles: real button elements with aria-pressed and a label that doesn't change meaning ("Microphone", pressed or not), plus visible state that isn't color alone.
• Announce joins, leaves, and "you are muted" through a polite aria-live region, rate-limited so a 30-person call doesn't flood a screen reader.
• Offer live captions, either from the platform or a speech-to-text service on the SFU's audio. Deaf and hard-of-hearing users, and anyone in a noisy room, depend on them.
• Keyboard shortcuts for mute and leave help everyone, but they must not clash with screen reader keys, and should be listed somewhere visible.

When a design needs WebRTC, and when it doesn't

WebRTC is the right tool for a narrow set of jobs and a costly one outside them. The question to ask is: does the payload need to move between people (or between a person and a server) in under about half a second, and is it media or data that's fine to lose a little of?

Good fits

• Voice and video calls, one-to-one or in groups: the main reason WebRTC exists.
• Screen sharing and remote help: getDisplayMedia() plus a peer connection, with low frame rate and sharp text.
• Interactive live video: a host talking with a few guests on stage, live auctions, watch parties where viewers can speak. Sub-second delay is the whole product.
• Low-latency data between users: two-player games, collaborative drawing with live cursors between a few people, peer-to-peer file transfer without uploading to a server first.
• Cloud gaming and remote desktops, where a server streams video to the browser and input must go back within tens of milliseconds.

The alternatives

WebSocket (and SSE) is the default for data between a browser and your server: chat, notifications, presence, collaboration operations. It's simpler, works through every proxy, and your server sees every message, which you usually want for storage and permissions. The WebSockets lesson covers it; in a call app, it's also your signaling channel.

WebTransport is a newer browser API for client-to-server connections over HTTP/3 (QUIC). Like a data channel, it offers unreliable datagrams and many independent streams without head-of-line blocking, but it talks to a server, not to a peer, and needs no ICE, SDP, or STUN. Chrome, Edge, and Firefox support it, and Safari added it in version 26.4, though you still need an HTTP/3 server and a WebSocket fallback for networks that block UDP. It suits game servers and custom media pipelines more than calls.

HLS and DASH (and their low-latency versions, LL-HLS and LL-DASH) cut video into small files served over normal HTTP and CDNs. Standard HLS runs 6 to 30 seconds behind live; low-latency modes get to roughly 2 to 5 seconds. In exchange, a CDN can serve millions of viewers at very low cost per viewer. Twitch, YouTube Live, and sports streaming work this way.

Using WebRTC in a system design interview

WebRTC comes up in prompts like "design Google Meet", "add video calls to Slack", "design a live streaming site with guests", or "design a multiplayer drawing game". It can also come up as a trap: a candidate who reaches for WebRTC to build a chat app is choosing the hard tool for an easy job.

What to draw

1. The client: media capture, the peer connection (or one connection to the SFU), the call UI, and a stats reporter.
2. The signaling path: the app's existing WebSocket, an authenticated room service behind it, and the messages (join, offer, answer, candidate, leave).
3. STUN and TURN, with a note that TURN credentials come short-lived from your API and that a share of calls will be relayed.
4. The media path: mesh for one-to-one, an SFU for groups, with simulcast and the client telling the SFU which tiles are visible.
5. Anything else the prompt needs: recording off the SFU, HLS for a large audience, captions.

Then do the bandwidth math out loud. "Eight people at 1 Mbps in mesh is 7 Mbps of upload each, which doesn't work on most home connections, so I'll use an SFU. Each client uploads about 2 Mbps of simulcast and downloads one 720p stream plus seven 180p tiles, around 2.5 Mbps." That one sentence shows you understand the core trade-off.

How deep to go, by level