Design a browser video editor

Requirements, and what makes video hard

A non-linear editor (NLE) is the kind of program where you place pieces of video anywhere on a timeline and change them later without touching the original files. Premiere Pro, Final Cut and DaVinci Resolve are desktop NLEs. Clipchamp, CapCut on the web, Canva's video editor and Kapwing bring a simpler version into a browser tab. There we share memory with every other tab, run JavaScript on one main thread, and reach the media hardware only through web APIs.

Several published guides cover nearby ground, and we link them instead of repeating them. The music studio (music-daw) guide covers the Web Audio graph, the look-ahead scheduler, AudioWorklet, waveform peaks, the clip-on-track model and offline audio export. The photo editor guide covers non-destructive edit stacks and GPU filters on a still image, and the video player and audio player guides cover playback, seek bars, captions and the media session.

The undo and redo guide covers command history, the file uploader guide covers resumable uploads of big files, and Google Docs and Figma cover real-time collaboration. This guide focuses on what only a video editor has: exact frames out of compressed files, several layers composited at 30 frames per second, edits that map to exact source frames, and a whole film encoded on the user's device. Start with questions; each answer changes the design.

Functional requirements

• Import video, audio and image files by drag and drop or a file picker, without waiting for an upload.
• Place clips on video and audio tracks; trim the start and end, split at the playhead, move, delete with or without closing the gap (ripple), and snap to other clip edges and the playhead.
• Add text titles, image and sticker overlays, simple transitions (cross-fade, wipe) and effects (brightness, contrast, saturation, blur, a LUT colour grade).
• Add music and voice-over; set volume per clip; fade in and out.
• Preview the frame at the playhead exactly while scrubbing, and play the project in real time with sound in sync.
• Export to MP4 at 720p or 1080p with a progress bar, an estimate, and a cancel button.
• Undo and redo every edit; save automatically; reopen the project on another computer.
• Author captions: type or import subtitles, place them on the timeline, and burn them in or export them as a separate file.

Non-functional requirements

• Exact frames: when paused, the preview shows the exact source frame the timeline maps to. A cut that looks right in preview must be the same cut in the export.
• Smooth preview: 30 frames per second on a mid-range laptop for 2 overlapping 1080p clips plus text; at most a few dropped frames per minute. Each frame has a budget of 33 ms.
• Responsive editing: dragging a clip or trim handle answers within one frame (16 ms at 60 Hz). Seeking to a new spot shows a picture within about 150 ms and the exact frame within about 500 ms (our targets).
• Memory: stay under about 1.5 to 2 GB for the tab on an 8 GB laptop, whatever the length of the footage (our budget, not a browser limit).
• Export speed: on a laptop with hardware encoding, export a 1080p30 project faster than real time (our target: a 5 minute video in under 5 minutes).
• Never lose work: every edit saved locally within a second and to the server within a few seconds.
• Accessible: the timeline and every edit work from the keyboard, and screen reader users hear the playhead position and the result of each edit.

Out of scope

We leave out live streaming (see the live streaming guide), multi-camera sync, motion tracking, AI features such as background removal (they would run as separate models on the GPU or the server), and real-time co-editing of one timeline. HDR and wide colour gamut are also out: we assume 8-bit SDR video in the Rec. 709 colour space, which covers almost all phone and screen footage people post.

What makes video hard in a browser

A photo editor holds one image. A music studio holds a few minutes of audio, which is small enough to decode fully into memory (the music-daw guide does exactly that). Video breaks both habits. Do the math once and every later decision follows from it.

So the compressed file is already a gigabyte, and the decoded clip is about 75 times our 2 GB tab budget, far more than any laptop's RAM. You cannot decode everything ahead. The editor must decode just the frames it needs, just before it needs them, and throw them away soon after. Even a one second cache of RGBA frames costs a quarter of a gigabyte, so a careful editor keeps frames in the decoder's own format or on the GPU, and only a short window around the playhead.

Frames depend on other frames

Compressed video is not a list of pictures. A codec such as H.264 (also called AVC) stores a full picture only now and then: a keyframe (an I-frame, intra-coded). The frames between keyframes store only what changed: P-frames predict from earlier frames, and B-frames predict from frames before and after. A keyframe plus the frames that depend on it is a GOP (group of pictures).

GOP length varies by device and settings. Phones often place a keyframe every 1 to 2 seconds, while screen recorders and some encoders space them 5 to 10 seconds apart or more. Two consequences follow. First, to show the 60th frame of a GOP you must decode all 60, starting at the keyframe, because each frame is built from earlier ones; the last frame of a 10 second GOP at 30 fps costs 300 decodes.

Second, B-frames mean the decode order in the file differs from the presentation order on screen, so the decoder hands frames back in a different order than you fed them. Every timestamp in the editor must therefore use presentation time.

Five forces that shape the design

1. Too big to hold: decode on demand, cache a small window, keep decoded frames on the GPU when possible, and close each frame as soon as it is drawn.
2. Seeking is expensive: any jump means going back to a keyframe and decoding forward. Scrubbing needs a cheap path (thumbnails, keyframes, low-resolution proxies) and an exact path that arrives a moment later.
3. Several streams at once: a cross-fade, a picture-in-picture or a background video means 2 to 4 decoders running inside one 33 ms frame budget. Hardware decoders are fast but limited in number, and pages cannot query the limit.
4. Sound is the clock: viewers forgive a dropped frame but not audio that drifts away from lips, so audio keeps time and video follows it.
5. Export is a long batch job: 9,000 frames for a 5 minute video, each decoded, composited and encoded, into a file that can be gigabytes. It must show progress, allow cancel, and never freeze or crash the tab halfway.

Getting exact frames out of a file

Everything the editor shows starts with one question: give me the picture at source time T of file F. The preview asks it 30 times a second per visible clip, the thumbnail strip asks it hundreds of times, and export asks it for every frame of the film.

The answer has two steps. Demuxing opens the container (MP4, MOV, WebM) and lists the compressed samples with their timestamps, sizes and keyframe flags. Decoding turns compressed samples into pictures.

The naive design uses what the browser already has: one hidden video element per clip. To show source time 12.4 s, set currentTime = 12.4, wait for the seeked event, and call drawImage(video, ...) on a canvas. It works for a demo and falls apart in an editor. Each seek is asynchronous and decodes from the previous keyframe internally, so a seek inside a long GOP can take hundreds of milliseconds, and you cannot batch or cancel it well.

Time is a float in seconds, so near a frame boundary you cannot be sure which frame is shown. requestVideoFrameCallback reports the mediaTime of each presented frame after the fact, which helps but does not let you ask for a frame. fastSeek() (Firefox and Safari only) is faster precisely because it gives up precision and lands near a keyframe. Media elements also live only on the main thread, cannot step backwards frame by frame, and drift apart when several play together.

Demuxing into a sample index

When a file is imported, a worker demuxes it once and builds a sample index: for each compressed sample, its presentation timestamp, its position in decode order, its byte offset and size in the file, and whether it is a keyframe. With mp4box.js, the onReady callback gives the track list and the codec string, for example avc1.640028 for H.264 High profile, level 4.0.

The track's avcC box becomes the description field of the decoder config. The index for a 10 minute clip is 18,000 small records, well under a megabyte, and it is cached in IndexedDB so reopening a project does not demux again.

The index makes seeking cheap to plan. To get the frame at source time T: binary search for the sample whose presentation time covers T, walk back to the nearest keyframe at or before it, then feed samples in decode order from that keyframe until the decoder outputs the frame you want. Frames before it are closed straight away.

// In a worker. targetUs is presentation time in microseconds.
async function frameAt(src: Source, targetUs: number): Promise<VideoFrame> {
  const target = src.index.sampleAt(targetUs);   // binary search by pts
  const key = src.index.keyframeAtOrBefore(target);
  let found: VideoFrame | null = null;
  const decoder = new VideoDecoder({
    output: (frame) => {
      const covers = frame.timestamp + (frame.duration ?? 0) > targetUs;
      if (!found && covers) found = frame;
      else frame.close();                        // never hold frames you will not draw
    },
    error: (e) => src.reportError(e),
  });
  decoder.configure(src.config);                 // codec string, avcC description, size
  for (const s of src.index.inDecodeOrderFrom(key)) {
    if (found) break;
    const data = await src.bytes(s.offset, s.size);  // File.slice(...).arrayBuffer()
    decoder.decode(new EncodedVideoChunk({ type: s.isKey ? "key" : "delta", timestamp: s.ptsUs, duration: s.durUs, data }));
    if (decoder.decodeQueueSize > 8) {
      await new Promise((r) => decoder.addEventListener("dequeue", r, { once: true }));
    }
  }
  await decoder.flush();
  decoder.close();
  if (!found) throw new Error("frame not found");
  return found;
}

Rules a decoder wrapper must follow

• Start from a keyframe. After configure() or flush(), the next chunk must be a key chunk, or decode() throws a DataError.
• Close every frame. A VideoFrame holds decoder memory, often a GPU buffer. The WebCodecs spec warns that holding output frames can stall decoding, because the decoder hands frames out of a small pool. Close each frame as soon as it is drawn or copied.
• Apply backpressure. decode() returns at once and queues work. Watch decodeQueueSize and the dequeue event so you never queue hundreds of chunks.
• Reuse, do not recreate. A seek while playing calls reset() (drop queued work), then configure() and feeds from the new keyframe. Creating a decoder costs time (a few milliseconds, estimate) and may claim a scarce hardware slot.
• Treat errors as normal. After the error callback the decoder is closed for good. The wrapper builds a new one, retries once, and then falls back to software decoding (hardwareAcceleration: 'prefer-software') or marks the clip as needing a proxy.

The table explains why long-GOP files feel sticky to scrub and why editors make proxies: a low-resolution copy encoded with a keyframe every few frames is cheap to seek. The preview chapter returns to proxies.

The project model: a timeline of references

The project is a small document that points at big files. A clip never contains pixels. It says: take media M, starting at source time S, play it for D frames, at timeline position P, on track T, with these transforms and effects. That is what makes editing non-destructive: trimming a clip changes two numbers, and the original file is untouched.

The music-daw guide builds the same clip-on-track model for audio, with the same split between a light project document and heavy media. We reuse it and cover only what video adds.

• Layer order: video tracks stack. Track 1 is the bottom layer; a clip on track 3 draws over clips on tracks 1 and 2 at the same moment.
• Transforms: position, scale, rotation, crop and opacity per clip, so a clip can be a picture-in-picture or a split screen. Values can be animated with keyframes (a list of time and value pairs with an easing).
• Transitions: a cross-fade needs both clips to have frames during the overlap, so a transition borrows extra source frames from the clips' handles (source material beyond the in and out points).
• Text and image overlays: clips with no media file, drawn by the compositor from their own properties.
• Effects: an ordered list per clip of GPU shader effects with parameters, like the photo editor's edit stack but evaluated per frame and animatable.
• Frame rate and orientation: sources can be 24, 25, 29.97, 30 or 60 fps, often with a variable frame rate on phones, and phone video stores rotation as container metadata rather than in the pixels.

How to represent time

The most important choice in the model is the unit of time. It sounds trivial and causes more off-by-one-frame bugs than anything else in an editor, because preview and export must agree to the exact frame.

type Rational = { value: number; timescale: number };  // value / timescale seconds

type VideoClip = {
  id: string;
  kind: "video" | "image" | "text";
  mediaId?: string;          // into project.media; missing for text
  start: number;             // timeline frame where the clip begins
  duration: number;          // in timeline frames
  sourceIn: Rational;        // where in the source the clip starts
  speed: Rational;           // 1/1 normal, 1/2 slow motion
  transform: { x: number; y: number; scale: number; rotation: number; opacity: number };
  keyframes: Record<string, { frame: number; value: number; ease: "linear" | "in-out" }[]>;
  effects: { type: "color" | "blur" | "lut"; params: Record<string, number | string> }[];
  transitionIn?: { type: "crossfade" | "wipe"; frames: number };
};

type Track = { id: string; kind: "video" | "audio"; clips: VideoClip[]; hidden: boolean; locked: boolean };

type Project = {
  id: string;
  version: number;           // server revision for autosave
  fps: Rational;             // { value: 30, timescale: 1 } or { value: 30000, timescale: 1001 }
  size: { width: number; height: number };
  tracks: Track[];           // index 0 is the bottom layer
  media: Record<string, MediaRef>;
};

// Which source time does timeline frame f show for this clip? Microseconds for WebCodecs.
function sourceTimeUs(clip: VideoClip, f: number, fps: Rational): number {
  const offsetFrames = f - clip.start;                       // 0 at the clip's first frame
  const inUs = (clip.sourceIn.value * 1_000_000) / clip.sourceIn.timescale;
  const offsetUs = (offsetFrames * fps.timescale * 1_000_000) / fps.value;
  const scaledUs = (offsetUs * clip.speed.value) / clip.speed.timescale;
  return Math.floor(inUs + scaledUs);
}
// The decoder then picks the source frame whose [pts, pts + duration) contains this time,
// which handles 24 fps sources on a 30 fps timeline and variable frame rate phone footage.

Notice what the model does not contain: no frames, no thumbnails, no waveform data. Those are derived data, rebuilt from the media whenever needed and cached locally. A 30 minute project with 50 clips is a JSON document of about 20 to 100 KB, which is why autosave can send all of it or a small patch every few seconds.

The preview engine: compositing, playback and scrubbing

The preview answers one question for any timeline frame f: what does the film look like here? It finds the clips that cover f, gets the right source frame for each, applies each clip's effects and transform, draws the layers bottom to top, adds text, and shows the result. During playback it does this 30 times a second, inside a 33 ms budget shared with everything else the page does.

The naive version draws with Canvas 2D: drawImage(frame, ...) per layer, ctx.filter for effects and fillText for titles. It breaks once effects get real: a LUT colour grade or a chroma key needs per-pixel math. In JavaScript that means getImageData, a loop over about 2 million pixels and putImageData, costing tens of milliseconds per layer per frame and copying 8 MB each way, where a GPU shader takes a millisecond or two (both estimates).

One frame, step by step

1. Resolve: find clips active at frame f. Each track's clips are sorted by start, so a binary search per track finds the clip in microseconds; a transition makes two clips active on one track.
2. Fetch: for each video clip, map f to source time with the shared function and ask that source's decoder for the frame. A hit in the frame cache returns at once; a miss starts or continues a decode.
3. Import: turn each VideoFrame into a texture (importExternalTexture in WebGPU, texImage2D in WebGL2) and apply the source's rotation from the container metadata.
4. Effects: run each clip's effect list as shader passes into an intermediate texture at preview resolution. Parameters come from keyframes evaluated at f.
5. Composite: draw layers from bottom track to top with their transforms and opacity; a cross-fade blends two layers by the transition's progress.
6. Overlay: draw text from textures that were rasterized once with Canvas 2D when the text or style changed, not every frame.
7. Present and release: the frame shows on the canvas. Then close every VideoFrame that is not kept in the cache, but only after the GPU work that samples it is submitted: an external texture made from a frame stops working once the frame is closed.

The frame cache around the playhead

Two caches make the preview feel instant. The source frame cache keeps a few decoded frames per active source, keyed by media id and presentation timestamp: during playback, the next 4 to 8 frames; when paused, the frame shown plus one on each side for frame stepping. It is bounded in bytes, not frames, so a 4K source gets fewer slots than a 720p one.

The composite cache keeps finished preview frames, at preview resolution, for the last range the user scrubbed over, so scrubbing back and forth over a cut costs nothing. Any edit that touches a time range evicts that range only.

Try the lab below. It adds up decode, upload, effects and composite time for a frame and compares the total with the 33 ms budget at 30 fps, and it shows the frame cache size in memory. Watch which settings break the budget first: it is almost always software decoding of 4K or too many simultaneous streams, not the GPU work.

Playback: the audio clock leads

Paused preview is about exactness. Playback is about keeping up: every 33 ms a new composite must be ready, with sound in sync. The naive loop is requestAnimationFrame, advance the playhead by the elapsed wall-clock time, render. It drifts, because wall-clock time and the audio device's clock tick at slightly different rates, and a slow frame pushes everything late.

The fix is the one the music-daw guide uses: the audio clock is the master. Audio plays through Web Audio, and its currentTime, corrected with getOutputTimestamp() or outputLatency for what the speakers play now, says where the film is. Web Audio is not available in workers, so the main thread posts pairs of audio time and performance.now() to the render worker, which extrapolates the current audio time on each display frame.

The worker converts that time to a timeline frame and draws it. If the frame is not ready, it shows the previous one again or skips ahead; audio never waits for video. A dropped video frame is barely visible, but drift is not: broadcast guidance (ITU-R BT.1359) finds sound about 45 ms early or 125 ms late already detectable.

Keeping decoders ahead

During playback each active source has a decoder running forward. Feeding is driven by the frame cache: when it drops below its low mark, the worker feeds more samples, respecting decodeQueueSize. Clips that start in the next second or so are pre-rolled: their decoder is configured and fed from the right keyframe before the cut arrives, so the next clip's first frame is ready on time.

A decoder that falls behind catches up by jumping to the next keyframe rather than decoding every late frame. When the device cannot keep up at all, the engine degrades in a fixed order: first halve the preview resolution (a quarter of the pixels for effects and compositing), then switch heavy sources to their proxies, then drop frames while keeping audio intact. A badge on the preview tells the user this affects preview quality only, not the export.

Scrubbing

Scrubbing (dragging the playhead) produces 30 to 60 new target frames a second, in random order. Decoding each exactly would queue far more work than the decoders can do. So the scrub path is layered: while the pointer moves, show the nearest thumbnail from the timeline strip (already in memory) or the nearest keyframe of the proxy, which decodes in one step.

Only the latest target matters, so each new target cancels the previous decode with reset(). When the pointer rests for about 100 ms, decode the exact frame. The user sees a picture on every pointer move and the exact frame a moment after stopping.

J, K and L give keyboard scrubbing, a convention from desktop editors: L plays forward and pressing it again speeds up, J plays backward, K pauses, and holding K while tapping J or L steps one frame. Reverse playback is the hardest case for decoders, because frames only decode forward. The engine decodes a whole GOP forward into the cache and then shows it backwards, which is one more reason to keep proxy GOPs short.

The timeline: ruler, strips, snapping and edits

The timeline is where users spend their time, and it must answer every drag within one display frame. It has a zoomable time ruler, one row per track, clips drawn as blocks with a strip of thumbnails (video) or a waveform (audio), a playhead, and handles on each clip's edges for trimming. Zoom runs from a whole 30 minute project on one screen to single frames several pixels wide, so the scale (pixels per frame) can vary by a factor of 1,000.

Thumbnail strips and waveforms

A thumbnail strip shows one small frame every N seconds, where N depends on zoom: a 160 by 90 pixel thumbnail every 10 seconds when zoomed out, every second when zoomed in. A thumbnail worker extracts them with its own decoder, at low priority, and stores them as ImageBitmap tiles keyed by media id, zoom bucket and time. Zoomed out, it decodes keyframes only, which costs one decode per thumbnail and no GOP walk.

Each RGBA thumbnail is 160 x 90 x 4 bytes, about 57 KB, so 2,000 cached thumbnails take about 115 MB. The memory cache is bounded, and tiles are also stored in IndexedDB as compressed JPEG blobs (a few KB each, estimate), so a reopened project draws its strips at once. Audio waveforms use the precomputed peaks from the music-daw guide.

Edits: ripple, overwrite, snapping

Every edit is a pure function from one project state to the next, wrapped as a command for undo and redo (see the undo-redo guide), and it is computed in frames, so results are exact. A trim can never go beyond the source: the start handle stops at source time 0 and the end handle stops at the source's duration.

Snapping collects candidate frames (clip edges on all tracks, the playhead, markers), and if the dragged edge is within 8 pixels of one at the current zoom, it jumps there. The threshold is in pixels, converted to frames on each pointer move, so snapping feels the same at every zoom.

Try the lab: trim, split and move clips on two tracks in ripple or overwrite mode, with snapping on or off. The project JSON updates on each edit, and the panel shows which source frame each layer needs at the playhead.

[{"id":"a","track":"V1","start":0,"duration":120,"sourceIn":30},
 {"id":"b","track":"V1","start":120,"duration":150,"sourceIn":0},
 {"id":"t","track":"V2","start":90,"duration":60,"sourceIn":null}]

The drag loop

While dragging a trim handle, the UI updates a draft state on every pointermove (throttled to one animation frame) and the preview shows the frame at the edge being trimmed, so the user sees exactly where the cut will land. Only on pointerup is the edit committed as one command, saved locally and queued for the server. That way a drag creates one undo step and one save, not 200.

Export: render, encode, mux and write

Export turns the project into an MP4. It is the same work as preview, done for every frame at full resolution, as fast as the device allows, and then compressed. A 5 minute project at 30 fps is 9,000 frames. Each must be composited exactly, encoded to H.264, interleaved with AAC audio into an MP4 container (that interleaving is called muxing), and written somewhere that can hold a file of hundreds of megabytes.

The naive approach records the preview: canvas.captureStream() into a MediaRecorder while the project plays. It runs only in real time (a 30 minute project takes 30 minutes), any dropped preview frame is baked into the file, the output format depends on the browser, and the tab must stay in the foreground the whole time.

The export loop

const encoder = new VideoEncoder({
  output: (chunk, meta) => muxer.addVideoChunk(chunk, meta),  // meta carries the avcC on the first chunk
  error: (e) => fail(e),
});
encoder.configure({
  codec: "avc1.640028",            // H.264 High profile, level 4.0: fits 1080p30
  width: 1920, height: 1080, framerate: 30,
  bitrate: 8_000_000,              // 8 Mbps
  hardwareAcceleration: "prefer-hardware",
  avc: { format: "avc" },          // MP4-style samples plus a description
});
for (let f = 0; f < totalFrames; f++) {
  if (signal.aborted) return cancel();
  await compositor.render(project, f);                    // same code path as preview
  const frame = new VideoFrame(canvas, { timestamp: Math.round((f * 1_000_000) / 30) });
  encoder.encode(frame, { keyFrame: f % 60 === 0 });      // a keyframe every 2 s
  frame.close();
  while (encoder.encodeQueueSize > 4) await nextDequeue(encoder);
  if (f % 30 === 0) postMessage({ type: "progress", done: f, total: totalFrames });
}
await encoder.flush();
await muxer.finalize();

Backpressure is what keeps memory flat. Without the encodeQueueSize check, the loop would render frames much faster than the encoder can take them, and thousands of 8 MB frames would pile up in the queue. With it, at most a handful of frames are alive at any time, whether the film is 1 minute or 60. The same applies to the source decoders feeding the compositor.

Where the file goes

A 10 minute 1080p export at 8 Mbps is about 600 MB. Holding it in an ArrayBuffer and making a Blob at the end doubles that and can crash a tab on a phone. Instead, the muxer writes to a stream. In desktop Chromium browsers, the File System Access API's showSaveFilePicker() returns a handle to a real file the user picks before the export starts, and its createWritable() stream receives the output.

Everywhere else, write to the Origin Private File System (OPFS), the site's own private storage. In a worker, createSyncAccessHandle() gives fast synchronous writes to an OPFS file. When the export ends, getFile() returns a disk-backed File that can be downloaded through an object URL without reading it into memory. Check navigator.storage.estimate() before starting, since OPFS shares the site's storage quota.

MP4 has one awkward detail. Its index (the moov box) lists every sample's size and position, so it can only be written once all samples are known. A plain file puts it at the end, which is fine for a download. For a file that should start playing while still downloading from a server, ask the muxer for a fast-start layout (index first), which costs memory or a known sample count up front, or a fragmented MP4, which writes small index pieces as it goes.

Progress is frames done over total; the estimate is the remaining frames times a moving average of recent frame times, so it settles after a few seconds. A cancel button aborts an AbortController, closes the encoder and decoders and deletes the partial file. An encoder error is retried once with a fresh encoder of the same configuration; the worker hands the muxer only complete GOPs, so it can resume at the start of the unfinished one. A second error ends in failed, with a server render on offer.

The export must not depend on requestAnimationFrame, which stops in background tabs. A worker loop is not tied to display frames and keeps going, though browsers, especially on phones, may still throttle or suspend hidden pages. A Screen Wake Lock (navigator.wakeLock.request('screen')) keeps the laptop from sleeping, but the browser releases it whenever the tab is hidden, so the page asks the user to keep the tab visible during a long export.

Importing and managing media

A user drops a 2 GB phone video on the editor. The old web answer is to upload it first and edit when the server is done. On a 20 Mbps upload link that takes about 13 minutes before the first edit, and the user leaves. A browser editor with WebCodecs can do better: read the file where it is, start editing within seconds, and upload in the background.

Keeping access to local files

A File from an input or a drop is only valid for the life of the page. After a reload it is gone. In desktop Chromium browsers, getAsFileSystemHandle() on a dropped item or showOpenFilePicker() returns a FileSystemFileHandle, which can be stored in IndexedDB and reopened later. On the next visit, requestPermission() asks the user to allow access again.

In other browsers, copy the file into OPFS in a worker, after checking navigator.storage.estimate() and asking for navigator.storage.persist() so the browser is less likely to evict it. If neither works, the media is marked offline and the editor shows a relink prompt; the timeline still opens because the project only holds references.

Checking that a file can be edited

1. Probe: demux the header to get the container, codec string, size, frame rate, rotation, duration and audio format. This reads a few hundred kilobytes, not the file.
2. Check: call VideoDecoder.isConfigSupported() with the exact codec string, coded size and description. H.264 is decodable almost everywhere; HEVC depends on the browser, the operating system and the hardware; ProRes and most camera RAW formats are not decodable with WebCodecs in current browsers.
3. Index: build the sample index and the first thumbnails in a worker; the clip is now editable.
4. Unsupported: upload the file, transcode it on the server to an H.264 working copy and a proxy, and show the clip as processing on the timeline. Its length and thumbnails come from the server when ready, and the user can keep arranging other clips.

Each media item has its own small state: probing, ready locally, uploading, uploaded, processing on the server, offline, failed. The media bin shows that state per file, and export checks it: a local export needs every source readable locally, a server export needs every source uploaded.

Architecture, data and the API

With the hard parts decided, the architecture is mostly about where each piece runs. The main thread must stay free for input, so anything that touches frames lives in a worker. The rule of thumb: the main thread owns the project document and the UI; workers own media; the server owns sharing, storage and heavy fallbacks.

Why these boundaries? The render worker is separate from the main thread because decode and composite work is bursty and must not be delayed by a React render of 300 clip elements. The media workers are separate from the render worker because building a proxy can take minutes, and playback must preempt it.

The export worker has its own decoders and compositor and works on a frozen snapshot of the project at a given version, so the user can keep editing while an export runs, and the export is not affected by those edits. Audio stays on Web Audio because only the audio rendering thread can deliver glitch-free sound and an accurate clock.

The main thread talks to workers with small messages, never frames: a project patch, a playhead position, a play or pause command. Frames and textures stay inside the render worker. Thumbnails cross to the main thread as ImageBitmap objects, which are transferred rather than copied.

type ToRender =
  | { type: "load"; project: Project }
  | { type: "patch"; version: number; ops: ProjectOp[] }
  | { type: "seek"; frame: number; exact: boolean }   // exact: false while scrubbing
  | { type: "play"; fromFrame: number; rate: 1 | 2 | -1 }
  | { type: "pause" }
  | { type: "clock"; audioTime: number; at: number }   // audio clock sample; at = performance.now()
  | { type: "quality"; previewScale: 1 | 0.5 | 0.25; useProxies: boolean };

type FromRender =
  | { type: "shown"; frame: number; exact: boolean }
  | { type: "stats"; dropped: number; decodeMs: number; compositeMs: number }
  | { type: "media-error"; mediaId: string; reason: "unsupported" | "decode" | "missing" };

The data flow figure follows one edit and one import end to end. Edits are tiny and go from the store to the render worker, IndexedDB and the server; media goes once from disk through the media worker to the server, in chunks. The server only sends back project versions, transcoded working copies and proxies.

Data, autosave and the API

The editor handles two kinds of data that differ in size by a factor of tens of thousands or more. The project document is tens of kilobytes and changes many times a minute. Media is gigabytes and never changes after import. Treat them separately: the document is saved often and versioned; media is uploaded once, addressed by a content id, and shared by every version that uses it.

Autosave works like the offline sync engine guide describes. Each committed command is written to IndexedDB at once (so a crash loses nothing) and added to an outbox. Every 2 seconds, or when the user pauses, the client sends the outbox to the server as a batch of operations with the version it was based on, and the server returns the new version. A save indicator in the header reads saved, saving or offline, and the beforeunload prompt appears only while the outbox is not empty.

Since we allow one editor at a time, the client takes an edit lease when the project opens and renews it every 30 seconds. If a second tab or device opens the project, it gets read-only mode and an offer to take over.

If a save comes back with a version conflict anyway (two devices edited while offline), the client keeps both: the server version stays as the main line, and the local edits become a copy named after the device. Merging two timelines automatically is out of scope; the Figma and Google Docs guides cover what it would take.

Versions are server snapshots taken every 10 minutes of editing and before every export, so the user can restore the cut they exported last week. They cost almost nothing because media is shared by content id.

GET   /projects/:id                -> 200 { project, version }
POST  /projects/:id/ops            { baseVersion: 41, ops: [...] }
                                   -> 200 { version: 42 }
                                   -> 409 { error: "version_conflict", serverVersion: 43 }
POST  /projects/:id/lease          -> 200 { leaseId, expiresAt } | 423 { holder: "Laptop" }
GET   /projects/:id/versions       -> 200 { items: [{ version, createdAt, reason }], next }

POST  /media                       { sha256, size, mime, probe: { codec, width, height, fps } }
                                   -> 200 { mediaId, uploaded: true }       (already on the server)
                                   -> 201 { mediaId, uploadUrl, chunkSize }  (resumable upload)
GET   /media/:id                   -> 200 { status: "processing" | "ready", workingUrl, proxyUrl }

POST  /renders                     { projectId, version: 42, preset: "1080p30" }
                                   -> 202 { renderId }
GET   /renders/:id                 -> 200 { status, progress: 0.37, url? }

The content hash in POST /media lets the server say it already has the file, so re-importing the same clip into a new project uploads nothing. crypto.subtle.digest() cannot hash a stream, so a worker feeds the file in slices to an incremental hasher (a WebAssembly SHA-256, for example). For a 2 GB file that takes several seconds (estimate), so it runs after the clip is already editable. Server renders report progress by polling or a server-sent event stream.

Performance, reliability and accessibility

Know the device before you promise anything

Capabilities vary more in video than in any other web app. At startup, and again when a project opens, the editor runs a short check: does VideoDecoder exist, is each source's config supported, is a WebGPU adapter available, does VideoEncoder.isConfigSupported accept the export preset with hardware acceleration, and how fast does a 2 second 1080p test clip decode?

The result picks the default preview scale, whether to build proxies, and whether the export button says export or render in the cloud. Load the heavy engine code (demuxer, compositor, muxer) only when a project opens, as the code-splitting lesson describes, so the project list stays fast.

• GPU loss: WebGPU reports device.lost and WebGL fires webglcontextlost after a driver reset or when the system reclaims memory. The compositor rebuilds its pipelines and textures from the project, which is easy because no state lives only on the GPU.
• Decoder errors: a corrupt sample or a hardware hiccup closes the decoder. Rebuild once, then fall back to software, then mark the clip and keep the rest of the timeline working.
• Memory: every cache has a byte budget, not an item count, and the budgets shrink when the project has many 4K sources. Close frames promptly and drop composite caches on visibilitychange when the tab is hidden.
• Observability: record dropped frames per minute, decode and composite times, seek latency, export frames per second and failures by codec string and GPU. These numbers tell you which devices need proxies by default.
• Security: media from other origins must be served with CORS, or the canvas and WebCodecs refuse to read its pixels. If you use multi-threaded WebAssembly, the page needs cross-origin isolation headers. Caption and title text is rendered as text, never as HTML.

An accessible timeline

A timeline is a dense grid of draggable boxes, which is the hardest kind of UI to make accessible. Every action needs a keyboard path (see the keyboard-navigation and aria-patterns lessons). Each track is one tab stop; arrow keys move between its clips with a roving tabindex, and up and down move between tracks. Each clip is a focusable element with a name such as Interview.mp4, video track 1, from 0:12 to 0:45.

Keys mirror the mouse: Alt with Left or Right nudges the selected clip by one frame, and adding Shift nudges it by several; I and O set in and out points; S splits at the playhead; Delete removes, and Shift with Delete ripple-deletes; J, K and L play, pause and shuttle.

The playhead is a slider (role="slider") whose aria-valuetext is a readable timecode, not a frame number. A polite live region announces the result of each edit in words (Trimmed end of Interview.mp4 to 0:45; 3 clips moved 2 seconds earlier) and the playhead position when playback stops, never continuously while playing. Respect reduced motion by not animating clip moves.

Treat captions as a product feature: a captions track whose cues the user can type, import from SRT or WebVTT, or generate with speech to text. Export then burns them into the picture or saves them as a separate WebVTT file.

Running it in the interview

In 60 minutes you cannot cover everything above, and you should not try. Show that you know why video is different, make the three or four decisions that matter with numbers, and draw the threads. A plan that works:

1. Minutes 0 to 7, requirements: codecs in, length of projects, export on device or server, devices, offline. Write the frame budget and the memory math on the board.
2. Minutes 7 to 15, the model: tracks of clips referencing media by id, with start and duration in frames and a rational source in-point. Say non-destructive and show the mapping function.
3. Minutes 15 to 30, decoding and preview: WebCodecs plus a demuxer versus the video element versus ffmpeg.wasm; keyframes and seeking; the compositor on the GPU in a worker; the audio clock; proxies.
4. Minutes 30 to 40, export: render, encode with backpressure, mux, stream to disk; the server fallback.
5. Minutes 40 to 50, architecture: draw main thread, render worker, media workers, export worker, storage and server, with the messages between them.
6. Minutes 50 to 60, follow-ups: whatever the interviewer picks.

Common mistakes: decoding all frames up front, float seconds for edits, drawing everything on the main thread, recording the canvas with MediaRecorder as the export, and forgetting that a long export needs progress, cancel and a plan for a hidden tab.