Uploading files
How a file gets from the user's device to your storage: through your server or straight to object storage with a pre-signed URL, in resumable chunks with honest progress, checked on both sides, and served back without opening a security hole.
Why uploads are harder than they look
Picture a video app for teachers. A teacher records a 40 minute lesson on her phone, a 1.2 GB file, and taps Upload on a train. At 70 percent the train enters a tunnel, the connection drops for eight seconds, and the bar jumps back to zero with "Upload failed".
At home she tries again. The app's API server reads the whole body into memory, runs out at about 900 MB, and crashes, taking every other user on that server with it. Nothing here is exotic: it is what happens when a team treats a file upload like any other JSON request.
Most API calls in this module send a few kilobytes and finish in under a second. A file upload breaks every one of those assumptions. It can be gigabytes, it can take minutes, it travels over the slowest link the user has (on home and mobile connections, upload bandwidth is often a fifth of download bandwidth or less), and its content is chosen by a stranger.
The five problems every upload design must answer
- Where do the bytes go? Through your API servers, or straight from the browser to object storage such as Amazon S3 or Google Cloud Storage?
- What happens when the network drops? Start again from zero, or continue from the last byte the server confirmed?
- What does the user see while waiting? A real progress bar needs real numbers, and not every browser API gives them.
- Is the file what it claims to be? A file named
cat.jpgcan hold an HTML page, a script, or malware. The browser can check a little; the server must check everything again. - How do you serve it back safely? A user upload served from your main domain can run script in your users' sessions.
A few words we will use
- Object storage: a service that stores files (objects) under keys, such as S3, Google Cloud Storage (GCS), Azure Blob Storage, or Cloudflare R2. It is cheap per gigabyte and built to take huge numbers of large writes.
- Pre-signed URL: a URL that carries a time-limited signature, letting whoever holds it do one specific thing (for example, PUT one object) without having storage credentials.
- Chunk or part: a slice of the file, sent as its own request, so a failure only costs that slice.
- Quarantine: a storage area for files that are uploaded but not yet checked. Nothing in it is shown to other users.
Check your understanding · Exercise 1
Which of these make a 1 GB upload different from a normal JSON API call? Select all that apply.
The simple way: multipart through your server
The oldest upload on the web is an HTML form with an input of type file and enctype="multipart/form-data". In JavaScript you do the same with FormData: append the file and any other fields, and hand the object to fetch as the body. The browser builds the request for you.
async function uploadAvatar(file: File) {
const form = new FormData();
form.append("avatar", file); // name, type, and bytes go together
form.append("crop", JSON.stringify({ x: 0, y: 0, size: 512 }));
// No Content-Type header: the browser sets it, with the boundary.
const res = await fetch("/api/me/avatar", { method: "POST", body: form });
if (!res.ok) throw await toApiError(res);
return (await res.json()) as { avatarUrl: string };
}
toApiError is the error helper from the API contracts lesson: uploads fail in the same shapes as other calls.What a multipart body looks like
multipart/form-data packs several fields into one body. The browser picks a random boundary, a string that does not appear in the data, and puts it in the Content-Type header. Each part starts with two dashes and the boundary, then its own small headers, a blank line, and its value; the body ends with the boundary and two more dashes.
-- and the boundary, has its own headers, then its value. The filename and the part's Content-Type come from the client, so the server treats them as hints.Two details catch people. First, never set Content-Type: multipart/form-data yourself in fetch: your header has no boundary, so the server cannot find where the parts start and fails with a parse error. Second, the filename and the part's Content-Type are written by the client. They are hints, never facts: the browser guesses the type from the file extension, and an attacker can write anything there.
On the server: stream, don't buffer
A server can buffer a multipart upload: read the whole body into memory (or a temp file), then parse it. Or it can stream it: parse the parts as bytes arrive and pipe the file part on to disk or object storage, holding only a small window in memory.
Buffering is what many frameworks do by default, and it is how the video app in chapter 1 crashed: ten users each uploading 500 MB need 5 GB of memory at once. A streaming parser such as busboy in Node.js holds only a small buffer per upload, whatever the file size. If you send files through your servers at all, stream them.
Limits pile up in the path
Even a streaming server sits behind other machines, and each has its own limits. A large upload has to pass all of them:
- Reverse proxy body size. nginx rejects bodies over 1 MB by default with status
413(theclient_max_body_sizesetting). Teams often find out when someone first uploads a 2 MB photo. - Timeouts. Load balancers, proxies, and serverless platforms close requests that run too long, often 30 to 60 seconds for idle connections, and serverless functions also cap the request body at a few megabytes. A 500 MB file on a 10 Mbps uplink needs about 7 minutes.
- Worker capacity. Each slow upload holds a connection and often a worker for minutes. Fifty slow uploads can tie up a server that normally handles thousands of quick requests per second.
So multipart through your server is the right answer for small files you handle right away: an avatar under 5 MB, a small CSV import, a receipt PDF. Past a few megabytes, move the bytes out of your server's path.
Check your understanding · Exercise 2
A developer writes fetch("/api/upload", { method: "POST", headers: { "Content-Type": "multipart/form-data" }, body: form }) and the server answers 400. What is true? Select all that apply.
Direct to storage with pre-signed URLs
Object storage is built for the traffic your API servers hate: many large, slow writes. So for anything bigger than a few megabytes, the standard design lets the browser send the bytes straight to storage, while your API stays in charge of who may upload what.
The browser cannot hold your storage credentials: anyone could read them from the page. Instead your API signs a pre-signed URL: an ordinary HTTPS URL to the bucket whose query parameters name the action, an expiry time, and a signature made with your secret key. Storage lets the request through only if the signature matches exactly.
The three-step flow
- Ask. The browser sends
POST /uploadswith the file's name, size, and type. The API checks the user is logged in, has quota left, and may upload this kind of file, then creates an upload record with statuspendingand picks the storage key itself, such asquarantine/u_8f3k2/original. - Send. The API returns the signed URL, valid for a few minutes. The browser sends the file to that URL with
PUT. Your servers never see the bytes. - Confirm. When storage answers
200, the browser callsPOST /uploads/u_8f3k2/complete. The API checks the object exists (aHEADrequest to storage) and moves the record to processing. Many teams also subscribe to storage events (S3 Event Notifications, GCS Pub/Sub notifications), so the backend learns about the object even if the tab closes first.
S3: pre-signed PUT or POST policy
S3 offers two kinds of signed browser upload, and the difference matters for limits.
| Question | Pre-signed PUT | Pre-signed POST policy |
|---|---|---|
| Request the browser sends | PUT with the raw file as the body | POST with a multipart/form-data body: policy fields first, file last |
| What the signature can pin | Key, expiry, and any headers you choose to sign, such as Content-Type | A policy document of conditions: key or key prefix, expiry, Content-Type prefix, and a size range |
| Size limit | No range. You can sign an exact Content-Length if you know it | content-length-range gives a min and max in bytes, enforced by S3 |
| Largest single upload | 5 GB | 5 GB |
| Progress and resuming | Easy progress with XHR; resume means multipart upload | Easy progress with XHR; no resume |
Use the POST policy when storage must enforce a size range; use PUT for the simplest client code and for multipart uploads. Either way, keep the expiry to minutes, not the 7 day maximum S3 allows.
Google Cloud Storage, Azure, and R2
Other clouds use the same idea under different names. Google Cloud Storage has V4 signed URLs (also up to 7 days) and POST policy documents. For large files it has resumable uploads: a first signed request starts a session and returns a session URI, and the client sends the file there in pieces, asking where to continue after a failure.
Azure Blob Storage calls its signed URLs shared access signatures (SAS). Cloudflare R2 speaks the S3 API, so S3 pre-signing code works with a different endpoint.
The bucket needs CORS
The upload is now a cross-origin request from app.example.com to my-bucket.s3.amazonaws.com, so for a PUT the browser first sends a preflight OPTIONS request. Add a CORS rule to the bucket that allows your origin, the PUT or POST method, and the headers you send, such as Content-Type. For multipart uploads, also list ETag in ExposeHeaders: the client needs each part's ETag to finish, and JavaScript cannot read a response header that is not exposed.
Decision Where should the bytes of a user upload go?
What we pick: Multipart through your API for small files you process at once (under about 5 MB). A pre-signed single upload for medium files on decent connections. Resumable chunks for anything over about 100 MB, or any upload from a phone that might take more than a minute. Large products often run two of these side by side, picked by file size.
Check your understanding · Exercise 3
Your API returns pre-signed S3 PUT URLs for profile photos. Which statements are true? Select all that apply.
Chunked and resumable uploads
A single request is all or nothing: if a 2 GB upload fails at 1.8 GB, the user starts again from zero. On a phone, a long upload almost always meets a drop: a tunnel, a lift, a switch from Wi-Fi to cellular, or the OS pausing a background tab.
The fix has two parts. Chunking splits the file into slices and sends each as its own request, so a failure costs one slice. Resuming lets the client ask the server "how much do you already have?" after any interruption, even a page reload, and continue from exactly there.
Slicing a file in the browser
A File is a Blob, and blob.slice(start, end) returns a new Blob for that byte range without reading anything into memory. The browser reads the bytes from disk as it sends each slice, so a 4 GB file does not need 4 GB of RAM.
const PART = 8 * 1024 * 1024; // 8 MiB
async function uploadParts(file: File, urls: string[], done: Set<number>) {
const etags: { PartNumber: number; ETag: string }[] = [];
for (let i = 0; i < urls.length; i++) {
const partNumber = i + 1;
if (done.has(partNumber)) continue; // already on the server
const body = file.slice(i * PART, (i + 1) * PART);
const res = await putWithRetry(urls[i], body); // backoff + jitter
const etag = res.headers.get("ETag"); // needs ExposeHeaders in CORS
if (!etag) throw new Error("Missing ETag: check bucket CORS");
etags.push({ PartNumber: partNumber, ETag: etag });
done.add(partNumber);
}
return etags;
}
S3 multipart upload
S3's version has three calls, all of which your API can make or pre-sign. CreateMultipartUpload returns an UploadId. The browser then sends each part with UploadPart (a PUT to a URL pre-signed for that part number) and keeps the ETag that S3 returns for it. Finally CompleteMultipartUpload sends the list of part numbers and ETags, and S3 joins them into one object.
CompleteMultipartUpload with every part number and ETag.- Part size. Every part except the last must be at least 5 MiB. A part can be up to 5 GiB.
- Part count. Part numbers run from 1 to 10,000, so the part size caps the file size: 10,000 parts of 8 MiB is about 78 GiB. A 200 GiB file needs parts of at least 21 MiB.
- Resuming.
ListPartswith theUploadIdreturns the parts S3 already has. After a reload, the client asks your API for that list and only sends the rest. - Cleanup. Parts of an unfinished upload are stored, and billed, until you complete or abort it. Add a lifecycle rule with
AbortIncompleteMultipartUploadafter a few days, or abandoned uploads quietly cost money forever.
tus: an open protocol for resuming
tus is an open, HTTP-based protocol for resumable uploads. Vimeo, Cloudflare Stream, and Supabase Storage accept it, and tus-js-client and Uppy implement the browser side. Instead of numbered parts it uses one growing upload and a byte offset: how many bytes the server has stored so far.
POSTwithUpload-Length: 104857600creates the upload. The server answers201 Createdwith aLocationURL for it.PATCHto that URL withUpload-Offset: 0,Content-Type: application/offset+octet-stream, and the bytes. The server answers204with the newUpload-Offset.- After a failure,
HEADon the URL returns the server'sUpload-Offset. The client slices the file from there and sends the nextPATCH. If the offsets disagree, the server answers409 Conflictinstead of writing bytes in the wrong place.
Every tus request except OPTIONS carries Tus-Resumable: 1.0.0, and optional extensions add checksums, expiry, and parallel uploads. The IETF HTTP working group is also drafting a resumable upload standard that grew out of tus.
Resuming after a reload
Chunks handle a drop while the page is open. Surviving a closed tab needs two more things. First, save the upload's identity (the UploadId or tus URL) in localStorage or IndexedDB, keyed by a fingerprint of the file such as name, size, and lastModified; tus-js-client does this for you.
Second, get the bytes back. After a reload the File object is gone, and a page cannot reopen a file the user picked earlier. Either ask the user to pick it again (then match the fingerprint and continue), or copy the file into the origin private file system (OPFS) or IndexedDB before uploading, which costs disk space but lets the upload continue on its own.
Choosing a chunk size
Smaller chunks lose less on each drop, but every chunk is another request with its own round trip, headers, and signature. Larger chunks are efficient on a stable link but lose more when it breaks. The lab below lets you try both: pick a chunk size, drop the network, and resume.
Ready to upload a 120 MB file.
In practice, 8 to 16 MiB parts work well for S3, with 3 or 4 in flight. More parallel parts rarely help: they compete for the same uplink and slow the rest of the page.
Check your understanding · Exercise 4
You upload a 120 GiB video with S3 multipart upload using 8 MiB parts. What happens? Select all that apply.
Progress the user can trust
A user who sees nothing move for ten seconds assumes the upload is stuck and retries, or leaves. A good progress display answers three questions: is it moving, how far along is it, and how long is left? Honest numbers depend on which browser API sends the bytes.
XMLHttpRequest has upload events
The old XMLHttpRequest is still the standard way to get upload progress. Its upload property is an event target that fires loadstart, then progress repeatedly (the spec says about every 50 ms), then load, error, abort, or timeout, then loadend. Each progress event carries loaded and total in bytes, plus lengthComputable, which is true when the total is known.
function putWithProgress(url: string, body: Blob, onBytes: (sent: number) => void, signal: AbortSignal) {
return new Promise<string | null>((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("PUT", url);
xhr.upload.onprogress = (e) => onBytes(e.loaded);
xhr.onload = () =>
xhr.status < 300 ? resolve(xhr.getResponseHeader("ETag")) : reject(new Error(`HTTP ${xhr.status}`));
xhr.onerror = () => reject(new Error("network"));
xhr.onabort = () => reject(new DOMException("Aborted", "AbortError"));
signal.addEventListener("abort", () => xhr.abort());
xhr.send(body);
});
}
AbortSignal lets the rest of the app treat it like fetch.upload.onprogress reaches 100 percent when the last byte leaves the browser. Storage still has to answer, so show "finishing" until the response arrives.Notice what loaded measures: bytes handed to the network, not bytes safely stored. The bar can reach 100 percent before storage has answered. Show 100 percent only after the load event with a success status, and show "finishing" in between.
One more quirk: adding a listener to xhr.upload makes a cross-origin request always send a CORS preflight, even when it would otherwise count as simple. For a pre-signed PUT that is already the case, but it explains a surprise OPTIONS request on a form-style POST.
fetch has no upload progress
fetch gives download progress through response.body, but no upload progress. The only partial workaround is a streaming body: pass a ReadableStream with duplex: "half" and count bytes as the browser pulls them. Today only Chromium-based browsers support this, only over HTTP/2 or HTTP/3, and the count runs ahead of what actually went over the wire.
So for a progress bar on a single large request, use XHR. With chunked uploads, fetch works too if you count the bytes of finished chunks, but the bar then moves one chunk at a time: with 8 MiB chunks on a 2 Mbps uplink it sits still for about 30 seconds. XHR per chunk adds smooth movement inside each chunk.
| Way to send | Progress inside a request | Cancel | Notes |
|---|---|---|---|
XMLHttpRequest | Yes, upload.onprogress | xhr.abort() | All browsers; the safe default for big single requests |
fetch with a Blob body | No | AbortController | Use with chunks and count finished chunks |
fetch with a stream body | Approximate | AbortController | Chromium only, HTTP/2 or newer, duplex: "half" required |
| Upload libraries (Uppy, tus-js-client) | Yes | Yes | Use XHR inside and handle chunks and retries |
Time left without the jitter
Upload speed jumps around, so a raw estimate flips between "2 minutes" and "40 minutes". Smooth it with an exponential moving average, for example speed = 0.2 * latest + 0.8 * previous, wait about 3 seconds before showing any estimate, and round to friendly units: "about 3 minutes left". For several files, base both the bar and the estimate on total bytes, not on files finished.
Accessible progress
- Use a
progresselement withmaxandvalue, orrole="progressbar"witharia-valuenow,aria-valuemin, andaria-valuemax, and give it a name such as "Uploading lesson.mp4". - Announce milestones in a polite live region (25, 50, 75 percent, done, failed), not every percent: a screen reader reading every change is unusable.
- Always offer Cancel, and Pause and Resume for big files. Make failures say what to do: "Connection lost. We'll keep trying" beats "Error".
- Warn with a
beforeunloadhandler while an upload is in progress and cannot resume, so a stray tab close does not lose ten minutes of work.
Check your understanding · Exercise 5
Which statements about upload progress are true? Select all that apply.
Validation on both sides
Validation happens twice, for two reasons. In the browser it is kindness: tell the user in 50 ms that their 3 GB file is too big, not after a 10 minute upload. On the server it is security: the only check an attacker cannot skip. Browser checks copy the server's rules; they never replace them.
What the browser can check
- The picker filter.
accept="image/png,image/jpeg,.pdf"on the file input shows only matching files in the picker. It is a hint: the user can switch the picker to "All files", and drag and drop ignores it entirely. - Size.
file.sizeis exact and free. Check it against the same limit the server enforces, and say the limit in the message: "This video is 2.4 GB. The limit is 2 GB." - Declared type.
file.typeis a guess the browser makes from the file extension. Renamesetup.exetocat.jpgandfile.typesaysimage/jpeg. For unknown extensions it is an empty string. - Real type, by sniffing. Read the first bytes with
file.slice(0, 16).arrayBuffer()and compare them with known magic numbers: the fixed bytes most formats start with. This catches honest mistakes, like a PDF renamed to.jpg, before any upload. - Image dimensions.
createImageBitmap(file)giveswidthandheight, so you can reject an image that is too small or too large before sending it.
const SIGNATURES: { type: string; offset: number; bytes: number[] }[] = [
{ type: "image/jpeg", offset: 0, bytes: [0xff, 0xd8, 0xff] },
{ type: "image/png", offset: 0, bytes: [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a] },
{ type: "image/gif", offset: 0, bytes: [0x47, 0x49, 0x46, 0x38] }, // GIF8
{ type: "image/webp", offset: 8, bytes: [0x57, 0x45, 0x42, 0x50] }, // WEBP after RIFF....
{ type: "application/pdf", offset: 0, bytes: [0x25, 0x50, 0x44, 0x46, 0x2d] }, // %PDF-
];
async function sniff(file: File): Promise<string | null> {
const head = new Uint8Array(await file.slice(0, 16).arrayBuffer());
const hit = SIGNATURES.find((s) => s.bytes.every((b, i) => head[s.offset + i] === b));
return hit?.type ?? null;
}
Try it below. Each sample has a name, the type the browser would guess from it, and its real first bytes. Notice which checks catch which tricks, and which trick gets past the browser entirely.
file.type says image/jpeg. Really: a real JPEG photo.
What the server must check again
The server repeats every rule, using facts it controls rather than anything the client said. With direct uploads, the "server" check splits in two: limits the storage signature enforces while the bytes land, and checks a worker runs on the stored object before it is accepted.
- Size where the bytes land: a POST policy's
content-length-range, a signed exactContent-Length, or a byte counter in your upload server that aborts at the limit. A check after the upload is too late to stop the cost. - Real type from content: sniff the stored bytes (a library such as libmagic, or the image decoder itself) and compare with your allow list. Ignore the client's
Content-Typeand file name. - Decode and re-encode images. Decoding proves the file is an image. Re-encoding to a fresh JPEG, WebP, or AVIF drops anything hidden in it: metadata, trailing data, and polyglot payloads (one file that is valid as both an image and, say, HTML).
- Bombs. Read image dimensions from the header before decoding, and reject anything over a pixel budget (a tiny PNG can claim 50,000 by 50,000 pixels and need 10 GB to decode). For archives, cap the number of entries and the total expanded size: a 42 KB zip can expand to petabytes.
- File names. Never use the user's name as a path or key. Keep it as display data, cut to a sane length, and strip control characters, path separators, and the right-to-left override character, which can make a name ending in
gpj.exedisplay asexe.jpg. - Active formats. SVG, HTML, and XML can carry scripts. Either refuse them, convert SVG to a raster image, or serve them only in the safe way chapter 8 describes.
| Check | Browser | Server or worker |
|---|---|---|
| Size | file.size, before upload | Signed policy or byte counter, while bytes land |
| Type | accept filter, file.type, sniff first bytes | Sniff stored bytes; decode with a real parser |
| Image pixels | createImageBitmap width and height | Read header first, then decode within a budget |
| Hidden content | Cannot check | Re-encode; virus scan |
| Name | Show it, trimmed | Sanitize; never a path; random key |
| Who may upload | Hide the button | Auth and quota check before signing |
Return rejections in your API's usual error format (see the API contracts lesson): 413 for too large, 415 for a refused type, 422 for anything else, so the UI can show the reason next to the file.
Check your understanding · Exercise 6
A user picks a file named photo.jpg. The browser reports file.type === "image/jpeg". What do you actually know? Select all that apply.
Shrinking images before they leave the device
A phone photo is usually 12 megapixels (4032 by 3024), up to 48 in high resolution modes, saved as a 3 to 8 MB JPEG or HEIC file. If your app never shows it wider than 2048 pixels, most of those bytes are wasted, and on a 2 Mbps mobile uplink each one costs time and data.
Resizing in the browser first is often the biggest single speed-up an upload feature can get. A 4.5 MB photo scaled to 2048 pixels and saved as JPEG at quality 0.85 typically lands between 300 and 600 KB: about ten times fewer bytes, and ten times less waiting.
The steps
- Decode the file with
createImageBitmap(file), which also works inside a Web Worker. - Scale by drawing the bitmap onto a smaller canvas. Keep the aspect ratio, and never scale up.
- Encode with
canvas.toBlob(callback, "image/jpeg", 0.85), or in a worker withOffscreenCanvasandconvertToBlob({ type, quality }). - Upload the new
Blob, giving it a file name with the right extension.
// Runs in a Web Worker so 40 photos don't freeze the page.
async function shrink(file: File, maxSide = 2048): Promise<Blob> {
const bitmap = await createImageBitmap(file); // applies EXIF orientation
const scale = Math.min(1, maxSide / Math.max(bitmap.width, bitmap.height));
const w = Math.round(bitmap.width * scale);
const h = Math.round(bitmap.height * scale);
const canvas = new OffscreenCanvas(w, h);
const ctx = canvas.getContext("2d");
if (!ctx) throw new Error("No 2D context");
ctx.imageSmoothingQuality = "high";
ctx.drawImage(bitmap, 0, 0, w, h);
bitmap.close(); // free the decoded pixels now, not at GC time
return canvas.convertToBlob({ type: "image/jpeg", quality: 0.85 });
}
blob.type before naming the file.EXIF orientation
A phone camera stores pixels the way the sensor sees them and writes a separate EXIF Orientation tag (values 1 to 8) saying how to rotate them for display. A portrait photo is often stored as a landscape image with a "rotate 90 degrees" tag.
Since about 2020, all major browsers apply that tag by default, both for an img (CSS image-orientation: from-image) and when drawing to a canvas, so the pixels you draw are already upright. The classic bug is old code, written when browsers ignored the tag, that reads EXIF and rotates by hand: in a modern browser it rotates twice. Test with real portrait photos from an iPhone and an Android phone.
Traps
- Memory. Decoded pixels take 4 bytes each: a 12 megapixel photo is 48 MB in memory, a 48 megapixel one about 192 MB. Process photos one or two at a time, call
bitmap.close(), and expect low-end phones to fail on huge images. iOS Safari has also limited canvas area in the past (about 16.7 million pixels on many devices), so test the largest images you accept. - HEIC. iPhones shoot HEIC by default. iOS usually hands web pages a JPEG copy when the user picks a photo, but desktop Chrome and Firefox cannot decode HEIC at all. If
createImageBitmapfails, upload the original and let the server convert it. - Formats with transparency. JPEG has no alpha channel, so a PNG logo with a transparent background turns black or white. Keep PNG or use WebP for those.
- Originals that matter. For a print shop, a photo archive, or medical images, the user wants the original. Resize only when the product shows a smaller version anyway.
Re-encoding also drops EXIF metadata, including GPS location, a real privacy win for photos posted publicly. The server should still strip metadata itself, because not every client resizes and an attacker never will. The server, or an image CDN as in the image optimization lesson, still makes the exact sizes and formats the app displays.
Check your understanding · Exercise 7
An app resizes photos in the browser before upload, with resize code written in 2018. Users report that some portrait photos arrive sideways. What is the likely cause and fix? Select all that apply.
After the upload: scanning, processing, and serving safely
When storage answers 200, the file is uploaded but not yet trusted. Before anyone else sees it, it must be checked and usually processed: scanned for malware, re-encoded, cut into thumbnails, or transcoded. This work is slow and can fail, so it runs in the background, never inside the upload request.
Quarantine first, then promote
New files land in a quarantine bucket (or prefix) that no user can download from. Each new object triggers a storage event (for example an S3 Event Notification into a queue), and a worker runs the checks from chapter 6 and a virus scan. Clean files are copied to the real bucket and the database record moves to ready. Bad files are deleted, and the record moves to rejected with a reason.
- The scanner. ClamAV is the common open-source choice, run as a worker that pulls from the queue. Clouds also sell managed scanning, such as Amazon GuardDuty Malware Protection for S3 and Microsoft Defender for Storage. Keep the signature database updated: a scanner with old signatures misses new malware.
- Hashes. Compute a SHA-256 of each file. It lets you skip scanning a file you have already cleared, block a file that was reported once, and store one copy of identical uploads.
- Scanner failures. A scan can time out or crash on a strange file. Retry a few times, then keep the file in quarantine and alert a human; never treat "scan failed" as "clean".
The file is a small state machine
The database record for each upload moves through a few states, and the UI shows each one. This is what makes the system honest: "uploaded" and "ready" are different words because they are different facts.
The client learns that a file moved from processing to ready by polling GET /uploads/:id every few seconds, or through a push channel the app already has (server-sent events or a WebSocket). A chat attachment can show a local preview at once, as an optimistic update, and swap in the real URL when the file is ready.
Serving uploads without opening an XSS hole
Here is the attack. A user uploads invoice.html, or an SVG with a script element inside, and shares the link. If you serve it from app.example.com with a type the browser renders, the script runs on your origin: it can read the victim's page data and call your API with their session. This is stored XSS, which the XSS lesson covers.
- A separate site. Serve user files from a different registrable domain, such as
example-usercontent.com, not a subdomain of your app. Google usesgoogleusercontent.comand GitHub usesgithubusercontent.comfor exactly this reason. A script that does run there has no cookies and no access to your app's origin. - Your own
Content-Type. Set it from the type the server detected, never from what the client claimed, and sendX-Content-Type-Options: nosniffso the browser does not guess a more dangerous type. Content-Disposition: attachmentfor anything that is not an allow-listed inline type (images, video, PDF if you trust the viewer). The browser then downloads it instead of rendering it. Put the cleaned file name in it, withfilename*=UTF-8''...for non-ASCII names.- A strict CSP on user content:
Content-Security-Policy: sandbox; default-src 'none'stops scripts even if a file is rendered. - Private files stay private. Keep the bucket private, put a CDN in front, and give allowed users short-lived signed GET URLs, as on the upload side.
HTTP/2 200
content-type: application/pdf
content-disposition: attachment; filename="invoice.pdf"; filename*=UTF-8''invoice.pdf
x-content-type-options: nosniff
content-security-policy: sandbox; default-src 'none'
cache-control: private, max-age=300
Check your understanding · Exercise 8
Which measures stop an uploaded HTML or SVG file from running script against your app's users? Select all that apply.
Limits, quotas, and what breaks in production
Uploads are the easiest way to cost a company money by accident, or on purpose: one script uploading 1 GB every second fills 86 TB in a day. Decide limits with the product team, and enforce each one at the earliest point that cannot be skipped.
The limits worth having
| Limit | Example | Where it is enforced |
|---|---|---|
| Size per file | Avatar 5 MB, video 4 GB | Signed policy or upload server; repeated in the UI |
| Types per feature | Avatars: JPEG, PNG, WebP | Worker sniffs and decodes; UI accept as a hint |
| Storage quota per user or team | 15 GB on the free plan | API checks before signing; settles on complete |
| Rate | 100 upload starts per user per hour | API rate limiter on POST /uploads |
| Pending uploads | At most 20 unfinished per user | API refuses new slots; cleanup job expires old ones |
| Signature lifetime | 5 to 15 minutes | Storage rejects expired URLs |
Quota has a timing problem: at signing time you only know the size the client claims. A good pattern is to reserve the claimed size when you sign, so ten parallel uploads cannot each squeeze under the quota, then settle using the real object size when the upload completes, and release the reservation if it expires.
Failure modes you will meet
| What you see | Usual cause | Fix |
|---|---|---|
403 SignatureDoesNotMatch on the PUT | The browser sent a Content-Type different from the one that was signed | Send exactly the signed headers; sign the type the client will send |
403 saying the request has expired | Slow start, a retry with an old URL, or a device clock far off | Short-lived URLs, fetched again on each retry |
| CORS error in the console, nothing at the API | The bucket has no CORS rule for this origin or method | Add a rule for the origin, method, and request headers |
| Multipart complete fails: missing ETags | ETag is not in the bucket's exposed headers | Add ETag to ExposeHeaders |
413 before any app log | Proxy body limit, such as nginx's 1 MB default | Raise it for the upload route, or go direct to storage |
| Two records for one file | The user double-clicked, or a retry created a new slot | Idempotency key on POST /uploads |
| Storage bill grows, nothing visible | Abandoned multipart parts and orphaned quarantine objects | Lifecycle rules plus a job that expires pending records |
| Mobile uploads stall when the app is in the background | The OS suspends the tab | Resumable protocol; resume on visibilitychange; tell the user to keep the tab open |
The double-record bug ties back to the retries lesson. POST /uploads creates something, so a retry after a timeout can create a second slot and a second file. Send an Idempotency-Key generated once per picked file, and have the API return the existing slot when it sees the key again. Chunk retries are already safe; only the creation step needs this care.
In real products
- Google Drive and YouTube APIs offer a resumable upload type: one request starts a session and returns a session URL, the client sends the file in chunks, and after a drop it asks how many bytes the server has.
- Dropbox has upload sessions: start, append chunks with an offset, then finish with the path. Same idea, different names.
- Uppy, from the team behind tus, is a widely used browser upload UI that can send to tus servers, to S3 with pre-signed multipart URLs, or as plain XHR multipart, with progress, retries, and pause built in.
Check your understanding · Exercise 9
A user on a free plan with a 15 GB quota and 1 GB used starts ten 2 GB uploads at once. What should the API do? Select all that apply.
Uploads in a system design interview
Uploads come up in many frontend design questions: a photo feed, chat attachments, a Drive or Dropbox clone, a video course platform, an avatar editor. Raise the topic as soon as the requirements mention files, and ask two questions first: how big can files get, and who will see them?
Walk the interviewer through this tree out loud. Most products need two branches: small files through a simple path, large ones through a resumable one. Saying where the line sits, and why, shows you thought about the users' networks and not only the happy path.
What to draw
- The client pipeline: pick, validate, (resize), request a slot, upload chunks, confirm. One box per step.
- The three-party sequence: browser, your API, storage. Make it clear the bytes skip your API.
- The background side: quarantine bucket, event queue, scanner and processing worker, clean bucket, CDN on a separate domain.
- The file record's states:
pending,uploading,processing,ready,rejected, and how the UI learns about each.
How deep to go by level
| Level | What a strong answer covers |
|---|---|
| Junior | FormData upload with a progress bar from XHR, size and type checks in the UI, and the knowledge that the server must check again |
| Mid-level | Pre-signed direct uploads with a completion step, chunked uploads with retries, image resizing on the client, and accessible progress with cancel |
| Senior | Resumable protocol choice (S3 multipart, tus, GCS resumable) with part sizing, quarantine and scanning, serving from a separate domain with safe headers, quotas with reservations, idempotent slot creation, and cleanup of abandoned uploads |
| Staff | Cost and abuse model (egress, storage growth, rate limits), regional storage and data residency, deduplication by hash, and how the upload state machine fits the app's offline and sync story |
Sentences that score
- "Small avatars go multipart through our API; anything over 5 MB goes straight to object storage with a pre-signed URL, so our servers never carry the bytes."
- "For videos I'd use resumable chunks, 8 to 16 MiB, three in flight, with backoff per chunk, and store the upload ID so a reload can resume."
- "Uploaded isn't ready: files sit in quarantine until a worker scans and re-encodes them, and the UI shows Processing until then."
- "User files are served from a separate domain with
Content-Dispositionandnosniff, because an uploaded SVG is an XSS."
Think first
Design the attachment upload for a chat app where users send photos and files up to 100 MB from phones. What do you say in two minutes?
The message appears at once with a local preview (an object URL of the shrunk image) and a sending state. Photos are resized in a worker to 2048 pixels; other files go as they are, after a size check and a sniff of the first bytes.
The client asks POST /attachments for a slot, with an idempotency key. The API checks auth, the conversation, and quota, picks a random key in the quarantine bucket, and returns a pre-signed URL. Files over about 10 MB use S3 multipart with 8 MiB parts, three in flight, so a drop costs one part, and the saved upload ID lets the app resume.
A storage event triggers a worker that scans, re-encodes images, and makes thumbnails, then marks the attachment ready and pushes that over the chat's WebSocket. Files are served from a separate user-content domain through a CDN with short-lived signed URLs, nosniff, and attachment for anything that is not an image or video.
Summary
- Treat an upload as a job with states (pending, uploading, processing, ready, rejected), not as one request. Uploaded is not ready.
- Small files can go multipart through your API; anything bigger goes straight to object storage with a short-lived pre-signed URL for a key the server picked.
- Large or mobile uploads must be resumable: chunks with per-chunk retries, and a way to ask the server how much it has (S3 multipart, GCS resumable, tus). Size parts for S3's 5 MiB minimum and 10,000 part limit.
- Use XHR's
upload.onprogressfor progress;fetchcannot report upload progress in most browsers. Confirm completion only on the server's answer. - Validate in the browser for speed and kindness, and again on the server for security: real type from the bytes, size enforced where bytes land, images decoded and re-encoded.
- Resize photos in a worker before upload for a roughly tenfold cut in bytes; modern browsers already apply EXIF orientation, so don't rotate twice.
- Quarantine, scan, and process in the background; serve user files from a separate domain with server-chosen headers,
nosniff, andattachment. - Set limits per file, per user, and per hour, reserve quota at signing time, make slot creation idempotent, and clean up abandoned uploads.
In the interview
Questions interviewers ask about this. Answer each out loud first, then compare.
Question 1
Why not just POST every file to your API server?
For small files it is fine and simplest: one request, one place for auth and validation. The problem is large or slow files. Every byte costs your bandwidth and holds a worker for minutes, buffering frameworks can run out of memory, and proxies cut you off with body size limits (nginx's 1 MB default) and timeouts.
So I'd keep multipart for small, immediate files like avatars and send anything bigger straight to object storage with a pre-signed URL. The API still decides who may upload what; it just stops carrying the bytes.
Question 2
How does a pre-signed upload stay secure if the browser talks to storage directly?
The API signs one narrow permission: one key it chose itself, one method, a short expiry, and, with an S3 POST policy, a content type prefix and a content-length-range. Storage verifies the signature, so the client cannot widen any of it.
Files land in a quarantine location nobody can read. A worker then checks the real type, scans, and re-encodes before promoting the file, and the API only marks it ready after that. Plus a bucket CORS rule limited to our origin, and quotas checked before signing.
Question 3
A 2 GB upload on mobile keeps failing. How do you fix it?
Make it resumable. Split the file with Blob.slice into parts (S3 multipart with 8 to 16 MiB parts, or tus), retry each part with backoff and jitter, and after any drop ask the server what it has (ListParts or a tus HEAD for Upload-Offset) instead of restarting.
Store the upload ID keyed by a file fingerprint so a reload or app switch can resume; the user re-picks the file, or we keep a copy in OPFS. Size parts so the count stays under 10,000, and add a lifecycle rule to abort abandoned multipart uploads.
Question 4
How do you show accurate upload progress?
Use XMLHttpRequest, whose upload.onprogress gives bytes sent; fetch has no upload progress outside Chromium's streaming bodies. With chunks, sum finished chunk bytes plus the in-flight progress of each chunk, weighted by bytes across files.
Only show done after the server confirms, smooth the time-left estimate with a moving average, and make it accessible: a progress element with a name, milestone announcements in a polite live region, and Cancel and Pause buttons.
Question 5
A user uploads an SVG profile picture. What can go wrong and how do you prevent it?
SVG can contain script. Served inline from the app's own origin, it runs with the viewer's session: stored XSS. Checking the extension does not help.
Convert SVGs to PNG or refuse them for avatars. For general file sharing, serve all user files from a separate registrable domain, with a server-chosen Content-Type, X-Content-Type-Options: nosniff, Content-Disposition: attachment for non-allow-listed types, and a sandboxing CSP.
Practice it
These design problems lean on uploading files.
References
- Using files from web applications (MDN) File, Blob, the file input, and drag and drop.
- XMLHttpRequestUpload (MDN) The upload progress events.
- Uploading and copying objects using multipart upload (Amazon S3) Part sizes, limits, and the three calls.
- Uploading objects with presigned URLs (Amazon S3)
- Resumable uploads (Google Cloud Storage)
- tus resumable upload protocol The core protocol and its extensions.
- File Upload Cheat Sheet (OWASP) Server-side validation and storage checklist.
- createImageBitmap (MDN)
Read next
- REST for frontendsA way to shape an HTTP API around resources and a few standard verbs, so the frontend can fetch, cache, retry, and handle errors in the same predictable way everywhere.
- Retries, timeouts, and backoffHow a client survives a flaky network and a struggling server: give every request a deadline, retry only the failures that can heal, space the retries out with exponential backoff and jitter, and stop retrying when the server or the network is clearly down.
- API contracts, versioning, and errorsThe written agreement between the frontend and the API: a typed schema both sides are checked against, rules for changing it without breaking old clients, and one error shape the UI can always understand, including for partial failures and safe retries.
- Cross-site scripting (XSS)Cross-site scripting is when text from outside your app ends up running as code inside your page, with every power your own code has: the most common and most damaging frontend security bug, and the one frontend engineers own most directly.
- Images and mediaSending each user an image of the right size and format, reserving its space, and loading it at the right moment: images are usually the largest share of a page's bytes and its LCP element.
- CDN cachingKeeping shared copies of your responses on servers near every user, so most requests never travel to your origin: the biggest lever for global speed and for surviving traffic spikes.