Webhooks

The problem webhooks solve

Picture a small online course shop. A learner clicks Pay, types a card number into the payment provider's form, and waits. The payment provider (say Stripe) talks to the card network, the bank asks for a one-time code, the learner types it, and about 6 seconds later the money is captured. The shop needs to know, so it can unlock the course.

The catch: the payment happens on Stripe's servers, not on yours. Nothing in your system knows the exact moment it finishes. You need some way to hear about an event that happens inside another company's system.

Option one: keep asking

The obvious fix is polling: your server calls GET /v1/payment_intents/pi_123 every few seconds until the status says succeeded. It works for one payment. Now scale it. With 2,000 payments in flight and a check every 2 seconds, you send 1,000 requests a second to Stripe, and almost every answer is "still processing".

• Wasted calls. Most requests return nothing new, and providers rate-limit you (Stripe's default limit is on the order of 100 requests per second in live mode).
• Delay. With a 2-second interval the shop learns about the payment up to 2 seconds late, on average 1 second late.
• Blind spots. Some events start on the provider's side with no request from you at all: a refund issued from the Stripe dashboard, a dispute opened by the bank 40 days later, a subscription that renews at midnight. You would have to poll every object you ever created, forever.

Option two: let them call you

A webhook flips the direction. You give the provider a URL on your server, such as https://api.courseshop.com/webhooks/stripe, and a list of events you care about. When one of those events happens, the provider sends an HTTP POST to that URL with a JSON description of the event. Usually that request arrives within a second or two.

That is the whole idea: a webhook is a server-to-server HTTP callback. The provider is the client, your server is the server, and the event is the request body. No new protocol, no open connection, no library needed to receive it: any web framework that can handle a POST can receive a webhook.

Why a frontend engineer should care

Webhooks never touch a browser. A browser has no public address, sleeps when the tab closes, and could not keep a secret even if it could receive the request. So why is this a frontend lesson? For two reasons that come up again and again in system design interviews.

1. They are the first hop of many live updates. The learner is still looking at a "Processing payment" spinner. The webhook reaches your server, and your server has to carry the news the rest of the way to that tab with server-sent events (SSE) or a WebSocket. Designing that path is a frontend design question.
2. Products that send webhooks need a settings page. If you build the Stripe in this story (a CRM, a continuous integration service, a form builder), your customers need a screen to add endpoints, pick events, see and rotate the signing secret, and inspect failed deliveries. "Design the webhooks settings page" is a real interview prompt.

Before either of those, you need to understand what makes webhooks tricky.

Anyone on the internet can send a POST to your URL, so you have to prove each request is genuine. The network can drop the request or your server can be down, so the provider retries, which means you will see some events twice. And events can arrive out of order. The rest of this lesson takes those problems one at a time.

The mental model: a courier with a sealed letter

Think of a courier company that promises to tell you when your parcel moves. You give them your address once. From then on, every time the parcel is scanned, a courier walks to your door with a short letter: "Parcel 481 left the warehouse at 09:12".

Three details of that service map one to one onto webhooks, and they are the three hard parts of the topic.

• The wax seal. Anyone could knock on your door with a letter that claims to be from the courier. So each letter carries a seal that only the courier and you know how to make. You check the seal before you believe the letter. In webhooks, the seal is a signature made with a shared secret.
• Knocking again. If nobody opens the door, the courier comes back later, and again, for a few days. Sometimes you did open, took the letter, but the courier did not see you nod, so they come back with the same letter. In webhooks, these are retries, and they mean the same event can arrive more than once.
• Letters cross in the post. "Out for delivery" can reach you before "left the warehouse" if the first courier got stuck in traffic. In webhooks, ordering is not guaranteed.

The words you will see

The difference between an event and a delivery is the key to most bugs. The event evt_1QpX happened once. Its deliveries are the attempts to tell you about it, and there might be four of them because your server was down for 10 minutes. Your code must act on the event once, no matter how many deliveries carry it.

What a webhook is not

It is not a request and reply. The provider does not wait for your business logic to finish, and it ignores most of what you send back: it only looks at the status code. A 2xx means "received", anything else (or no answer in time) means "try again later". If you need a value from the provider, call its API.

It is not a live connection either. Unlike a WebSocket or an SSE stream, there is nothing open between deliveries. Each one is a fresh, short HTTP request, often from a different IP address in the provider's fleet. That makes webhooks cheap for the provider to send to millions of endpoints, and it is why they suit events that happen now and then (a payment, a push, a form submission) rather than a stream of 60 updates a second.

One delivery, step by step

Let's follow one real event through the course shop, with real values. Setup happened once, weeks ago: a developer opened Stripe's dashboard, added the endpoint https://api.courseshop.com/webhooks/stripe, ticked the event types payment_intent.succeeded and charge.refunded, and copied the signing secret into the server's environment as STRIPE_WEBHOOK_SECRET.

1. The provider sends a POST

At 08:53:20 UTC the learner's payment succeeds. Stripe creates the event evt_1QpX, stores it, and puts a delivery on its outgoing queue. A moment later one of its workers sends this request:

POST /webhooks/stripe HTTP/1.1
Host: api.courseshop.com
Content-Type: application/json; charset=utf-8
User-Agent: Stripe/1.0 (+https://stripe.com/docs/webhooks)
Stripe-Signature: t=1792400000,v1=f44d4b291d5e4b72200aa7c3808d4ed880a2f50b8e7b726a8b3f708bf6512765
Content-Length: 100

{"id":"evt_1QpX","type":"payment_intent.succeeded","data":{"object":{"id":"pi_3Qp9","amount":4999}}}

Read it top to bottom. The Stripe-Signature header carries a Unix timestamp (t=1792400000, which is 2026-10-19 08:53:20 UTC) and a signature (v1=..., 64 hex characters, which is 32 bytes of HMAC-SHA256). The body says what happened (type), gives the event a unique id, and includes the object involved, here a payment of 4999 cents.

2. Your handler does four short things

1. Read the raw body as bytes, before any JSON parsing. The signature was made over these exact bytes.
2. Verify the signature and the timestamp. If either check fails, answer 400 and stop. Do not look inside the body first.
3. Record the event in your database, keyed by its id, so a second delivery of the same event is noticed. Then put a job on a queue to do the real work.
4. Answer 200 quickly. The whole handler should take tens of milliseconds.

app.post("/webhooks/stripe", express.raw({ type: "application/json" }), async (req, res) => {
  const rawBody: Buffer = req.body; // bytes, not parsed JSON
  const header = req.get("Stripe-Signature") ?? "";

  if (!verifyStripe(rawBody, header, process.env.STRIPE_WEBHOOK_SECRET!)) {
    return res.status(400).send("bad signature");
  }

  const event = JSON.parse(rawBody.toString("utf8"));
  await inbox.insertIfAbsent(event.id, event.type, rawBody); // a repeat is a no-op
  // Enqueue every time: the worker skips processed events, and a crash
  // between insert and enqueue is healed by the provider's retry.
  await queue.add("stripe-event", { eventId: event.id });

  res.sendStatus(200); // ack fast; a worker does the slow part
});

3. A worker does the slow part

A background worker picks up the job a few milliseconds later. It loads the event from the inbox table, finds the order linked to pi_3Qp9, marks it paid, grants access to the course, and queues a receipt email. Then it publishes order.paid so the learner's open tab can update (chapter 7 covers that last hop).

Why split the work? Because providers wait only a short time for your answer. GitHub ends the delivery as failed if you take more than 10 seconds, and Stripe's docs tell you to return 2xx before running any complex logic. Sending an email, calling another API, or waiting on a slow database lock can easily blow that budget, and then the provider retries an event you were in the middle of handling.

4. What your status code tells the provider

Signing and verification

Your webhook URL is public, and URLs leak: they end up in logs, screenshots, and support tickets. Anyone who finds /webhooks/stripe can send a hand-written payment_intent.succeeded and unlock a course without paying. So the first job of every handler is to answer one question: did this request really come from the provider, unchanged?

HMAC in plain words

Almost every provider answers that question with an HMAC (hash-based message authentication code). An HMAC takes a secret key and a message and produces a short fixed-size code, 32 bytes for HMAC-SHA256. Two facts make it useful: the same key and message always give the same code, and without the key you cannot produce a valid code for any message, not even one you have seen signed before with a single byte changed.

When you register an endpoint, the provider generates a random secret and shows it to you once. Both sides now hold the same secret.

For each delivery, the provider computes HMAC-SHA256(secret, signed_content) and puts the result in a header. Your server computes the same thing from the bytes it received and compares. Match means two things at once: the sender knew the secret (authenticity) and nobody changed a byte on the way (integrity).

Three real formats

The idea is the same everywhere, but the details differ, and the details are where verification breaks. Here is the same 100-byte body signed three ways, all computed in Node with crypto.createHmac. Stripe and GitHub use the example secret whsec_test_courseshop. Standard Webhooks needs a base64 key of at least 24 bytes, so its value uses a separate example key and the id evt_1QpX.

Standard Webhooks is an open specification, started in 2023 by Svix with people from companies like Zapier, Twilio, Kong, and ngrok, that writes down one shared format so receivers can use one library for many providers. Besides the three headers above, it allows several signatures in one header separated by spaces, which is how secret rotation works, and it defines a v1a variant that uses an asymmetric ed25519 key pair instead of a shared secret.

Verifying a Stripe-style header

import { createHmac, timingSafeEqual } from "node:crypto";

const TOLERANCE_SECONDS = 300;

export function verifyStripe(rawBody: Buffer, header: string, secret: string): boolean {
  const parts = header.split(",").map((p) => p.split("="));
  const t = Number(parts.find(([k]) => k === "t")?.[1]);
  const sigs = parts.filter(([k]) => k === "v1").map(([, v]) => Buffer.from(v, "hex"));
  if (!Number.isFinite(t) || sigs.length === 0) return false;

  const age = Math.abs(Date.now() / 1000 - t);
  if (age > TOLERANCE_SECONDS) return false; // too old: maybe a replay

  const expected = createHmac("sha256", secret)
    .update(`${t}.`)
    .update(rawBody)
    .digest();

  return sigs.some((s) => s.length === expected.length && timingSafeEqual(s, expected));
}

Four rules, and the mistake behind each

• Sign the raw bytes. The most common bug: a framework parses the JSON, then the code calls JSON.stringify(req.body) and verifies that. Spacing, key order, number formats, and escaped characters can all change. Re-serialising our example body with two-space indentation gives 6904c351... instead of f44d4b29...: a valid event rejected. In Express, use express.raw on that one route; in Next.js route handlers, read await request.text() before anything else.
• Compare in constant time. A normal === on strings stops at the first different character. In theory an attacker could time thousands of tries and learn the right signature byte by byte. crypto.timingSafeEqual always takes the same time. It throws if the two buffers differ in length, so check the length first.
• Reject old timestamps. The timestamp is inside the signed content, so an attacker cannot change it. If they capture a real delivery (from a leaked log, say), they can only replay it unchanged, and after 5 minutes your check rejects it. Keep server clocks in sync with NTP (Network Time Protocol), or you will reject real deliveries.
• Keep the secret on the server, one per endpoint. Never ship it to a browser or a mobile app, and keep test-mode and live-mode secrets apart: verifying live events with the test secret is the second most common bug.

Rotating a secret without dropping events

Secrets leak, and teams rotate them on a schedule. A hard switch would break every delivery between the moment the provider starts using the new secret and the moment your deploy lands.

So providers overlap the two: Stripe's "roll secret" lets you keep the old secret valid for up to 24 hours, and during that window each delivery is signed with both, as two v1= entries in one header. Your verifier accepts the delivery if any v1 signature matches the secret it holds, old or new, which is why the code above loops with some.

Retries and at-least-once delivery

Deliveries fail all the time, for boring reasons. You deploy and the old process stops for 20 seconds, or the database is locked for a migration. A handler throws on an event type nobody expected, or a load balancer drops one request in a million. If the provider gave up after one try, you would silently lose payments.

So every serious provider retries: if it does not get a 2xx within its timeout, it schedules another attempt later, with growing gaps between attempts. This is exponential backoff, the same idea the [retries and backoff lesson](/system-design/concepts/retries-and-backoff) covers for browser requests, applied on the provider's side.

Real retry schedules

Why retries mean duplicates

Here is the uncomfortable part. A retry happens when the provider thinks the delivery failed, which is not the same as when it really failed.

Picture the timeline: your server receives evt_1QpX, marks the order paid, and sends 200. On the way back, the connection resets. The provider never sees the 200, waits, and sends evt_1QpX again. Your server did the work once and now receives the event a second time.

No protocol can fix this from the provider's side. Over an unreliable network, the sender can never be sure the receiver acted, so it has two choices: send once and risk losing the event (at most once), or keep sending until it hears back and risk repeats (at least once). Every provider picks at least once, because a lost payment is worse than a repeated one. "Exactly once" is something you build on the receiver side by ignoring repeats.

• Lost acknowledgement: the work was done but the 200 never arrived.
• Slow answer: the work took longer than the provider's timeout, so it counted as failed even though it finished.
• Provider-side repeats: providers say plainly that, rarely, they may send the same event more than once even without a failure.
• Manual redelivery: someone on your team clicked "Resend" in the dashboard while debugging.

The life of one delivery

On the provider's side, each event and endpoint pair moves through a small set of states. Knowing them helps you read a delivery log and design one (chapter 8). Notice that "succeeded" only means "the provider saw a 2xx", not "the customer's business logic ran".

When you come back from an outage

Retries also change what your outages look like. If your receiver is down for 2 hours at 50 events a minute, about 6,000 events pile up at the provider. When you come back, they do not arrive in a neat line: each one is on its own backoff clock, so they arrive in waves over the next hours, mixed with new events.

Plan for that burst: a receiver that only verifies, stores, and enqueues can absorb thousands a minute, while one that does the work inline will fall over again and extend the outage.

Idempotent handlers and out-of-order events

An operation is idempotent when doing it twice has the same effect as doing it once. "Set the order status to paid" is idempotent. "Add 1 to the learner's credit balance" and "send a receipt email" are not. Since deliveries repeat, your handler has to make the whole reaction idempotent, even the parts that are not by nature.

The tool is the same one the [retries and backoff lesson](/system-design/concepts/retries-and-backoff) uses for idempotency keys on browser requests, pointed the other way. There, the client sends a key and the server remembers it. Here, the provider already gives every event a unique id (evt_1QpX at Stripe, the X-GitHub-Delivery header at GitHub, webhook-id in Standard Webhooks). You remember which ids you have handled.

The inbox table

Keep a table of received events with the event id as the primary key. The receiver inserts each verified event, and the database's unique constraint does the deduping for you, even when two copies of the same event arrive at the same millisecond on two different servers. This pattern is often called the inbox. Try it below: send deliveries, lose an acknowledgement, and switch the dedupe check off to see what a retry does.

async function processEvent(eventId: string) {
  await db.transaction(async (tx) => {
    const row = await tx.one(
      "SELECT * FROM webhook_inbox WHERE id = $1 FOR UPDATE", [eventId]);
    if (row.processed_at) return; // handled before: do nothing

    const event = JSON.parse(row.body);
    if (event.type === "payment_intent.succeeded") {
      const pi = event.data.object;
      await tx.query(
        "UPDATE orders SET status = 'paid' WHERE payment_intent = $1", [pi.id]);
      await tx.query( // email goes out later, from an outbox, keyed by event
        "INSERT INTO outbox (key, kind) VALUES ($1, 'receipt') ON CONFLICT DO NOTHING",
        [`receipt:${eventId}`]);
    }
    await tx.query("UPDATE webhook_inbox SET processed_at = now() WHERE id = $1", [eventId]);
  });
}

The transaction is the important part. If you marked the event processed first and crashed before updating the order, the retry would be skipped and the payment lost. If you updated the order first and crashed before marking, the retry would run it again.

One transaction removes both gaps. Side effects that leave the database, like the email, go through an outbox row in the same transaction, with a key that includes the event id, so a second run cannot queue a second email.

Events do not arrive in order

Providers send deliveries in parallel from many workers, and each failed delivery sits on its own retry clock. So the order of arrival is not the order things happened. Stripe says so in its docs: it does not guarantee delivery in the order events were generated.

A real example: a subscription is created, then cancelled 2 seconds later. If the customer.subscription.created delivery fails once, its retry can arrive minutes after customer.subscription.deleted, and a naive handler marks a cancelled subscription active again.

From webhook to the user's screen

Back to the learner staring at a spinner. The webhook has reached your server, but the browser tab cannot receive it: a browser has no public URL, is often asleep or closed, and could not keep the signing secret safe. The event has to travel the last stretch through your own system. This is the part of the topic that belongs to frontend system design.

The pipeline

1. Receive: the webhook route verifies, writes the inbox row, enqueues a job, and answers 200. Tens of milliseconds.
2. Process: a worker applies the event (order to paid) inside a transaction.
3. Publish: after the commit, the worker publishes a small message of your own, such as { type: "order.paid", orderId: "481" }, to a publish/subscribe (pub/sub) channel (Redis pub/sub, NATS, a managed service like Ably or Pusher) named after the user or the order.
4. Push: whichever server holds that user's live connection is subscribed to the channel. It writes the message down a server-sent events stream or a WebSocket.
5. Render: the page swaps the spinner for "You're in. Start lesson 1".

Two details in that list matter in an interview. First, publish after the commit, never before: otherwise the browser can hear "paid", fetch the order, and still see pending. Second, publish your own event, not the provider's: the browser should never see raw Stripe payloads, which contain fields it has no business reading, and your own event names keep the frontend independent of which payment provider you use this year.

The redirect often wins the race

On a hosted checkout, the learner pays on the provider's page and is redirected back to /checkout/success?order=481. That redirect and the webhook leave the provider at about the same moment, and the redirect is often faster. So the success page loads, asks your API for order 481, and gets pending. If the page says "Payment pending", the learner panics and pays again.

The fix is a UI state designed for the gap. Show "Confirming your payment" with a spinner, and listen for the update. If the order flips to paid in the first few seconds, which is the usual case, show success.

If it takes longer than about 30 seconds, change the message to say the receipt will arrive by email and the course will unlock on its own, with a link to the orders page. Never show a red error just because the webhook is slow.

Choosing the last hop

For a checkout confirmation, polling for 30 seconds is honest and enough: it is one page, one event, and a few requests. For a dashboard that should light up whenever a GitHub build finishes or a Shopify order arrives, use one long-lived channel per tab: see the [server-sent events](/system-design/concepts/server-sent-events) and [WebSockets](/system-design/concepts/websockets) lessons for how to run them at scale. Whatever you pick, the page should also refetch the order when the connection reopens, because a message published while the tab was offline is gone.

Sending webhooks, and the settings page

Now switch sides. Your product is the provider: a form builder that tells customers when someone submits a form, or a CI service that reports finished builds. Your customers write the receivers. Everything in the earlier chapters becomes your job to make easy, and the settings page is where they meet it.

What the sending side needs

• Record before you send. Write the event in the same transaction as the change that caused it (an outbox row), then let a dispatcher send it. Sending from inside the request that changed the data loses events whenever the process dies.
• One queue per endpoint. A single customer's endpoint that hangs for 30 seconds per request must not slow everyone else. Short timeouts (5 to 15 seconds) and a cap on concurrent requests per endpoint keep one slow receiver contained.
• Retry with backoff and jitter, keep every attempt in a log, and disable an endpoint after days of failures, with an email to its owner first.
• Block internal addresses. Customers type URLs that your servers will call. Without checks, someone registers http://169.254.169.254/ (the cloud metadata service) or http://10.0.0.5/admin and your webhook system reaches inside your own network. This attack is server-side request forgery (SSRF). Resolve the hostname, refuse private, loopback, and link-local ranges, check again at send time (DNS can change), and send from an egress proxy with no route to internal networks.

The settings page, piece by piece

• Endpoint form. A URL field that requires https:// and validates as you type, a description, and an event picker grouped by area (form.*, payment.*) with a select-all per group. Default to the few events most people need, not all 80.
• Signing secret. Show it in full once, right after creating the endpoint, with a Copy button. Afterwards show it masked (whsec_••••••••3f9a) with a Reveal toggle that asks for the password again. A Rotate action asks how long the old secret should stay valid (now, 1 hour, 24 hours) and explains why.
• Send test event. Sends a sample, clearly marked as a test, so a developer can check their verification code without making a real payment. Show the response inline.
• Delivery log. One row per attempt: time, event type, event id, status code, duration, and attempt number. Expanding a row shows the exact request headers and body you sent and the first few KB of the response. A Redeliver button sends the same event again, with the same id, so the receiver's dedupe still works.
• Health. A success rate for the last 24 hours, a banner when deliveries are failing, and a clear "Disabled after 5 days of failures. Enable" state instead of silent dropping.

The delivery log answers most webhook support tickets on its own. "We never got the event" becomes "your server answered 502 four times between 02:10 and 02:40, here is the body". Spend design time on it: filters by status and event type, search by event id, and response bodies that are readable, not one long line.

Accessibility on this page

• The Reveal toggle is a real button with aria-pressed, and the masked value has a text label, so a screen reader does not read out a row of bullets.
• Copy and Redeliver report their result in a polite live region ("Secret copied", "Redelivery queued").
• Status in the log is text plus colour (200 OK, 500 Failed), never colour alone, and the log is a real table with column headers.
• Rotating or deleting asks for confirmation in a dialog that names the endpoint, with focus moved into the dialog and back afterwards.

Failure modes and real products

Webhook bugs hide during normal days and show up during deploys, outages, and secret changes. Here are the ones teams hit most, how you notice each, and the fix.

Recovering missed events

Assume that one day you will miss events: an endpoint disabled over a holiday, a bug that answered 200 without saving. Plan the recovery before you need it. Most providers let you list recent events through their API (Stripe keeps 30 days of events in GET /v1/events), so a small reconciliation job can compare their list with your inbox table and feed any missing ids through the same worker. Because the worker is idempotent, running it over events you already have is harmless.

In real products

• Stripe signs with Stripe-Signature (t= plus one or more v1=), retries live-mode events for up to 3 days, does not promise order, and moved its newer API towards thin events that you follow with a fetch. Its CLI command stripe listen --forward-to localhost:3000/webhooks forwards real test events to a laptop.
• GitHub signs with X-Hub-Signature-256 over the body only, identifies each delivery with X-GitHub-Delivery, expects an answer within 10 seconds, and does not retry automatically; you redeliver from the settings page or the API.
• Slack's Events API checks a new URL before using it: it sends a url_verification request with a challenge value that your endpoint must echo back. This proves the endpoint is yours and willing to receive events.
• Shopify signs with X-Shopify-Hmac-Sha256 (base64 of an HMAC over the body) and also offers delivery to a queue such as Amazon EventBridge or Google Pub/Sub instead of HTTPS, which removes the receiver's uptime from the picture.
• Svix and other webhook-sending services sell the whole sending side, including the settings UI as an embeddable portal, because building retries, logs, and SSRF protection well takes months.

Using webhooks in a system design interview

Webhooks come up in frontend interviews in two shapes. Knowing which one you are in tells you how deep to go.

Shape one: an async third party in your design

Checkout with a payment provider, a document sent for e-signature, a video uploaded for transcoding, a background check. Anything where a third party finishes the job later. Bring up webhooks the moment you notice the result does not come back in the same request. Then:

1. Draw the five-box pipeline from chapter 7 and say "the webhook lands on our server, never on the browser".
2. Name the three receiver rules in one breath: verify the signature over the raw body, dedupe by event id, answer fast and work in the background.
3. Spend most of your time on the UI during the gap: the "Confirming" state, the live update, the fallback after 30 seconds, and what happens if the user closes the tab (email, push, or the state is correct on the next visit).

Shape two: design the webhooks feature

"Design the developer settings for our API product" or "design a webhook delivery log". Here the settings page is the main deliverable. Use the order from chapter 8: endpoint form, secret lifecycle, test event, delivery log with redeliver, health and disabling. Then show the data the page needs: endpoints, subscriptions, events, and attempts as four entities, with the delivery log paginated by time and filterable by status.