Webhook Idempotency, Done Properly: Keys, Storage, and Replay Windows

You will build a webhook receiver that can receive the same event five times and process it exactly once. Not approximately once. Not usually once. Exactly once from the perspective of your database, your billing side effects, and your downstream jobs.

If you consume webhooks from a payment provider, a communications platform, a code host, or any system you do not control, duplicates are not an edge case. They are the contract. Providers retry because they cannot distinguish a lost response from a lost request. Your network drops a packet. Your server restarts mid-handler. Your load balancer closes an idle connection. From the provider’s side, all of those look identical: no acknowledgement. So it sends again. Your job is to make that second, third, and tenth delivery boring.

What idempotency actually means for webhooks

An operation is idempotent when applying it more than once has the same effect as applying it once. For an incoming webhook, that means three separate guarantees working together.

First, you can recognise a duplicate. Second, you can prove you have already processed it. Third, you can respond to the duplicate in a way that stops the provider from retrying forever without lying about what happened.

Most broken implementations only do the first part. They check for a key, find nothing, proceed, and race with a concurrent duplicate that also found nothing. Or they store the key after processing, so a crash between the side effect and the store reopens the window. Or they return an error for a duplicate, which triggers more retries.

You need to design for all three at once.

Choose your idempotency key

You have three common sources for a key, and they are not equal.

For third-party webhooks, use the provider event ID as the primary key, namespaced by provider. Store stripe:event_id or github:delivery_event_id, never just the raw ID. Two providers will eventually collide if you do not namespace, and even a single provider can reuse ID shapes across event types or test and live modes.

If the provider gives you no stable event ID, fall back to a hash, but hash the raw bytes exactly as received, before any JSON parsing or normalisation. Parsing and re-encoding will change whitespace, key order, and number formatting, and your hash will stop matching on retry. Read the body as text first, verify any signature against those same raw bytes, hash them, then parse.

Also store the event type and the provider’s account or installation identifier alongside the key. The same event ID delivered to two different connected accounts in a multi-tenant system is not a duplicate. It is two legitimate events that happen to share an identifier in different scopes. Scope your uniqueness constraint to tenant plus provider plus event ID.

Store the key before you do the work

The ordering is the entire design. Claim the key atomically, in the same database transaction or with a single atomic operation, before you run any business logic.

Here is a receiver in TypeScript using a relational database with a unique constraint doing the heavy lifting:

import { createHash } from "node:crypto";
import type { Request, Response } from "express";
 
type ClaimResult =
  | { kind: "claimed" }
  | { kind: "duplicate_completed" }
  | { kind: "duplicate_in_progress" };
 
async function claimEvent(
  db: any,
  scopedKey: string,
  eventType: string,
  rawBody: string
): Promise<ClaimResult> {
  const bodyHash = createHash("sha256").update(rawBody).digest("hex");
 
  // INSERT ... ON CONFLICT is atomic. Only one concurrent caller wins.
  const inserted = await db.query(
    `INSERT INTO webhook_events
       (scoped_key, event_type, body_hash, status, received_at)
     VALUES ($1, $2, $3, 'processing', now())
     ON CONFLICT (scoped_key) DO NOTHING
     RETURNING scoped_key`,
    [scopedKey, eventType, bodyHash]
  );
 
  if (inserted.rowCount === 1) {
    return { kind: "claimed" };
  }
 
  const existing = await db.query(
    `SELECT status, body_hash FROM webhook_events WHERE scoped_key = $1`,
    [scopedKey]
  );
  const row = existing.rows[0];
 
  if (row.body_hash !== bodyHash) {
    // Same key, different payload. This is not a retry. Investigate.
    return { kind: "duplicate_in_progress" };
  }
 
  return row.status === "completed"
    ? { kind: "duplicate_completed" }
    : { kind: "duplicate_in_progress" };
}
 
export async function handleWebhook(req: Request, res: Response) {
  const rawBody = req.body as string; // configured as raw text upstream
  const eventId = req.header("x-provider-event-id");
 
  if (!eventId) {
    return res.status(400).json({ error: "missing event id" });
  }
 
  const scopedKey = `provider:${eventId}`;
  const claim = await claimEvent(db, scopedKey, "unknown", rawBody);
 
  if (claim.kind === "duplicate_completed") {
    return res.status(200).json({ received: true, duplicate: true });
  }
  if (claim.kind === "duplicate_in_progress") {
    // Tell the provider to try again later instead of assuming success.
    return res.status(409).json({ error: "already processing" });
  }
 
  // You own the claim. Do the work, then mark completed in the same flow.
  return res.status(200).json({ received: true });
}

Notice what this code refuses to do. It does not check and then insert in two steps. That gap is where concurrent duplicates slip through. A unique constraint plus an atomic insert-or-nothing is what closes the race. Application-level checks alone will not save you under real retry timing, where two deliveries can arrive milliseconds apart on different server instances.

It also does not delete the row after processing. The row is your proof. Deleting it reopens replay.

Handle the in-progress window honestly

The hardest state is not completed or unseen. It is processing. A first delivery claimed the key, started work, and has not finished. A second delivery arrives. What should you return?

If you return success, the provider stops retrying, which is fine if the first attempt completes, and catastrophic if it crashes. If you return an error, the provider retries, which is correct, but you need to make sure one of those retries eventually becomes the owner if the first attempt died.

The practical pattern has three parts:

1. Claim with a status of processing and a timestamp.
2. For a duplicate in the processing state, return a retryable status such as 409 or 503, unless the processing claim is stale.
3. Define stale clearly. If a claim has been processing longer than your handler should ever take, plus a safety margin, a new delivery may reclaim it by updating the row conditionally where the status is still processing and the timestamp is older than your threshold.

That conditional reclaim must also be atomic. Update only if the old timestamp matches what you read, or use a version column. Otherwise two retries can reclaim at once and you are back to double processing.

Your handlers should also be designed to resume, not restart blindly. If processing means multiple steps, record step completion in the event row or in a related table. A reclaimed event can then skip steps that already committed and finish the rest. This matters most when a step has an external side effect you cannot roll back, such as sending an email or creating a resource in another system. For those steps, pass your scoped key to the downstream system as its idempotency key if it supports one, so a resume does not create a second resource.

Separate receipt from processing without losing the guarantee

Many teams want to acknowledge quickly and process asynchronously. That is a sound instinct, especially when providers enforce short timeouts, but it changes where your guarantee lives.

The safe version looks like this. In the request handler, verify the signature, claim the key, and persist the full raw event in the same transaction. Only then enqueue a job and return success. The queue message should carry the scoped key, not a copy of business logic decisions made in the handler.

Your worker then loads the stored event, checks its status, processes it, and marks it completed. If the worker crashes, the event remains in a recoverable state and can be re-driven from the stored record, not from a provider retry you already acknowledged away.

The unsafe version returns success after enqueueing to an in-memory buffer, or after writing to a queue without first persisting the event. If the process dies between acknowledgement and persistence, the event is gone and the provider believes you have it.

If you go async, you also need a sweeper. A periodic job that finds events stuck in processing beyond your threshold and re-enqueues them closes the loop that provider retries used to close for you. Once you acknowledge, you own recovery.

Replay windows and retention

Providers do not only retry in the moment. Most let an operator replay an event from a dashboard hours or days later, and some resend automatically after an outage on their side. Your deduplication store has to live longer than the longest realistic replay.

Set retention by asking how long the provider keeps events available for replay, and how long an incident on your side might require reconciliation. For most systems that argues for keeping completed records for weeks at minimum, archiving rather than hard-deleting. The row is small. The side effect it prevents is not.

If you purge after a short window and the provider replays an older event, you will process it again as new. Where that matters, keep a lightweight tombstone far longer than the full payload: the scoped key, the body hash, and the completion time. Be explicit in your documentation about the window you guarantee, rather than promising exactly-once delivery end to end, which no webhook system can honestly make.

Verification, ordering, and the failures nobody tests

Idempotency does not replace signature verification. Verify first, claim second. An unauthenticated caller who can guess event IDs could otherwise poison your store by claiming keys with junk payloads, blocking the legitimate event that arrives later. That is why the code above compares the stored body hash when a key collides. A collision with a different body is a signal, not a duplicate. Log it, alert on it if it repeats, and do not mark the legitimate path as completed based on an attacker-supplied row.

Ordering is a separate trap. Webhooks are not guaranteed to arrive in the order the events occurred. An “updated” event can arrive before the “created” event it depends on if the first delivery was delayed. Idempotency stops you processing the same event twice. It does not stop you processing events in the wrong order.

Handle ordering at the domain level. Store the provider’s event timestamp or sequence if one exists. When an update arrives for a resource you have not seen, either create a minimal placeholder and reconcile when the creation event arrives, or park the update in a pending state and retry it after a delay. What you should not do is discard it and rely on a future event to fix the state. There may not be one.

Test these cases deliberately before you ship:

• The same event delivered twice concurrently, not just sequentially.
• A crash after the side effect but before the completion mark.
• A replay of a completed event long after the original.
• The same event ID with a tampered body.
• An update event arriving before its creation event.
• A processing claim that never completes, and the reclaim path that follows.

Sequential duplicate tests pass in almost every broken implementation. Concurrent and crash-window tests are the ones that expose the design.

A storage shape that stays simple

You do not need a distributed system for this. You need one table, one unique constraint, and discipline.

Add an index on status plus received_at so your sweeper can find stuck rows efficiently. Keep payload storage separate or archived if volume grows, but keep the key, hash, and status in your primary store for the full retention window.

For very high volume, a cache or key-value store can absorb the first check, but your relational unique constraint should remain the source of truth. A cache eviction must never be what allows a duplicate through. Caches forget. Ledgers should not.

Verdict

Build your receiver in this order. Verify the signature against the raw body. Claim the scoped key atomically before any side effect. Store the full event durably before you acknowledge. Process in a way that can resume, forwarding the same key to downstream systems. Keep the record, or at least its tombstone, longer than any realistic replay.

Skip any one of those steps and you do not have an idempotency strategy. You have a hope that retries will be kind. They will not be, usually on the day the event is a payment, a provisioning step, or a cancellation you could not afford to apply twice.