Webhooks arrive twice: making handlers safe to repeat

Picture a booking system. A reservation event arrives from another service. The system makes a temporary door code and sends it to the guest. Then the same event arrives again.

If the handler runs from the top a second time, it makes a second record, a second code and a second message. Nobody did anything wrong. The event simply came twice.

This note is about why that happens and how I make a handler safe to repeat. It comes from the system in the booking and access case.

Why the same event arrives twice

Most services that send webhooks promise “at least once”. They do not promise “exactly once”. The sender sends the event and waits for a reply. If the reply is slow, or never arrives, the sender can’t tell whether you got it, so it sends the event again.

Your handler may have finished all its work and still be late with the reply. The sender then sends a copy of an event you have already handled. Someone replaying events by hand after an outage has the same effect.

So the fix is not to hope the repeat never comes. The fix is to make the repeat harmless.

What safe to repeat means

The word for it is idempotent. A handler is idempotent when running it twice leaves things exactly as running it once would.

The usual way is to use the event’s id. Before the handler does anything, it asks whether it has seen that id. If it has, it says so and stops. If it hasn’t, it records the id and carries on.

// A sketch of the idea, not production code.
async function handleEvent(event: IncomingEvent): Promise<void> {
  const firstTime = await events.recordIfNew(event.id);

  if (!firstTime) {
    log.info("Already handled", { eventId: event.id });
    return;
  }

  await queue.add("set-up-access", { bookingId: event.bookingId });
}

The event id guards the front door. The work behind it needs the same care, because a job that failed halfway will run again, and the part that already succeeded must not repeat.

Answer quickly, do the slow work later

The sender only needs to hear that you got the event. In the booking system, the web request validates the event, including its signature, and records it. Then it hands the rest to a background queue and replies.

Access setup and the guest message run from the queue, outside that request. A slow provider no longer holds up the reply, so the sender has less reason to send the event again.

Give every stage its own record

Each stage of a booking writes its own status when it finishes. That record is the audit trail.

Two things follow from it. A failed stage can be retried on its own, and the stages that already succeeded are left alone. And when something goes wrong, staff can open the booking and see which stage stopped, instead of guessing which service failed.

Try it

The model below is a simplified version of that system. Start a booking, then send the same event again.

Here is what to look for:

1. Press New booking and let it run until access is active. Then press “Send the same event again”. Event deliveries goes to 2. Booking records and Access codes created stay at 1.
2. Press Reset, then make the lock provider fail and start another booking. Access setup retries, and the wait between tries grows. Bring the provider back and it goes through. The stages before it do not run again.

When a queue is justified

A queue is one more thing to run and monitor, so it has to earn its place.

In the booking system it does. Several external services have to finish in the right order, and any of them can fail for a while. A queue lets the work wait and try again without holding up the request.

If the work is one quick step with one dependency, a queue is usually extra machinery. Doing the work inside the request, behind the event id check, is simpler and often enough.

Either way, the id check comes first. Most queues also deliver a job at least once, so the queue does not remove duplicates by itself. The handler still has to be safe to repeat.