Engineering
Idempotent webhooks: making sure the money only moves once
Payment providers promise to deliver a webhook at least once, which is a polite way of saying sometimes twice. If a repeat delivery confirmed the booking a second time, awarded loyalty twice and emailed two receipts, our customers would notice fast. Here is the small idea that makes every downstream effect happen exactly once, no matter how many times the same event lands.
The short version
- Payment webhooks are delivered at least once, so the same event can arrive more than once.
- Every downstream effect of a payment (confirm, loyalty, receipt, gift-card issue) has to be safe to run twice.
- We record each event id the first time we see it, inside the same transaction that does the work.
- A repeat delivery finds the id already recorded and stops before touching anything.
- The whole chain either happens once, in full, or not at all.
A booking is not really paid because a card cleared. It is paid because a series of things happened afterwards: the appointment moved to confirmed, the client earned their loyalty, a receipt went out, and anything they bought alongside it (a gift card, a package) got issued. The card clearing is the trigger; the value is in the follow-on work. So the interesting reliability question is not "did the payment succeed" but "did every consequence of it happen, exactly once."
The reason exactly once is hard is that the notification we get is only guaranteed to arrive at least once. Networks drop responses. A handler that takes a moment too long gets the same event sent again. This is not a flaw in the payment provider; it is the honest behaviour of any system that would rather tell you twice than risk never telling you at all. Our job is to make a second telling harmless.
The shape of a double delivery
Picture the naive version. A payment event arrives, we confirm the booking, add loyalty points, send a receipt. Then the same event arrives again a few seconds later because our first response was slow. Now the booking is confirmed twice (probably fine), the client has double the points (not fine), and they have two receipts in their inbox (they will email us about it). Multiply that across a busy Saturday and you have eroded the one thing a payments system sells, which is trust.
At least once delivery is not a bug to fix. It is a fact to design around.
One gate, before the work
The fix is a single decision made before any of the follow-on work runs: have we already handled this exact event? Every event carries a stable id from the provider. We keep a small table of the ids we have processed. When an event lands, the first thing the handler does is try to record its id. If the id is already there, this is a repeat, and we stop. If it is new, we record it and carry on with the real work.
The part that matters is that recording the id and doing the work live in the same database transaction. They commit together or not at all. If the machine dies halfway through awarding loyalty, the transaction rolls back, the id is not recorded, and the retry that follows gets to do the whole thing cleanly. There is never a state where we have marked the event handled but left half the consequences undone.
This is the difference between checking first and gating atomically. A check ("select, then if absent do the work") has a gap: two copies of the event can both check, both see nothing, and both proceed. Making the id insert itself the gate closes that gap, because the database will only let one insert of a given id win. The loser of that race is, by definition, a duplicate, and it backs out quietly.
Make each effect idempotent anyway
The event gate is the first line, not the only one. Belt and braces means each downstream effect is also written so that running it twice is a no-op. Loyalty is awarded against the payment id, so a second award for the same payment collides and is ignored. A gift card is issued keyed to the order line that bought it, so the second attempt finds it already issued. The receipt is deduped on the invoice. Even if some future refactor accidentally called one of these twice, nothing doubles.
Belt and braces is not paranoia here. The gate protects against duplicate deliveries; the per-effect keys protect against our own mistakes, like a code path that fires the same internal event twice. Reliability at the money layer is worth two independent defences that each catch what the other might miss.
What we get for it
- A webhook that is safe to retry, so we can be aggressive about retrying when something looks stuck.
- A payment whose consequences are all-or-nothing: a client is never left confirmed-but-unrewarded, or receipted-but-unconfirmed.
- A clean log: the processed-events table is also a record of exactly what we have acted on, which is the first thing you want when a customer asks why they did or did not get an email.
None of this is exotic. It is one table, one transaction, and the discipline to make the id the gate rather than a thing you check. But it is the reason a burst of duplicate deliveries during a payment provider hiccup is a non-event for us, instead of a morning of apologising for double charges that never happened and double points that did.
Building on Bookatu?
Bookatu has a public REST API and webhooks. Have a look at the developer docs.
Developer docs