Back to engineering

Engineering

Idempotency keys that survive a cut-off request

The Bookatu engineering team7 min read

Our REST API and MCP server take an idempotency key on the writes they share, so a retry never creates the same thing twice. Then we found that the case the keys exist for was the one they handled worst: a request cut off halfway locked its key for a whole day. Here is the lease that fixed it, and the other rules that make a retry safe.

The short version

  • A write may carry a key the caller chooses. The same key with the same request within 24 hours gets the first call's answer back, and nothing runs again.
  • The key is claimed with one insert on a unique index before the write runs, so two retries that race run the write once.
  • A claim with no answer that is older than any request can run was cut off. The next retry takes it over with one conditional update.
  • A write that was made is answered as made, even if storing its answer fails. A write that was refused frees its key.
  • Every refusal carries a code and says what to do next, in a form an AI client reads.

Writes to a booking API get retried. A client gives up at ten seconds while the server finishes at eleven. An AI agent decides its last call probably failed and sends it again. Without protection, either can make a second booking or a second product.

The usual answer is an idempotency key, the pattern Stripe made familiar. The caller sends a unique key with the write. The server remembers the key and the answer. A retry with the same key gets the same answer, and nothing runs twice. Our REST API takes the key as the Idempotency-Key header. Our MCP server takes it as an idempotencyKey argument on every write. Both use the same stored rows.

Claim first, then run

The order is the whole design. The key is claimed before the write runs, with one insert that a unique index can refuse:

ts
const claimed = await db
  .insert(idempotencyKeys)
  .values({ orgId, credentialId, key, tool, requestHash, expiresAt: addHours(now, 24) })
  .onConflictDoNothing()
  .returning({ id: idempotencyKeys.id });

if (claimed.length > 0) return { kind: "new", id: claimed[0].id };

The unique index covers the business, the credential and the key. Two integrations can both use keys like order-1 and never collide. If the insert lands, this request owns the key and runs. If it does not, someone already holds it, and we read the row to find out what that means.

The request hash is a SHA-256 of the tool's name and the tool's own arguments as canonical JSON, with the keys sorted, so the same request always hashes the same. The same key with a different hash is a mistake by the caller, and we say so instead of guessing:

ts
if (held.tool !== tool || held.requestHash !== hash) {
  throw refusal("IDEMPOTENCY_KEY_REUSED",
    "That key was used for a different request in the last 24 hours. Use a new key for a new change.");
}
if (held.status !== null) {
  // The first call finished: answer what it answered.
  return { kind: "replay", status: held.status, response: held.response };
}

A replay on REST carries an Idempotent-Replayed: true header, so the caller can tell it apart from a fresh write. A refusal on REST is a 409. On MCP it is a tool error with its code, which the model reads and acts on. A protocol error would only reach the client software, not the model.

The case we got wrong

A row with no answer yet means a request is still running. Our first version answered every retry in that state with IDEMPOTENCY_IN_PROGRESS: try again in a moment. That is right while the first request is alive.

Now cut the first request off. A function hits its time limit, or a deploy replaces it mid-request. Either way the code that stores the answer, or frees the key, never runs. The row stays as it was, with no answer, until it expires.

Ours expired after 24 hours. So the request most likely to be retried, the one that died halfway, was the one whose retries were refused for a full day. Each retry was told to try again in a moment, and every moment said the same.

A retry key that cannot tell a slow request from a dead one punishes the caller for the server's crash.

A lease, not a lock

The fix is a lease. A claim with no answer is trusted to be running only for as long as a request could possibly run. We set the lease a little longer than the longest time limit any of our functions has, so a live request is never taken over. Past that, the claim was abandoned.

A retry that finds an abandoned claim takes it over. Two retries arriving together must not both win. One conditional update handles that:

ts
if (held.status === null) {
  const abandonedBefore = new Date(now.getTime() - LEASE_MS);
  if (held.createdAt <= abandonedBefore) {
    const taken = await db
      .update(idempotencyKeys)
      .set({ createdAt: now, expiresAt: addHours(now, 24) })
      .where(and(
        eq(idempotencyKeys.id, held.id),
        isNull(idempotencyKeys.status),                  // still no answer
        lte(idempotencyKeys.createdAt, abandonedBefore), // still that old
      ))
      .returning({ id: idempotencyKeys.id });
    if (taken.length > 0) return { kind: "new", id: taken[0].id };
  }
  throw refusal("IDEMPOTENCY_IN_PROGRESS",
    "A request with this key is still running. Try again in a moment.");
}

The where clause repeats the conditions we just read. When two updates race for the same row, Postgres makes the second wait for the first, then checks the where clause again against the row as the first left it. The first retry resets created_at and gets the row back. The second now sees a fresh created_at, matches nothing, and is told the request is in progress, which by then is true.

Taking a claim over resets its clock, so the new attempt gets a full lease of its own.

When storing the answer fails

There is a quieter way to lose the answer. The write succeeds. Then the update that stores its answer fails, because the database hiccuped at that exact moment.

What should the caller hear? Our first version returned the error and freed the key. That sounds reasonable and is the worst choice. The caller sees a failure, retries with the same key, finds it free, and makes the change a second time.

Now the write is answered as made, because it was. The key is not freed. It stays claimed with no answer, so a quick retry is told the request is in progress:

ts
async function keepAnswer(id: string, status: number, response: unknown): Promise<void> {
  try {
    await db.update(idempotencyKeys).set({ status, response }).where(eq(idempotencyKeys.id, id));
  } catch (e) {
    // The change is done, so the caller still gets its answer.
    // The key stays claimed, and lapses after the lease.
    log.error("could not keep the answer of a change that was made", e);
  }
}

Once the lease runs out, a retry can take the claim over and run again. So this case can still repeat a change, but only after a long gap, and only after two faults in a row. Freeing the key at once made a repeat the normal outcome.

When the write is refused

The opposite case is simple. If the write itself is refused, say the time was taken a second earlier, nothing changed. Keeping that refusal would stop a corrected retry from ever running under the same key. So a refusal deletes the claim, and the next try runs afresh.

A dry run never claims a key at all. It changes nothing, so there is nothing to protect.

Keeping it tidy

A stored answer can hold personal data, such as the email address of a client the call just added. So keys are not kept past their 24 hours. A claim deletes its own expired row before it inserts, so an old key is free again at once. A daily job deletes every other expired row.

Testing it

The lease is easy to get almost right, so the tests set the clock instead of waiting. A claim with no answer that is a minute old must still be in progress. One older than the lease must be taken over, on MCP and on REST, run the write once, and answer the next retry as a replay. And when storing the answer fails, the call must still be answered as made, and a quick retry must not make the change again.

Our post Idempotent webhooks: making sure the money only moves once covers the other direction: the same event reaching our server twice from a payment provider. That one is keyed on the provider's event id. This one is keyed on the caller's own key.

What we take from it

  • Claim before you run, with an insert a unique index can refuse.
  • A claim with no answer needs an end. Make it a lease just longer than any request can live.
  • Take a claim over with one conditional update that repeats the conditions you read.
  • Answer a change that was made as made, and never free its key on the way out.
  • Free the key when the write is refused, so a fixed request can be retried.
apiidempotencypostgresreliabilityengineering

Building on Bookatu?

Bookatu has a public REST API and webhooks. Have a look at the developer docs.

Developer docs