Back to engineering

Engineering

Dry runs that never write: letting an AI check a change before making it

The Bookatu engineering team7 min read

An AI app working on a business's bookings should be able to ask what a change would do before it does it. Our MCP server and REST API take a dryRun flag on writes. It reads the real records and says who would be told, without changing anything. Here is how it works, and the two ways our first version told the AI something that was not true.

The short version

  • Send dryRun: true with a write, on MCP or REST, and the server answers what the call would do instead of doing it.
  • The answer names the real record the write touches, read from this business and the caller's own column, and says who would be told.
  • A dry run with a made-up id is refused in the same words the real call would use.
  • Nothing is written or sent. No audit row is recorded and no idempotency key is claimed.
  • The flag is honoured on every write, including writes that do not advertise it.

Inside the admin, our assistant proposes a change and a person confirms it on a card. Our post Propose then confirm explains that pattern, and Undo for AI actions covers how each change records its own reverse. A connected AI app talking to our MCP server, or a script calling the REST API, has no card. It needs another way to look before it acts.

That way is a dry run. A write sent with dryRun: true is worked out but not made. The server says what the call would do, and stops there.

What a dry run answers

With made-up details, and shortened a little, a dry run of cancelling a booking answers like this:

json
{
  "dryRun": true,
  "changed": false,
  "tool": "cancel_booking",
  "wouldDo": "Cancel a booking (appointmentId: ...)",
  "record": { "id": "...", "what": "Private lesson, 60 min", "who": "Sam Lee", "when": "Thursday 12 November at 10:00" },
  "tells": "The client is not told: let them know yourself. A paid deposit is kept or refunded under the business's cancellation rules.",
  "next": "Call again without dryRun to make the change."
}

changed: false is there so no caller can mistake the answer for a result. record says what the change would land on. tells says who finds out. next says how to go ahead.

Read the real record

A preview is only useful if it is true. The cheap version of a dry run echoes the arguments back: you asked to cancel booking X, so cancelling booking X is what would happen. That tells the caller nothing new, and it says yes to calls the real write would refuse.

Ours reads the record the write names, from this business only. For a tool that acts on one booking or visit, it reads from the caller's own column only, the same filter the real call applies. If the record is not there, the dry run is refused, in the words the real call would use:

ts
// Each write that names one of these records, the argument,
// and the words its real call uses for a missing one.
const NAMED_RECORDS: Record<string, Array<[arg: string, missing: string]>> = {
  update_product: [["productId", "Product not found"]],
  update_service: [["serviceId", "Service not found"]],
  update_customer: [["customerId", "I could not find that client."]],
};

async function namedRecord(ctx: ToolContext, tool: string, args: Args) {
  for (const [arg, missing] of NAMED_RECORDS[tool] ?? []) {
    const id = String(args[arg] ?? "").trim();
    if (!id) continue;
    const row = await readers[arg](ctx.orgId, id); // this business only
    if (!row) throw refusal(missing);
    return describe(arg, id, row, ctx.canSeeClients);
  }
  return null;
}

Our first version looked records up only for bookings, visits, classes, posts and forms. For a product, a service, a client or a webhook, a dry run with a made-up id came back with record: null and the line Call again without dryRun to make the change. The real call then failed with not found. An AI that trusted the dry run would tell the person all was well, then fail in front of them.

Now every write that names one of those records is on a list, with the real call's own words for a missing one. A test fails when a new write declares such an argument and is on no list. The one write left off has a written reason: its real call does not refuse a missing record either.

A client's name appears in the answer only for a caller allowed to see clients. Anyone else gets the record without the name.

Say who is told

Before an AI acts, the question a person most wants answered is who finds out. An email to a client about a cancelled lesson is a very different thing from a quiet change to a stock count.

So every dry run says who is told, in plain words that match the tool's own published description:

  • Moving a booking: the client is told the new time.
  • Cancelling a booking as the business: the client is not told, so let them know yourself. A paid deposit is kept or refunded under the business's own rules.
  • Declining a booking request: the client is emailed, and a paid deposit is refunded.
  • Cancelling a place in a class: the client is emailed, and what they paid comes back to them as account credit.
  • Adding a webhook: events start going to an address outside the business.

Writes that tell nobody say so: Nobody outside the business is told. Some lines depend on the business. A new visit at a business that approves online bookings is held as a request until someone says yes, so the dry run reads that setting before it answers.

The quiet default was our second false answer. The dry run borrowed its lines from the in-app assistant's confirm cards. The assistant never proposes the writes that hand money back, so its cards had no line for them, and they fell through to the default. A dry run of declining a request said nobody would be told, while the real call emailed the client and refunded the deposit.

A test now reads every write's description. If it says a client is emailed or told, or that money comes back, or that a deposit may be kept, the dry run must say so too.

Change nothing, anywhere

A dry run must have no side effect, not even a small one. It is answered early in the one runner every product shares, so it never reaches the handler:

ts
export function runTool(tool: Tool, ctx: ToolContext, args: Args) {
  return withToolMark(ctx, tool, async () => {
    await assertInColumn(ctx, tool, args);
    // Before the idempotency key, the handler and the audit row.
    if (isDryRun(tool, args)) return dryRunPreview(ctx, tool, args);
    return withIdempotency(ctx, tool, args, () => runHandler(ctx, tool, args));
  });
}

No row is written. No email or text is sent. No audit row is recorded, because nothing happened. No idempotency key is claimed, so a later real call with the same key is not mistaken for a retry.

tools/list advertises dryRun on the writes where a preview helps most. The server honours it on every write. In our first version the flag was read only where it was advertised. Sent to any other write, it was dropped, and the change went ahead. A caller who says change nothing means it, whether or not we expected the question.

ts
export function isDryRun(tool: ToolSpec, args: Args): boolean {
  // Honoured on every write, advertised or not.
  return args?.dryRun === true && tool.access === "write" && !("dryRun" in tool.inputSchema.properties);
}

The last condition leaves room for a tool that has a dryRun argument of its own meaning. On REST a dry run is dryRun: true in the JSON body, or ?dryRun=true in the query string. Either way it passes the same scope and permission checks as the real call, so a dry run never describes a record the caller could not change.

Refusals an AI can read

A dry run is half of letting an AI act carefully. The other half is what happens when the real call says no. A refusal written only for a person leaves the AI guessing.

Our refusals carry a code and, where there is one, a next step. On MCP, for a client on a 2025 protocol version, they come back as tool results with isError set, so the model sees them, not only the client software. A time that was taken a moment ago comes back as BOOKING_REFUSED with the reason. A reused idempotency key comes back as IDEMPOTENCY_KEY_REUSED with Use a new key for a new change. On the REST routes the registry serves, a call refused for a missing scope names the scope and points to how-do-i, which answers with the steps and a link to the screen where a person can do it.

Trade-offs

  • A dry run is a real read, with the same checks as the real call. It is not free, and it should not be.
  • It is a snapshot. Between the dry run and the real call, the time can be taken or the client can cancel. The real call checks everything again. A dry run never holds anything.
  • Every new write needs its line saying who is told. Tests make a missing line a failing test, not a review comment.
  • Tool descriptions are now part of the contract. Tests read them, so changing a tool's wording can fail a test about its dry run. We think that is right.

What we take from it

  • A dry run that echoes its arguments is worse than none. Read the real records.
  • Refuse in a dry run exactly where the real call would, in the same words.
  • Say who is told. It is the question people care about most.
  • Honour change nothing on every write, advertised or not.
  • Give every refusal a code and a next step the model can act on.
aimcpapisafetyengineering

Building on Bookatu?

Bookatu has a public REST API and webhooks. Have a look at the developer docs.

Developer docs