The allowlist is the boundary: keeping an AI assistant on rails

The most useful question to ask about an AI assistant is not what it can do, it is what it cannot. Ours can do a short list of things. Read your catalog, draft a new product or service, receive stock, and book, reschedule or block time. That list is the whole of its power. It is written down, it is small, and everything outside it is simply unavailable.

We did not arrive at a small list by trimming a large one. We started from nothing and added tools one at a time, each considered on its own. The default state for any capability in the product is that the assistant cannot touch it. To make something reachable, someone has to put it on the list on purpose.

Allowlist, not denylist

It is tempting to frame safety as a set of prohibitions. Let the model do anything except delete records, except change roles, except touch billing. That framing is backwards, and it is dangerous for a reason that has nothing to do with how clever the model is. A denylist is a list of the bad things you have thought of so far. The first bad thing you forget is a hole. Worse, every new feature you ship is reachable by default, so the assistant grows more powerful every time the product does, whether or not anyone meant it to.

An allowlist fails the safe way. A new feature is invisible to the assistant until a person decides it belongs on the list. If we forget to add a capability, the cost is that the assistant cannot do something useful yet, and an owner asks us for it. If we forget to remove a capability from a denylist, the cost is that the assistant can do something it never should. One of those failures is a feature request. The other is an incident.

A denylist asks you to imagine every mistake in advance. An allowlist only asks you to approve each thing you actually want.

The boundary is code, not a prompt

You can tell a model, in plain words, to stay away from billing. You should. But an instruction in a prompt is a request, not a wall. The wall is the dispatcher that sits between the assistant and the rest of the system. It holds the map of allowed tools. If the assistant asks for a tool that is not in the map, the dispatcher returns an error. It does not improvise, and it does not fall back to something close.

// The allowlist, as a map from name to handler. This map is the
// boundary. A tool that is not in here cannot be called, no matter
// what the model asks for.
const TOOLS = {
  list_catalog: listCatalog,
  create_product: proposeCreateProduct,
  update_service: proposeUpdateService,
  receive_stock: proposeReceiveStock,
  book_appointment: bookAppointment,
  reschedule_appointment: rescheduleAppointment,
  block_time: blockTime,
} as const;

async function dispatch(name: string, args: unknown, ctx: OrgContext) {
  const tool = Object.prototype.hasOwnProperty.call(TOOLS, name)
    ? TOOLS[name as keyof typeof TOOLS]
    : undefined;
  if (!tool) throw new ToolNotAllowed(name); // unknown tool, full stop
  return tool(args, ctx);
}

Notice that there is no branch for handling an unknown tool gracefully. There is nothing to handle. A name that is not a key in the map is not a tool, and the request ends there. The model can name anything it likes, and the only names that mean something are the ones we chose.

What is never a tool

Some capabilities are kept off the list on principle, not because they are hard to build, but because they must always run through a person. Anything that grants access or changes who can do what, like inviting staff, changing a role, or adjusting permissions. Anything that moves money or changes how it is collected, like payouts, plan changes, or refunds. And deletes of anything at all. The assistant can archive a draft it just proposed, but it cannot erase a client, an appointment, or a record of a sale.

These are exactly the actions where a confident wrong answer is expensive and hard to walk back. A mistaken product name is a quick fix. A mistaken permission change is a security problem, and a mistaken payout is real money in the wrong place. So those doors do not open from the chat at all. They live in the admin, where a person clicks through them with the full context on screen.

The allowlist is not the only check

Being on the list makes a tool callable. It does not make every call allowed. Each tool still runs the same org-scoping and permission checks the manual UI runs. The list decides what could be called. The per-call checks decide whether this caller, in this business, with this role, may do it right now. A staff member's assistant sees the tools a staff member is allowed, not an owner's.

Reads are fenced the same way. Even listing the catalog is scoped to the tenant on the session and limited to the kind of data the assistant has any business seeing. It reasons over products and services, not staff records or payout details. A tool that reads is still a tool, and it lives on the same list as the ones that write.

A small surface is the point

People sometimes read the short tool list as a limitation we have not gotten around to fixing. It is the opposite. A small surface is one you can hold in your head and test end to end. Every tool we add is a thing we have to reason about forever, so we add them slowly and we say no often. The assistant feels capable not because it can do everything, but because the few things it can do are the things owners actually ask for, and each one is safe by the time it ships.

If someone asks what our AI assistant is allowed to do, the honest answer fits on one page, because it is a list, and the list is the boundary. Everything else is not blocked, exactly. It simply was never wired up, and that is the safest kind of no there is.