Every discussion of AI risk in business gets stuck on hallucination, which is the wrong problem to be frightened of.
A model saying something false in a chat window is embarrassing. A model saying something false that binds your company — a price, a discount, a delivery date, a confirmed booking, a refund — is a different category of event. The first is a bad answer. The second is a contract your business did not agree to, made by a system with no authority to agree to anything.
The distinction is not about accuracy. It is about authority, and it produces a rule we apply to every AI system we build:
An AI may retrieve, compose and propose. It must never be the last actor before a commitment.
That single sentence determines the architecture. Everything below is how it is enforced, drawn from systems we run in production.
Pattern 1: remove the capability, do not instruct against it
The weakest form of control is telling the model not to do something. Prompts are guidance, not a permission system, and an instruction that holds for a thousand conversations will fail on the one that matters.
The strong form is removing the ability.
In our AI voice receptionist, which books appointments in Uzbek and Russian, no tool accepts a price. Not "the model is told not to negotiate" — there is no parameter anywhere in the tool schema through which a price could enter. Services, availability and business details come from the business's own records, and that is the only path.
Similarly, the Bronjoy help bot holds no write tools at all. It can retrieve and explain. It cannot change an account, move a booking or issue anything. Its worst possible failure is a wrong explanation, which is recoverable.
The design question to ask is not "how do I stop it doing X". It is: what is the smallest set of tools that lets it do its job, and what is the worst thing that set permits?
Pattern 2: make state changes require a token the model cannot invent
If an AI can act, the action must be gated on something it could not have made up.
The receptionist cannot book a time directly. hold_slot reserves a specific time and returns a hold_token, and book_appointment will not run without it. The model cannot book a slot the calendar never offered, and two callers cannot be given the same slot.
The same idea appears in the Bronjoy MCP servers, which let assistants like Claude search and book across a marketplace of 23,727 venues. Creating an appointment requires a six-digit code sent to the customer's own phone or email, valid five minutes for one booking. The assistant never sees the code. The user reads it back.
The pattern generalises: the model proposes an action; a separate, deterministic system issues the permission; the action requires the permission. The model's confidence is removed from the causal chain.
Pattern 3: verify the claim against what actually happened
This is the one almost nobody builds, and it is the most important.
A model can perform a tool call, have it fail, and then tell the customer it succeeded. It is not lying — it is completing a sentence in the most probable way. But the customer now believes they have a booking.
So the receptionist has a claim guard. Every sentence claiming a booking, cancellation or reschedule — in Russian, Uzbek or English — is checked before it is spoken against the tool calls that actually succeeded. Without a match, the sentence is rewritten into a confirmation question, and the interception is logged.
The general rule: never let the model be the source of truth about its own actions. Compare its output against your system's record and intervene on divergence. If you build one guardrail this year, build this one.
Pattern 4: tier the risk and queue the expensive half
Not every decision needs a human, and demanding one for everything produces a system so slow nobody uses it.
Nomadeo, an AI sales agent we build for WhatsApp and Telegram in Russian, Uzbek and Kazakh, handles this with explicit tiers. Facts and commitments are extracted from conversations, risk-tiered, and reconciled against earlier ones. Low-risk facts confirm automatically. High-risk ones — anything committing the business to money, such as a price, a discount or a deadline — wait in a decision queue, where a manager confirms or rejects with a tap or a swipe, with undo. Only confirmed facts are pushed onward to the CRM, over an HMAC-signed webhook.
Around that sit nine checks on every draft before it is sent: grounding, discount authority, forbidden claims, prompt injection, personal data, stale facts, night mode, rate limits and budget. A blocked draft is recomposed or handed to a person.
The same shape appears in the Knigopoisk admin assistant, which does have real tools over orders, products, content and users. When it proposes an action needing sign-off, the run pauses, the proposal is recorded, and it resumes only after an admin approves or rejects it — leaving an audit trail of what was proposed and what was done.
Note the difference between pausing a run and asking up front. Asking up front produces consent fatigue. Pausing at the decision point produces a record.
Pattern 5: let the model propose, let something deterministic decide
The most complete version of the principle, and the one with the most upside.
We built Keystone at the AMD Developer Hackathon to test it. The task: staff scheduling with hard rules, like never putting an unqualified nurse on an ICU shift. Ask a chatbot for a roster and you get a plan that reads well, often costs less, and quietly breaks a rule somebody is legally responsible for.
The architecture splits one decision into three roles:
- Gemma reads the data, the rules in messy natural language, and the objective, and proposes a formal integer model.
- Google OR-Tools CP-SAT solves that model to proven optimality — or returns the minimal set of conflicting rules.
- An independent verifier re-checks every hard constraint with exact integer arithmetic and issues a machine-checkable certificate.
The LLM does the part it is genuinely best at — turning ambiguous human rules into structure. It never decides, and it never vouches for the answer. The same verifier grades a raw-LLM plan side by side, so you can see the difference.
That is the highest-value use of a language model in a business process: as a translator into something checkable, not as an oracle.
The same rule, in the evidence
Two more of our tools apply the principle to claims rather than actions, which is worth seeing because it is the version most companies need.
Testify audits recorded debt-collection calls against FDCPA and TCPA rules. A requirement passes only with a verbatim, timestamped quote. Missing evidence does not become a fail and does not become a pass — it becomes "insufficient evidence" and goes to human review.
Legible remediates documents for screen readers. When a vision model cannot read a value from a chart, the value stays null and is flagged, because in a workflow someone is legally accountable for, a confident wrong number is worse than no number.
Both encode the same idea: a three-state output — yes, no, do not know — is what makes an AI system usable in a regulated process. Two-state outputs force confabulation.
One incident, traced through all five patterns
Abstract principles are easy to agree with and hard to apply, so here is a single plausible failure and where each pattern would have stopped it.
The setup. A furniture retailer runs an AI assistant on WhatsApp. A customer asks about a sofa, mentions a competitor's sale, and says they will buy today if the price matches. The assistant, trying to be helpful, replies: "I can do 15% off for you today — I'll hold one in the warehouse."
Nothing about that message is a hallucination. Every word is plausible, well-written and exactly what a good salesperson might say. It is also a discount nobody authorised, on stock nobody reserved, and the customer now has it in writing.
Where it breaks, and where each pattern catches it:
| Pattern | Would it have stopped this? |
|---|---|
| 1. Remove the capability | Yes, at the root. If no tool and no prompt path can produce a percentage, the sentence cannot be composed. This is the cheapest fix and the one most teams skip because it feels restrictive. |
| 2. Gate on a token | Yes, for the reservation half. "I'll hold one" is only true if a hold was issued. No hold token, no claim. |
| 3. Verify the claim | Yes, for both halves. The outbound check compares "15% off" and "hold" against tool calls that actually succeeded, finds neither, and rewrites the sentence into a question: "Let me check what I can do on price and whether we can reserve one — one moment." |
| 4. Tier and queue | Yes, and this is the version that still closes the sale. The discount is recognised as a money commitment, goes to a manager's queue, and comes back approved or countered within minutes — with the customer told a person is confirming. |
| 5. Propose, then decide deterministically | Yes, structurally. The model proposes "offer a discount"; a pricing rule engine decides whether this customer, this product and this margin permit one. |
Notice that four of the five keep the conversation moving. The lazy reading of this article is "lock the assistant down until it is useless". The accurate reading is that a bounded assistant can be trusted with more autonomy, because the expensive outcomes are structurally unavailable to it.
Notice also the cost of the failure if none of them are in place. Not the 15% — the precedent, the customer who tells three friends the discount exists, and the fact that you will find out weeks later from an accounts query rather than from a log.
A checklist before you deploy anything
- List every tool. For each: what is the worst thing that happens if it fires wrongly?
- Remove every tool whose answer to that question is unacceptable. Do not mitigate it — remove it.
- For remaining state changes: what deterministic token gates them?
- What checks the model's claims against your system's record?
- Which outputs commit you to money? Those go in an approval queue, with an audit trail.
- Is "I do not know" a valid output? If not, you have designed a system that must invent.
- What is the handoff path to a person, and does it carry the context?
- What is the spend ceiling, and what happens at it?
That last one is not a safety question until it is. Nomadeo caps spend globally and per tenant, and past the cap the agent degrades step by step — cheaper model, then cache-only answers, then a human — rather than failing mid-conversation.
Why this is a commercial argument, not a cautious one
None of this is about slowing AI down. It is the opposite.
A system with hard boundaries can be given far more autonomy inside them, because the blast radius is known. The reason Bronjoy can expose booking to any AI assistant on the internet without an account is precisely that the verification, the limits and the claim checking all live behind the interface. The constraints are what make the openness safe.
The companies that will get the least out of AI over the next three years are the ones that either refuse to deploy it, or deploy it with no architecture and then retreat after the first incident. Both end in the same place.
Our AI integration and automation work starts at $299 with a 1–3 week delivery, and a workflow audit is the first phase for exactly this reason: the question is not what AI can do, it is which of your steps it can safely take over.
Tell us what you need. You get a fixed price and a delivery date back — free, and with no obligation to proceed.
Frequently asked questions
Can't a good prompt handle this?
A prompt reduces the frequency of a failure. It does not bound the consequence. For anything that commits you, you want a mechanism that cannot be talked out of its behaviour by a cleverly worded message — see also prompt injection.
Does all this approval slow the agent down?
Only where it should. Tiering is the whole point: the high-frequency, low-consequence work runs at full speed, and only the commitments queue. In practice customers experience a fast assistant that occasionally says "let me confirm that with someone" — which is exactly what a good junior employee does.
Who is liable if an AI promises something wrong?
Assume you are, and design accordingly. That assumption is why the approval queue and the audit trail matter more than any model choice: you want to be able to show what was proposed, what was approved, and by whom.
Is this only relevant for regulated industries?
No. Any business that quotes prices, takes bookings, promises dates or handles refunds is making commitments. Regulation changes the penalty, not the design.
What is the most common mistake you see?
Giving an assistant write access to a production system on day one because it demoed well. The demo is not where these systems fail. They fail on the unusual conversation, three months in, when nobody is watching.
