Not a confidence score

A confidence threshold routes on the wrong axis

Take the most common ticket in commerce: where is my order. Hand it to a person and watch what they do. They open the order record, read the carrier feed, check the shipping policy — the same three sources the AI already holds open. The handoff adds a queue and a wait, not accuracy, because the human path terminates in the same data.

Now the opposite case. A customer wants a refund four days outside the return window, and the model is certain about every fact: the order, the date, the item, the wording of the policy. Certainty about facts says nothing about permission to move money. The brand may allow it for a repeat customer and refuse it for a first purchase. Under a threshold design, high confidence releases the action — and the property that should have governed it was never measured.

One dial, two independent questions. Do I have the facts and am I allowed to act are not points on the same scale, and no cutoff on one of them answers the other.

Informational cases belong to the AI by design, not by score

Razviel's architecture routes a named class of case to the AI tier by design: order status and tracking, stock, product features, sales recommendation, and the brand's terms — warranty, delivery, returns, privacy. Confidence does not run on any of them. There is no such thing as low confidence on a "where is my order".

What looks like low confidence there is almost always a missing fact. A tracking number that has not scanned in three days. A package marked delivered to a customer standing in an empty doorway. The correct behavior is to keep investigating — pull the next source, read the scan history, ask the one question that resolves the ambiguity. Escalating hands a person the same gap and the same sources, one wait later.

Routing that volume to a queue costs twice. The queue fills with cases a person cannot decide better, and the cases that do need a person wait behind them. A threshold produces exactly this, because it reads hesitation about a fact as a reason to change hands.

Confidence has a job, and it is narrower than the market gives it

The narrower job is real. It lives inside the financial bucket, and starts only after a different mechanism has finished.

In the design, an action that moves money clears a deterministic gate first, and confidence never runs on what the gate refused.

It runs only on what the gate permitted, and the question it answers is narrow: not whether the action is allowed, but whether the system has earned the right to execute this class of action without a person confirming it. It cannot lift a ceiling the brand set. On a permitted action it chooses between executing and asking a human operator for a one-click confirmation on a case already assembled.

A score the model assigns to itself is the weakest part of that design

Inside that narrow job, where the number comes from matters more than the number. The usual answer is to ask the model. That answer has been measured, and it does not hold.

Guo and colleagues, at ICML in 2017, established that modern neural networks are poorly calibrated — the confidence a network reports is not the probability that it is right. For language models specifically, Xiong and colleagues at ICLR 2024 evaluated confidence elicitation across five models and five kinds of dataset, and found that models verbalizing their confidence "tend to be overconfident, potentially imitating human patterns of expressing confidence."

The mechanism is not mysterious. A verbalized confidence is generated text, produced by the same process that produced the answer. It reads like a measurement and it is an output. Wiring it to a financial control makes the safety layer a second opinion from the party being checked.

A single number also gives no reason. It cannot name the pattern it is confident about or the evidence behind it, so it cannot be audited after a bad outcome and cannot be tightened for one class of case without tightening all of them.

What replaces it in Razviel's design is a cascade in deterministic code, reading a curated record of what the operation has actually resolved. Three layers, in order.

Vetoes first, and they are binary. The policy gate did not permit the action. A recent failure exists on this pattern. The value sits above the absolute ceiling the brand set. The pattern has not accumulated enough validated history. Any one of them and the case goes to a person — no score is computed at all.

Then the contributive factors, which run only if no veto fired: how many validated successes this pattern has, how good those successes were, and how close the case in hand is to them.

Then a threshold that rises with the consequence. The bar for a small store credit and the bar for a large refund are not the same bar, and the impact tier is read from the policy cell, not guessed from the text.

The order is the whole point. In a weighted sum, a large volume of past success outweighs a failure last week — the strong factor drowns the alarm. As a veto, it cannot. And "validated" is specific: resolved and confirmed by the operation itself. History imported from the brand's previous helpdesk fills content and tone, and never buys autonomy, because it was produced by the operation being replaced.

Confidence in this design is always per pattern and per impact tier. There is no such state as "the AI is confident in general".

The asymmetry is what makes the whole thing safe to grow: autonomy rises slowly and falls fast. Rising takes accumulated validated successes on that specific pattern. Falling takes one failure, which vetoes the pattern outright rather than subtracting from a total, and the pattern goes back to asking a person until it rebuilds the record. The arithmetic behind that inequality is plain: escalating a case the system could have handled costs a confirmation click, and executing one it could not costs the brand's money.

The category is a property of the turn, not of the thread

A conversation opens as a tracking question. Three messages later it is a refund request. The ticket did not change; the case did.

The design re-classifies every incoming message from scratch — case type and risk both — and a turn never inherits the routing of the turn before it. A system that classifies once at intake is one subject change away from running a financial decision under an informational route, with no gate consulted, because the thread was labeled before the money was mentioned.

The same rule settles a request usually treated as a special case. "Let me talk to a human" is a message like any other and gets the same triage. With context, the case routes by its nature: an informational one is answered at the AI tier first, and passes to a person with the full context if the customer still wants one; a financial one without validated history goes to a person because of what it is, not because it was requested. Without context, it does not advance at all.

The takeaway

The line between AI work and human work is the nature of the case, not a confidence value. Informational cases are settled by a lookup and belong to the AI by design. Financial cases are settled by permission: a deterministic gate decides what is allowed, and confidence — computed from the operation’s own validated record, never reported by the model — decides only who executes what is already allowed. For a Shopify Plus brand, one dial becomes three mechanisms, each of which can be inspected.

What to check in any AI support design:

1. What the routing reads. Order status, tracking, stock, product questions and the brand’s terms route by their nature, with no score. A design that sends a “where is my order” to a person on low confidence is routing a missing fact to a queue.
2. What runs before confidence. An action that moves money clears a deterministic policy check first, and confidence never runs on what that check refused. A threshold that can release an action the brand’s policy does not allow is on the wrong axis.
3. Where the number comes from. A confidence the model states about itself is generated text, and the published research finds it overconfident. A usable number is computed in code, from the operation’s own validated history, per pattern and per impact tier.
4. How autonomy is lost. One recent failure on a pattern vetoes it outright and returns it to a person until it rebuilds the record. A weighted sum lets past volume outweigh last week’s failure.
5. When the case is classified. Every incoming message is re-classified from scratch, case type and risk both. A thread labeled once at intake runs a refund under an informational route the moment the subject changes.

None of that is a percentage, and all of it can be audited afterward, which a percentage cannot.