Human Review Threshold Optimizer for Confidence-Gated Automation

The model routes each output by one rule: automate it when its stated confidence is at or above the cut, send it to human review below the cut. A reviewed item costs the review price and is assumed caught if wrong; an automated correct item costs nothing; an automated wrong item costs the error price. Both prices are in whatever unit you choose, and only their ratio moves the recommendation: doubling both costs doubles every bill without moving the cut. This is the classic reject-option tradeoff from Chow (1970), read off your observed data instead of a fitted model. A confidence cut is one gate among the several a production pipeline stacks, and where it sits alongside verifiers, retries, and escalation is set out in what holds up in a production agent system. Those neighboring policies are priced on their own terms in the retry and fallback optimizer. The review price and the error price are the same two unit costs the cost-per-successful-task calculator prices end to end; its two closer siblings size each one on its own, what one uncaught failure is worth and how low an error share has to sit before automating pays at all. All four, and the rest of the money side of running agents, sit together where we work out what agent output costs once failures, retries and review are counted.

The sweep. Expected cost is a step function of the cut: it only changes where the cut crosses an observed confidence value, and records tied on the same confidence cross together. So the tool evaluates exactly the achievable policies, one cut per distinct observed confidence plus the two corners (automate everything, review everything), and reports the cheapest. Ties and flat stretches are real; when several cuts tie, the tool takes the conservative end of the plateau, the tied cut that sends the most items to review, and says so in a note.

Worked example, small enough to check by hand. Four records with confidences 0.5, 0.6, 0.8, 0.9, where only the 0.6 record is wrong; review cost 1, error cost 10. Cutting at 0.8 reviews the 0.5 and 0.6 records and automates the other two with no errors: (2 · 1 + 0 · 10) / 4 = 0.5 per item. Automating everything ships the wrong record: (0 + 1 · 10) / 4 = 2.5. Reviewing everything costs 1 flat. The 0.8 cut halves the best single policy.

The preloaded sample. The 24 records shown on load, at review cost 2 and error cost 30, land on a cut of 0.95: 5 records automate at 1.583 per item, against 2 for reviewing everything and 10 for automating everything. The cut sits that high because one record is wrong at 0.94 stated confidence; with errors 15 times the review price, no cut below it pays. The sensitivity table shows the same data relaxing to a 0.80 cut when the error cost drops to a quarter.

Coverage and risk. Selective prediction is the setting where a model may decline an item instead of answering it; declining is called abstention, and here it means sending the item to a person. Coverage is the share of records the gate automates, and the risk paired with it is the error rate inside that automated slice, one minus the auto accuracy column; the escalation rate is coverage's complement, the share that goes to review. The sweep table is therefore the risk-coverage curve of selective prediction (El-Yaniv and Wiener, 2010) in table form, the same tradeoff used to gate deep-model predictions on an observed confidence score in practice (Geifman and El-Yaniv, 2017). The uncaught-errors column restates those same errors as a share of all records, which is the figure the cost arithmetic uses. Reading coverage and risk alongside the cost column shows what the recommended cut trades away and what a neighboring cut would buy.

Sensitivity. The error cost is usually the softest input, so the recommendation is re-derived at 0.25x, 0.5x, 2x, and 4x of the entered error cost and shown beside the entered-cost row. A cut that holds across that whole range is robust to the estimate; a cut that swings warrants pinning the error cost down before automating.

Honest limits. Everything here is an in-sample estimate. The observed correctness above each cut rests on however many records happen to sit there, and small counts move the cut around; the recommendation inherits whatever miscalibration the stated confidences carry (Guo et al., 2017), and it transfers to future traffic only as far as that traffic resembles this sample. Review is assumed to catch what it reads; imperfect reviewers make real costs higher than these figures. The measurement discipline for collecting records worth fitting a threshold on is covered in how to measure agent reliability.

Does checking pay

The second mode prices the review step itself. A check removes a failure only when it both flags the item and the reviewer then resolves it, so the chance one check removes a failure is the failure rate times the catch rate times the reviewer accuracy. Multiply that by what one uncaught failure costs and you have the loss one check removes. Against it sits what the check costs: its own price, plus the handling on the good items it flags anyway, which is the share of good items times the false escalation rate times the cost of handling one. Checking pays when the first number is larger.

The threshold, and whose algebra it is. The break-even ratio is the check price over the cost of one uncaught failure. Both sides are money, so the ratio is a plain number and it compares directly against the chance a check removes a failure. The algebraic form is equation 2 of the verification-threshold section in Cao's 2026 position paper on generative models and verification; the quantities in it are ours. That paper does not define the failure rate times catch rate times accuracy product, and it does not say the quality gap is a consequence cost. We chose both mappings, and the page prints the difference rather than borrowing the paper's authority for them. A second line adds the false alarms into the same ratio, and the two lines are identical when nothing is falsely escalated.

Where a fixed review budget goes. Slices differ, and that is the whole reason an allocation exists: under a uniform model the value per item checked does not change with the share you review, so the honest answer would be all or nothing. Given slices, the rule is a continuous knapsack. Rank each slice by net value per dollar, priced at what a review there actually costs rather than at the bare check price, and fund down the list until the budget runs out. The leftover is one review short of the next item, never slack in the rule, and the tool asserts that total spend never exceeds the budget rather than assuming it. The other well-known rule here, Neyman allocation, puts reviews where a measurement is noisiest. That answers a different question: catching failures is not estimating a rate, and we do not use it.

Intervals on every measured rate. Any of the four rates can rest on a small sample, so each one can be entered as counts instead of a percentage and carries a 95 percent Wilson interval. The net per item rises with the failure rate, the catch rate and the reviewer accuracy, and falls with the false escalation rate, so its whole range over those intervals sits at two corners. When one corner pays and the other does not, the page says the verdict is undecided rather than picking the point estimate.

Where the rates come from. The catch rate, the reviewer accuracy and the false escalation rate are all figures a judge validation run publishes, and the paste box reads that run's JSON export directly. A judge card alone carries no rates: it pins what a judge run was, not what it scored. If your confidence scores come from a judge, measure the gap first with the judge calibration calculator. When more than one reviewer scores the same items, the inter-annotator agreement calculator computes how much they agree, now with Gwet's AC1 alongside kappa; read the coefficient it returns against the published kappa and AC1 threshold bands before trusting a single reviewer's accuracy figure on its own. This mode prices checking as a policy over a whole stream; to price the next move for a single output you already hold, use the mode that compares six moves at one confidence.

Which move is cheapest

The third mode holds one output at one stated confidence and prices six moves: accept it, keep trying, switch model, run verification, escalate to a person, or drop it. Each of the first five costs what you pay to do it, plus the chance it still ends up wrong times what one wrong answer costs. Accepting has no direct cost and carries the full failure term. Dropping the task carries no failure term at all, because a dropped task ships no wrong answer. The break-even column reads the same arithmetic backwards: the accepted accuracy at which each branch stops being worth taking. The escalate-versus-cost shape is the one in a decision-theoretic characterization of LLM cascades, and it is the same inequality the second mode's threshold states, with the consequence cost read as the cost of one wrong answer.

The clamp that keeps verification honest. A check cannot recover more failure probability than is left to recover. At an accepted accuracy of 98 percent, a check that adds ten points can only add two, so its real gain is two points and not ten. Without that clamp the tool recommends verification at a cost of 1.000 while accepting costs 0.600, which is simply wrong. Both rules are on the page: one says a break-even exists at all, the other decides the branch.

Accepted accuracy, and what it is not. It is the share of your records at or above the entered confidence that were right, the same suffix the confidence-cut mode walks, so the two modes cannot disagree about it. It averages everything above the cut, so it is not the chance this one output is right: at the boundary the true rate is lower and the escalation cutoff can read early. That is why it is seeded and then yours to edit, why the cutoff carries the record count behind it, and why a thin sample is labeled indicative. The switch-model branch needs a rate you measured rather than a vendor's claim, for the reason set out in what a premium tier's refusal rescue actually served. Where a retry or a fallback is the move, that tool prices the attempt ladder in full, and fallback chain is the concept the switch-model branch stands on. This mode answers for one output at one confidence; whether running a check pays across the whole stream, and where a fixed review budget goes, is the mode that prices verification itself.

Formula: expected cost per item = reviewed share · review cost + uncaught-error share · error cost

See Sources below