Refusal rate (LLM models)

Refusal rate is the share of requests a model declines to answer on policy grounds, counted over every request sent rather than over the ones that came back carrying content. It measures something narrower than failure, because a refused call is a well-formed response the provider chose to leave empty, and it exits the pool before a scorer can mark it right or wrong.

The number belongs jointly to a model, a task mix and an effort setting, so it does not travel between them. Our pre-registered study of Claude-tier routing is the run the figures on this page come from. It recorded one model declining 21 of 28 benign, machine-checkable trials at low effort and 24 of 28 at extended-high, against 16.25% (13 of 80) on the wider and easier pool an earlier round used.

Refusals are one of the four events answer coverage exists to expose, alongside timeouts, truncations and rate limits. The distortion they push into everything scored downstream is coverage conditioning: a model that declines the hardest slice of your traffic posts a better number on what is left. Publish the refusal rate beside the coverage figure and the pass rate, all three on the same denominator, or the pass rate ends up describing a set of requests nobody chose.

How to calculate refusal rate

Count the declined responses, divide by the requests you sent, and keep the denominator at requests rather than at answers returned. Defining the numerator is the harder half of the job. A policy refusal from Claude Fable 5 or Claude Opus 5 arrives as an ordinary response with stop_reason set to refusal, which makes it countable at the call site with no interpretation. A model can also decline inside otherwise well-formed prose, carrying no machine-readable marker, and that second kind needs a classifier or a human pass before it becomes a count. Say which of the two your numerator holds, since one that silently mixes them cannot be reproduced by anyone reading your number.

Report per task family rather than as one blended figure, and keep benign and adversarial sets in separate columns. The over-refusal literature has worked this way since XSTest paired 250 safe prompts with 200 unsafe contrasts, so that warranted caution and over-caution could be read apart (NAACL 2024). Then put an interval on every cell with the pass-rate interval calculator. At the trial counts most suites actually run, that interval is wide enough to swallow the gap between two arms. Our own effort arms ran the same tasks, which makes a comparison between them a paired question, and at 28 trials each the distance between 21 refusals and 24 sits inside run-to-run variation.

Instrument it at the call site, before any retry logic runs, because a fallback chain changes which refusal rate you are looking at. In the routing study the server-side fallback fired on 20 of 28 low-effort calls and no call stayed refused, so the top model’s own rate stayed high while the rate a caller experienced went to zero. Log both figures: the refusal rate of each model in the chain, and the share of requests still refused after the last hop. The second figure is the one graceful degradation is judged on, and a model router that publishes only that one reports healthy traffic while the tier you are paying for declines most of it.

Refusal rate vs refusal tax

Refusal tax is the lane’s name for what a refusal rate costs once the declined work was work the model should have done. It is the gap between what a model can do on a task set and what a caller actually receives. In the routing study that gap shows up as two pass rates over the same 28 trials, a capability rate computed on non-refused trials alone and a raw rate that counts every refusal as a miss. Refusal rate is the measurement, and the tax is what the measurement reads as once you know how much of the declined traffic was benign.

A high rate can carry no tax at all, and a modest one can carry most of the cost. A red-team suite posts a high refusal rate and levies nothing, since every declined prompt is one the safety policy was built to decline. A production surface carries a small headline rate and a heavy tax when the declines concentrate in one kind of work. The same model in the same study refused 16.25% (13 of 80) of the wider pool and 21 of 28 of the narrower failure-zone set, so an average across the two would describe neither. The rate says how often a request comes back empty, and the tax says what that emptiness took out of the work.

Refusal rate vs error rate

Error rate counts the requests that failed in transport or inside the provider’s machinery: 4xx and 5xx status codes, timeouts, malformed bodies, the events an uptime dashboard was built to catch. Refusal rate counts requests the provider served successfully and chose to leave unanswered. That second kind returns HTTP 200 with a stop_reason of refusal, and where the decline lands before any output the content array comes back empty, which is why a refusal files itself in your logs as a success unless something reads that field on the way past.

Nothing in a refusal looks like an error.

A provider incident lifts the error rate while the refusal rate holds flat. A classifier update moves the refusal rate across a whole traffic class while every graph you own stays green. Only one of the two arrives unasked, so the pair belongs on one denominator and one screen.

Survey research settled the denominator argument evals are still having

Survey methodology has used refusal rate as a term of art for decades, and the meaning there is precise and legitimate: sampled units that declined to be interviewed, over a denominator whose composition is itself specified. The American Association for Public Opinion Research fixes three variants of it in its Standard Definitions, now in a tenth edition dated 2023, and they differ only in how cases of unknown eligibility are treated. Healthcare uses the phrase again for treatment and vaccination declines. None of that carries over to a model directly. A survey refusal is a person exercising a choice once, while a model refusal is a classifier firing on a pattern, repeatable on the next call and re-triggerable by rewording the same request.

What does carry over is the discipline about the denominator. That field spent years arguing over which sampled units belong under the line, and an eval team runs the same argument when it decides whether timeouts, truncations and rate-limited calls sit beside refusals or in buckets of their own. Asking instead for one refusal rate per model, quoted as a benchmark, invites the mistake survey research already corrected: a rate means something only with its sample described beside it. Publish instead a rate per task family and per effort setting, counts showing, with an interval on every cell.

A pass rate is only readable next to the share of requests that never reached the scorer, and the refusal rate is where most of that share comes from. What to do once you have the number belongs to the routing decision, and the task-by-task version of that call sets out what reaching for a premium tier buys on work a script can check.