TOOL | Eval statistics
Thumbs-down rate: what does 2% really mean?
Enter thumbs-up and thumbs-down counts beside total conversations, see the range consistent with silent conversations, and size a review that narrows it.
Two percent of conversations got a thumbs-down. What does that mean when most conversations got no rating? The page shows the range consistent with every possible rate among silent conversations, and how many reviews narrow it to the precision you need. Set your own bar, and the page tells you whether the evidence meets it, misses it, or leaves the question open. For eval-run counts, the pass-rate calculator's unscored-runs mode does the same arithmetic.
Enter the total number of conversations, between 1 and 1,000,000,000.
Off by default. When on, the link also carries the counts you typed. Rows, labels and file names stay out of every link.
How this is calculated
The worst-case bounds assume every silent conversation was satisfied (lower bound) or every silent conversation was dissatisfied (upper bound). No model of silence is needed, and no assumption about why people do or do not click.
In the "this period" view, the bounds are exact: thumbs-downs divided by all conversations at the low end, and thumbs-downs plus silent conversations divided by all conversations at the high end. In the "ongoing" view, the envelope runs from the Wilson lower bound on the lower worst-case rate to the Wilson upper bound on the upper worst-case rate, each at the entered confidence level.
The review narrows the bounds by sampling silent conversations at random. The page estimates the overall rate as thumbs-downs plus the reviewed dissatisfied share times the silent conversations, divided by total conversations. The interval maps the Wilson bounds on that share through the same formula.
The reviews-needed count inverts the Wilson interval on the dissatisfied share, using the reviewed share as the planning rate, or 50% before any review, finding the smallest sample where the half-width on the overall rate is at or under the target. The finite pool is ignored, which makes the count conservative.
Questions
Is 2% a good thumbs-down rate?
This page does not answer that, because no primary source publishes a benchmark that applies across products and populations. Set your own bar based on what dissatisfaction costs you, and the page checks whether the evidence clears it.
Is that 2% of all conversations, or of those with feedback?
The same count of thumbs-downs is a smaller share of all conversations than of conversations with feedback. In the returns-desk example, 1,000 thumbs-downs are 2.00% of all 50,000 conversations and 40.00% of the 2,500 with feedback. The page shows both and names the one you have been quoting.
Why do silent conversations matter?
In the returns-desk example, 5% of conversations have feedback and 40% of those are thumbs-downs, so thumbs-downs are 2% of all conversations. The other 95% said nothing. If every silent conversation was satisfied, dissatisfaction is 2.00%; if every one was dissatisfied, it is 97.00%. A random review of 400 silent conversations that finds 32 dissatisfied puts the estimate at 9.60% [7.44, 12.52].
How should I draw the review sample?
Draw it at random from conversations without feedback. A convenience sample (the ones easiest to find, the most recent, or the ones flagged by another signal) introduces a bias the page cannot correct. Plan the draw separately.
Why are thumbs not a random sample of satisfaction?
People choose whether to click, so the conversations with a thumb are a self-selected group. Nothing in the counts says whether the silent conversations went better or worse than the rated ones, which is why the bounds run from every silent conversation satisfied to every one dissatisfied.
How does this relate to the unscored-runs reading on the pass-rate calculator?
The unscored-runs mode uses the same arithmetic: scored only (thumbs-down divided by conversations with feedback), all attempted (thumbs-down divided by all conversations), and a bound pair (lower and upper bounds). This page adds the denominator check, the review narrowing, and the bar verdict. Use the unscored-runs mode for eval-run counts; use this page for production feedback.