TOOL | Evaluation design
Build a monitoring plan for your AI feature
Set your traffic, signal kinds and review share. The page computes the review hours and the rarest failure a random sample catches with 95% probability, the judge cost against a budget, the expected false pages per 30 days for each alert threshold with a suggestion that keeps them within a tolerance, and the canary set's noise floor. Gaps in the plan are listed. Export a monitoring plan file that carries the numbers and assumptions. No live data is read.
Enter traffic per day and turn on the signals you plan to watch, or load the returns-desk example to see a finished plan.
Load a plan you saved from this page as JSON, version 1.x. The page restores the form and recomputes.
Your file is processed in this browser tab, and you can check by going offline after the page loads. This page never uploads it.
Stored on this device only. Export the plan as JSON to keep a durable copy.
This browser is not keeping data for this page, so your plan is lost when you leave. Export it as JSON to keep a copy.
How this is calculated
Review load Reviews per week equals 7 times traffic per day times the review share, floored. Review hours per week equals reviews per week times minutes per review, divided by 60. The share that fits is the available hours times 60, divided by minutes per review, divided by the weekly traffic, times 100, floored to three decimals and capped at 100%.
Sight line The sight line is the chance a sample shows a failure at least once. The chance that a failure affecting p of traffic appears at least once in n randomly drawn conversations is 1 minus (1 minus p) to the n. The rarest failure seen with 95% probability is 1 minus the n-th root of 0.05, where n is the sample size: at 350 reviews, 0.85% of traffic. Both figures assume the conversations are drawn at random from all traffic. The same formula applies to the weekly review count and to the audit size.
Judge cost Judged per day equals traffic per day times the judge share, floored. Cost per day equals judged per day times calls per conversation times cost per call. Cost per 30 days equals 30 times cost per day. The share the budget affords is the budget divided by the product of 30, traffic per day, calls per conversation and cost per call, times 100, floored to three decimals and capped at 100%.
Alert false pages and the suggested threshold Traffic per window: traffic per day divided by 24 for an hour, traffic per day for a day, 7 times traffic per day for a week, each floored. For a judge signal the window traffic is traffic per day times the judge share, divided by 24 for an hour or times 7 for a week, floored. The threshold count k is the ceiling of window traffic times the threshold rate. The probability that k or more conversations with the event occur in one window, when the true rate is the baseline, is the upper tail of the binomial distribution. Expected false pages per 30 days is that probability times the number of windows in 30 days (720 for hours, 30 for days, 30 divided by 7 for weeks). The suggested threshold is the smallest count whose expected false pages sit within the tolerance. A window with more than 10,000,000 conversations is refused because the binomial tail is measured accurate only up to that size.
Catch chance The catch chance is the probability that k or more conversations with the event occur in one window when the true rate is the rate to catch, using the same binomial tail. A higher rate to catch gives a higher catch chance; a higher threshold lowers it.
Canary noise floor The noise band is z(0.975) times the square root of the rerun disagreement divided by the canary size. The detectable change at 80% power (two-sided 5%) is (z(0.975) plus z(0.80)) times the square root of the rerun disagreement divided by the canary size. Both formulas use the same definitions as the repeated-run variance planner's "Run rows" mode, so both pages print the same numbers for the same inputs at one trial per side. Expected flips per run equal the rerun disagreement times the canary size.
Questions
What does this page produce?
The page produces a static plan with planning estimates under stated assumptions. Nothing on this page connects to your production system. Export the plan, attach it to the launch review, and re-import it to revise the numbers after the first weeks.
Which signals catch failures nobody reports?
Sampled human review catches failures the user never reported. Automated code checks catch format, schema and retrieval failures on every conversation. A judge on a sample catches failures the user may not notice, such as a wrong recommendation given politely. Periodic audits catch slow-moving patterns, such as a category that changed meaning over time. Explicit feedback catches only what users choose to report. Turn on more than one signal kind so that the plan does not rely on a single source.
How do I choose a review share?
Start from what you want to see. A share of 0.25% at 20,000 conversations per day gives 350 reviews a week. The sight line shows what that sample is likely to catch: a failure on 1% of traffic appears at least once a week with 97% probability, but a failure on 0.1% of traffic shows up in those 350 reviews with only 30% probability; seeing it with 95% probability takes about 3,000 reviews. Balance the share against the reviewer's hours and the rarest failure you need to find.
Why does an hourly threshold page so often?
An hourly window holds fewer conversations, so a small count crosses the threshold more easily. At 833 conversations per hour and a baseline of 2%, the probability of 25 or more thumbs-downs in one hour is about 3.2%, which means about 23 false pages per 30 days. A daily window of the same traffic holds 20,000 conversations, where 30 daily windows produce fewer false pages than 720 hourly ones at the same threshold rate. Lengthen the window, raise the threshold or widen the tolerance. The suggested threshold shows the most sensitive count that stays within the tolerance.
How is the suggested threshold chosen?
The page picks the smallest count per window whose expected false pages sit within the tolerance you entered. A lower count catches smaller increases sooner, and a higher count pages less often.
Why must the judge be validated?
A judge that disagrees with people lets real failures through or flags passing outputs. The gap shows while Judge on a sample is on and Judge validated reads anything other than "Yes." Validate the judge against human labels, class by class: read its catch rates on the judge scorecard and record the check on the judge validation report.
What can the canary show, and what can it not?
The canary re-runs a fixed set on a schedule and compares the latest score to a pinned baseline. It catches a regression that affects the fixed cases, such as a silent model change or an infrastructure shift. It cannot catch a failure that depends on live traffic patterns, new user inputs or a distribution change that the fixed set does not cover. The canary's detectable change depends on the set size and the rerun noise; a noisy set with few cases misses a small drop.
How do confirmed failures reach the test set?
The feedback path names a person, a destination (such as a regression suite or a labeling queue) and a rule for when a confirmed failure becomes a test case. Review the path on the cadence you set. Without a feedback path, failures found in review or audit stay in a report.
Is a 2% thumbs-down rate good?
It depends on the feature and the wording. A 2% rate on a high-stakes feature with a clear prompt may be too high, while the same rate on a low-stakes assistant may be fine. When most conversations get no rating, the true dissatisfaction rate sits between a best case and a worst case. The thumbs-down rate page computes that range and how many reviews of silent conversations narrow it.