Topic
Evaluation design
Testing agents like software: harnesses, regression suites, and what a passing run does and does not prove.
Research
- Analysis
Tool output truncation: per-framework limits and what breaks
Tool output truncation cuts a tool's return value before the model reads it. A per-runtime table of the defaults, each read from its own docs or code, plus the downstream failures a silent cut causes.
- Analysis
Why AI agents hang: timeouts, stalls, and stops that don't
Agent runs hang in three shapes: an unbounded wait, an inactivity timeout a slow stream keeps alive, and a cancellation the work declines. Verified framework defaults, and how to tell them apart.
- Analysis
How coding agents fail: a trajectory study of CLI agents
Across 1,794 annotated CLI coding-agent trajectories, the decisive error landed at a median of step 7, locked in one step later, and stayed invisible until around step 16. What that means for evals.
- Analysis
Silent failures: when agents report success and are wrong
Four 2026 papers on silent failure in AI agents: how often failed runs carry explicit success claims, why LLM judges barely beat chance at catching them, and what detects false success instead.
- Analysis
Structured output failures across models and frameworks
Structured output breaks four ways: a schema rejected at compile, the framework swapping enforcement methods, generation truncating mid-object, and output that validates while carrying a wrong value.
- Study
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
- Analysis
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Analysis
Make AI agents reliable in production by budgeting for failure
Reliable AI agents in production start with the compounding math: 0.95 per step over 20 steps is ~36% end to end. A topology-aware playbook of levers: retry, fallback, checkpoint, human-gate, degrade.
- Analysis
Agentic AI testing: the five dimensions one eval run cannot reach
How to design the test rather than pick a metric: run count and power, run-to-run variance, fault injection across topologies, and the protocol that turns five dimensions into a release decision.
- Analysis
How to measure agent reliability past a single pass rate
How to measure agent reliability with metrics that capture the consistency a single pass rate cannot: pass@k versus pass^k, a reliability@k suite aggregate, and a confidence interval on every rate.
- Analysis
AI agent reliability, from consistency to containment
AI agent reliability is a discipline of five properties: consistency, robustness, predictability, safety, and error propagation, with a map of where each is measured.
Guides
- Guide
Testing an LLM feature, from a red test to the test report
Eight moments of a tester's week, from a test that failed in CI to the test report the release is decided on, each with the page that reads it.
- Guide
Agent evals: the four practices that change
Four practices that change when you evaluate an agent: grade the end state, report pass^k (succeeds every time in k tries), isolate trials and trace the first upstream failure.
- Guide
Prove your eval suite can fail
Five checks whose right answer is known, each on a demo file and a tool, that show whether your graders, your judge and your suite can catch a broken system.
- Guide
The eval loop for an AI feature
Each step of the evaluation loop, who does what and the page that does it.
- Guide
When the model changes: the playbook
Nine steps, in order, for a retirement, an upgrade, an alias move, a reroute, a host change or a vendor switch, each with the page that helps you do it.
- Guide
Your first error-analysis session
Take a random sample from your logs, read each trace with the person who owns quality, name the failure categories, count them and export a taxonomy.
- Guide
Run every eval on every pull request and teams stop running them
A suite that runs the same checks on every PR, every night, and at every release is too slow or too thin for at least two of those jobs. Three tiers give each stage one question.
- Guide
Golden set blueprint: coding agents
Seven case families for coding agents: must-flip and must-not-break tests on separate denominators, solution leakage measured, and the test suite treated as the evaluator it is.
- Guide
Golden set blueprint: customer support
Seven case families, five slices, a coverage skeleton, and ten starter rows for testing a support agent, with a finance overlay and CSV/JSONL downloads.
- Guide
Golden set blueprint: regulated content
Seven case families for content under regulatory constraints: per-output constraint checks, risk-disclosure preservation, audience mismatch, and a pharma overlay. Not compliance advice.
- Guide
Golden set blueprint: research agents
Seven case families for research agents: citation existence, citation support, evidence strength, source coverage, and silent fabrication, each graded per claim against a named source.
- Guide
Golden set blueprint: transactional agents
Seven case families for agents that change world state: expected writes, confirmation integrity, silent tool failure, budget exhaustion, and a finance overlay.
- Guide
We scored Fable 5 vs Opus 5 vs Opus 4.8 and only the old one answered every call
While Claude Fable 5 and Claude Opus 5 are tied on our reliability benchmarks, Opus 4.8, surprisingly, is still the better choice for two specific kinds of work.
- Guide
We scored Fable 5 vs Sol vs Kimi K3 and the winner refused the most calls
Claude Fable 5 wins on points, GPT-5.6 Sol comes last and answers everything, and Kimi K3 is quick until it hangs. Pick by the failure your pipeline can absorb.
- Guide
Model fallback swaps the model unseen, and refusals log as success
When Claude Fable 5 refuses, it returns HTTP 200 with an empty content array, so error dashboards read it as a success, and with fallback on the model can swap to Opus 4.8 unseen. Instrument now.
- Guide
The OWASP AI agent security cheat sheet, read as a reliability spec
OWASP's AI Agent Security Cheat Sheet and the 2026 Agentic Top 10 name the same agent failures. Each maps to a reliability control you can test: least-privilege tools, isolated memory, bounded loops.
- Guide
AI observability proves the run finished. An eval proves it was right.
LangChain's State of Agent Engineering survey of 1,340 practitioners: 89% run observability, only 52.4% run offline evals. An offline eval is what scores whether the answer was right.
- Guide
How benchmarks get gamed, and how to check yours
BenchJack, a Berkeley auditing tool, found 219 flaws in ten popular agent benchmarks and gamed nine to near-perfect scores. Why a benchmark number is a claim about the harness, and how to check it.
Terms
-
Back-to-back testing (LLM features)
Back-to-back testing runs a reference version and the version under test on the same inputs and compares their outputs, using the reference as a pseudo-oracle. On LLM features, each disagreement is checked against how often the same cases flip on a plain rerun.
-
Chaos engineering for AI agents
Chaos engineering for AI agents is the disciplined injection of controlled faults into an agent system to measure how far each fault propagates and what fraction of it the architecture contains, each result reported with a confidence interval.
-
Fault injection (agents)
Fault injection (agents) is a controlled experiment that introduces a chosen fault at a known point in an agent topology and measures how far it propagates, so its reach and containment become recorded quantities a system can be scored on.
-
Flaky test (LLM features)
A flaky test is the tester's word for a case whose single run cannot settle a verdict. On an LLM feature, the case has a pass rate across reruns of the same version and input, and the job is to measure that rate.
-
Mutation score (LLM test suites)
A mutation score is the share of deliberately planted faults a test suite catches (kills, in the field's word), counted as killed faults over faults planted, with an interval.
-
reliability@k and pass^k
pass^k is the probability an agent solves all k runs of one task (closed form p^k). reliability@k is the lane's suite-level aggregate of pass^k: the mean across a representative task suite. It is the consistency counterpart to pass@k (best-of-k capability), not its inverse.
-
Simulated user (agent tests)
A simulated user is a model that plays the user's role in a multi-turn agent test, with its own error rate measured on a reviewed sample of conversations the same way a model judge is checked against human labels.
-
Test oracle (LLM features)
A test oracle is the source of the expected result a test compares against. On an LLM feature, a grader (a code check, a person or a model judge) applies the oracle, and every grader has its own catch rate and false alarm rate that move the observed pass rate.
Calculators
- Calculator
Irreversible Action Inventory: What Your Agent Cannot Take Back
List every action your agent can take, mark what reverses it, and name the point of no return. A written verdict per row, never a score or a budget.
- Calculator
LLM Token Counter: Context Budget and Cost per Attempt
Count tokens for a named model, see what the request reserves in the context window once output is set aside, and price one attempt at your own rates. Exact where we bundle the encoding.
- Calculator
Human Review Threshold Optimizer for Confidence-Gated Automation
Derive the human review threshold from your own records: paste stated confidence and observed correctness, price one review against one missed failure, and read the cut that costs least.
- Calculator
AI SLO Error Budget Calculator: Split a Failure Budget Across Classes
Turn a completion-rate target and a task volume into a failure budget split by class, with a Wilson interval on each class observed rate against its allowance.
- Calculator
Prompt Robustness Analyzer: Perturbation Stability for LLM Evals
Is a prompt's pass-rate drop real or noise? Paste pass/fail results for a baseline prompt and its semantically equivalent variants and read each change with a paired interval and an exact McNemar p.
- Calculator
Golden Set Version Tracker: Dataset Diff and Changelog
Diff two versions of your golden eval set in the browser, keep a dated changelog of what changed and why, and freeze the baseline metric values the next run is judged against.
- Calculator
Release Gate Designer: Go/No-Go Rules and Decision Log
Set the go/no-go rule for your eval figures, get the call and save each decision to a dated log. Export the latest as a memo.
- Calculator
Change Impact Worksheet: What to Rerun Before You Ship
Describe one change to a shipped AI system and get the rerun list it invalidates: which suites, slices and graders to run again, why each is on the list, and a change record for the ticket.
- Calculator
AI Model Deprecation Dates, Replacements and Breaking Changes
Retirement dates, vendor replacements and what breaks in your eval when you swap.
- Calculator
Agent Trace Analyzer: Step Counts, Repeated Calls, Handoff Edges
Upload an agent run trace and get step counts, repeated-call detection with index ranges, handoff edges, and per-class run tallies. Deterministic checks on the file you already have.
- Calculator
Context-of-Use Profiler: Turn a System Description into Evaluation Consequences
Describe what your AI system does, who it affects and how it fails, and get the failure families, case classes, slices and evidence expectations those answers demand.
- Calculator
LLM Latency Budget Calculator: TTFT and Token Time Across a Chain
Compose per-step time-to-first-token and token-stream latency into a chain deadline. See which step spends the budget, what kind of time it spends, and the timeout each layer should hold.
- Calculator
Evaluation Blueprint Builder: Write the Argument Before You Run the Eval
State each claim your evaluation will make, name the evidence that would support or refute it, and write the inference step connecting the two, all before any results exist.
- Calculator
Risk-to-Test Mapper: Which Failures Need Which Cases
Name the failures you care about, then see the test-case shapes, slices, evaluators and metrics each one needs. Every mapping says what it cannot establish. The mapping is a LatentEval synthesis.
- Calculator
Acceptance Criteria Builder: Set the Bar Before the Run
State each pass/fail criterion, the risk axes it covers, the reasoning behind the threshold and who set it. Get the sample size each one needs and export the rules for the release gate designer.
- Calculator
How long your eval suite adds to CI, and what a PR subset detects
Enter cases, concurrency and cadence across three tiers. The calculator reports wall-clock minutes under parallelism, monthly eval spend, and how much detection the per-PR subset gives up.
- Calculator
What is your structured output failure rate?
Paste or upload a file of model outputs and see your failure rate by class. Each rate carries a Wilson interval, and the page produces the two numbers the retry optimizer needs.
- Calculator
Should this task be an agent, a workflow, or ordinary software?
Ten questions about the task you are about to automate produce one verdict: build an agent, wire a fixed workflow, or skip the model. See which single answer you would change to flip it.
- Calculator
How much evaluation does this feature need?
Six questions about your feature, one of four evaluation tiers, the minimum setup for that tier, and the single answers that would change the verdict.
- Calculator
Error analysis workbench for LLM traces and eval results
Review traces or eval results one case at a time: mark pass or fail, code each failure, note its first failing step, and build a failure taxonomy. Your work saves to a session file you download.
- Calculator
Criteria and rubric builder for LLM eval grading
Turn failure categories into pass or fail criteria with must and must-not statements and examples, lint vague wording, and version the rubric as a file for labelers and whoever builds the checks.
- Calculator
Code grader playground: score one expected and actual pair
Paste an expected answer and the model's output, pick code graders, and see each score with its reason and near-miss hints.
- Calculator
Score a results file with code graders
Load a results file, choose code graders, see pass, fail, near-miss and disagreement counts per grader, and download a scored file, all in your browser.
- Calculator
Build a normalization pipeline and check it against your harness
Choose normalization steps in order, load a sample file, and see which rows each step flips, whether the pipeline reproduces your harness, and where the gaps are.
- Calculator
Write a CI eval gate with three verdicts and a false-block forecast
Set pass-rate floors and regression margins for PR, nightly and release tiers, pin a baseline by hash, quarantine flaky cases, and see how often the gate will block an unchanged version.
- Calculator
Model upgrade checklist
Track sixteen steps for a model or vendor switch, eight of which block it. Record each step's status, export a dated record, and re-import it to pick up where you left off.
- Calculator
Build a monitoring plan for your AI feature
Set traffic, review share, judge budget and alert thresholds. See review hours, false pages with a suggested threshold, and a canary noise floor, then export the plan as a file. No live data is read.
- Calculator
Acceptable Error Rate Calculator: The Reliability Bar Automation Needs
Invert your own manual cost, automation cost, and consequence cost into the minimum reliability that makes automating pay, and see if a measured rate clears it.
- Calculator
Eval Run Register: Preregistration and Run Cards for LLM Evals
Keep one row per eval run in this browser: freeze the plan before results exist, log what happened, and export a run card or a preregistration record for a PR.