Measured finding
From the first study: the premium tier's refusal rescue quietly served the cheaper model on 20 of 28 calls.
FAILURE MODES | PROPAGATION | EVAL RIGOR
AI agent reliability, measured beyond the eval.
Independent research on how agentic systems fail: how errors cascade across agents, how far they propagate, and whether the evaluations meant to certify them hold up.
- Research with confidence intervals
- Cited analyses of how agents fail
- Nineteen tools for eval statistics
The site
What's here
One question runs through every section: whether agentic systems are reliable enough to trust, and how you would know.
- 40 articles
Research
Primary studies we run ourselves, and cited analysis of what the field has actually proven.
Pre-registered | Wilson intervals - 24 guides
Guides
How to build agents that hold up, and how to tell when the one you are using does not.
Prompt injection | Observability - 45 calculators
Tools
Calculators for the statistics an eval run turns on, each one showing its work.
Sample size | Significance - 60 terms
Glossary
Plain definitions for the vocabulary the research uses, each tied to where it is measured.
Eval terms | Cited usage
Research | First study
The first study
Pre-registered before it ran, reported with confidence intervals.
In order
Essential reading
Five pieces in reading order. Start at the top if you are new here.
-
Agentic AI testing beyond a single eval run
Single-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.
-
LLM-as-a-judge bias, and the tests that catch it
LLM-as-a-judge bias is systematic, measurable distortion in an evaluator. A per-bias map pairs each bias with a detection test and a correction, so you can tell when a judge's ranking would flip.
-
We scored Fable 5 vs Opus 5 vs Opus 4.8 and only the old one answered every call
While Claude Fable 5 and Claude Opus 5 are tied on our reliability benchmarks, Opus 4.8, surprisingly, is still the better choice for two specific kinds of work.
-
Why AI sounds confident when it's wrong
A language model's confidence reads like clean handwriting: the page stays just as neat whether the claim underneath is solid or hollow. Why right and wrong arrive in the same voice.
-
AI observability proves the run finished. An eval proves it was right.
LangChain's State of Agent Engineering survey of 1,340 practitioners: 89% run observability, only 52.4% run offline evals. An offline eval is what scores whether the answer was right.
More on the site169 pages
Research 3
Analyses 37
- AdaMAST: an adaptive taxonomy for multi-agent failures
- Why AI agents hang: timeouts, stalls, and stops that don't
- Agentic AI testing beyond a single eval run
- The agent reliability testing checklist your green eval skips
- Agentic RAG architecture, and where each pattern breaks
- AI agent evaluation that follows the whole trajectory
- AI agent reliability, from consistency to containment
- Benchmark contamination: what you can check and what you cannot
- Do microservices resilience patterns port to AI agents?
- Cascading failures in agent systems, from trigger to containment
- How coding agents fail: a trajectory study of CLI agents
- How many runs a reliable eval needs to catch a regression
- How to evaluate a RAG pipeline beyond a single score
- How to measure agent reliability past a single pass rate
- Is grep all you need? Agentic search vs vector retrieval
- Is your eval difference statistically significant?
- Kappa thresholds for LLM judges, and who published each one
- Limitations of multi-agent debate: what the paper found
- Is your LLM-as-a-judge reliable? Test the evaluator
- LLM-as-a-judge bias, and the tests that catch it
- The LLM-judge bias checklist that gates your ranking
- LLM evals: which methods to trust and where they lie
- LLM judge bias vocabulary: six terms and their tests
- Multi-agent LLM failure modes, and how to contain error propagation
- Multi-agent failure vocabulary: the eight terms, defined
- Multi-agent orchestration patterns and the failures they amplify
- Multi-agent systems, defined by how they fail
- RAG evaluation metrics, and what each one certifies
- RAG pipeline failure modes, and the gate that stops each one
- Where RAGAS wins at RAG evaluation, and where it stops
- Make AI agents reliable in production by budgeting for failure
- Bias-correct your LLM-as-a-judge eval before reporting it
- Silent failures: when agents report success and are wrong
- Eval statistics: which number needs which test
- Structured output failures across models and frameworks
- Tool output truncation: per-framework limits and what breaks
- What LLM evals are, and what each type can certify
Guides for everyday use 7
- Amazon v. Perplexity: the agent-blocking ruling that didn't survive appeal
- Prompt injection by calendar invite, and how to stop it
- Is Chrome's agentic browser safe when it clicks Buy for you?
- Do you need Fable 5? A task-by-task verdict, every number sourced
- What your AI remembers, and whether it trains on your chats
- Whose side is your AI shopping agent on?
- Why AI sounds confident when it's wrong
Guides for builders 17
- Datadog LLM Observability vs Langfuse for agent evals
- We scored Fable 5 vs Opus 5 vs Opus 4.8 and only the old one answered every call
- We scored Fable 5 vs Sol vs Kimi K3 and the winner refused the most calls
- Hamming vs Cekura: voice agent eval platforms compared
- Helicone alternatives after the Mintlify acquisition
- How benchmarks get gamed, and how to check yours
- Langfuse vs LangSmith: the real split, and what all three still miss
- Model fallback swaps the model unseen, and refusals log as success
- Every tool behind one MCP gateway. One breach reaches all of them.
- SEP-2567 deletes Mcp-Session-Id, and sticky routing with it
- AI observability proves the run finished. An eval proves it was right.
- OpenAI Evals is winding down. The alternatives skip the statistics.
- Opus vs Sonnet: the routing question your task already answered
- The OWASP AI agent security cheat sheet, read as a reliability spec
- Which Claude model for your task, and what reaching up really costs
- The prompt wording is a hyperparameter you never swept.
- The best Claude model for coding is rarely the one at the top.
Calculators 45
- Acceptable Error Rate Calculator: The Reliability Bar Automation Needs
- Agent Action Risk Matrix: Which Actions Need an Approval Gate
- AI SLO Error Budget Calculator: Split a Failure Budget Across Classes
- Benchmark Rank Uncertainty Calculator: Score Intervals and Rank Ranges
- Multi-Step Agent Reliability Calculator (Independent Steps)
- Change Impact Worksheet: What to Rerun Before You Ship
- Multi-Step Agent Reliability with Shared Failures (Cascade Simulator)
- Cost per Successful Task Calculator: What One Acceptable Result Costs
- Cross-Split Overlap Checker: Find Eval Items in Both Train and Test
- LLM Eval A/B Comparator: Paired Difference with Bootstrap CIs
- Eval Budget Calculator: What an LLM Evaluation Run Will Cost
- Eval Dataset Schema Validator: Golden-Set Checks and Export Profiles
- Exact Duplicate Row Checker for Eval and Golden Datasets
- Eval Run Register: Preregistration and Run Cards for LLM Evals
- Evaluation Card Generator: Markdown and JSON on a Published Schema
- Expected Cost of Failure Calculator for AI Agents
- Golden Set Size Planner: How Many Examples an Eval Set Needs
- Golden Set Version Tracker: Dataset Diff and Changelog
- Independent Two-Proportion Test Calculator for Eval Arms
- Cohen's Kappa Calculator (Two Raters): Inter-Rater Reliability
- Irreversible Action Inventory: What Your Agent Cannot Take Back
- Judge Agreement Tracker: Log Kappa Over Time
- LLM Judge Bias Correction Calculator
- LLM Judge Calibration Calculator: Brier Score and ECE
- Judge Validation Report Builder: Report, Version Card, Drift Plan
- LLM Token Counter: Context Budget and Cost per Attempt
- McNemar Test Calculator for Paired Eval Runs
- Fleiss' Kappa, Krippendorff's Alpha and ICC Calculator
- N-Gram Overlap Checker: Find Contamination Between Two Eval Files
- Near-Duplicate Row Detector: Find Eval Rows That Differ Only in Formatting
- Paired vs Independent: Choosing an Eval Comparison Design
- Pairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
- Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
- LLM Judge Position Bias Calculator (Swap-Consistency Test)
- Prompt Robustness Analyzer: Perturbation Stability for LLM Evals
- RAG Chunk Size Calculator: Index Size and Context Budget
- RAG Retrieval Quality Calculator: Precision@K, Recall@K, nDCG, MRR, MAP
- Release Gate Designer: Go/No-Go Rules and Decision Log
- pass^k, pass@k and reliability@k Estimator: Chance That All k Runs Pass
- Repeated-Run Variance Planner: How Many Trials per Eval Case
- Reproducibility Checklist and Manifest Validator
- Retry, Verification, and Fallback Reliability Optimizer for LLM Agents
- Human Review Threshold Optimizer for Confidence-Gated Automation
- Eval Sample Size and Power Calculator
- Slice and Class Balance Analyzer: See What an Eval Set Contains
Glossary 60
- Answer coverage
- Answer relevancy
- Backpressure (agent pipelines)
- Bootstrap resampling (eval intervals)
- Bulkhead isolation (agent systems)
- Capability tier (model routing)
- Cascade resistance
- Chaos engineering for AI agents
- Circuit breaker (AI agents)
- Cohen's kappa (eval agreement)
- Consensus and voting reliability
- Construct validity (benchmarks)
- Containment rate
- Context handoff loss (agent to agent)
- Context precision (RAG evals)
- Context recall (RAG evals)
- Conversation reset (agent sessions)
- Coverage conditioning
- Effect size (eval deltas)
- Error budget (agent systems)
- Error propagation (multi-agent)
- Eval confidence interval
- Eval reproducibility
- Exponential backoff with jitter
- Failure to ask for clarification
- Failure attribution (agents)
- Failure domain (agent topology)
- Faithfulness (RAG evals)
- Fallback chain (model routing)
- Fault injection (agents)
- Graceful degradation (AI agents)
- Groundedness (LLM answers)
- Ignored agent input
- Information withholding (multi-agent)
- Inter-agent misalignment
- Judge calibration (LLM evals)
- Krippendorff's alpha (eval agreement)
- Model router
- Multi-agent debate failure
- Orchestrator-worker reliability
- Pairwise vs pointwise scoring
- Position bias (LLM judges)
- Premature termination (AI agents)
- Propagation depth
- Propagation radius (agent systems)
- Reasoning-action mismatch
- Refusal rate (LLM models)
- reliability@k and pass^k
- Retrieval drift
- Retry storm (agent systems)
- Role drift (multi-agent systems)
- Rubric drift (LLM judges)
- Self-preference bias (LLM judges)
- Specification violation (agent tasks)
- Statistical power (eval design)
- Task derailment
- Task verification failure (agents)
- Timeout budget (agent runs)
- Variance decomposition (eval runs)
- Verbosity bias (LLM judges)
Independent research, built and maintained by LatentEval. We show our methods and cite our sources. See our methodology and disclaimer.