Research
Studies are runs we did ourselves, with the data and method attached. Analyses weigh the field's published evidence and say what it supports.
Studies
-
Claude Fable 5 vs Opus 5 vs Opus 4.8 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, Claude Opus 5 and Claude Opus 4.8 on identical tasks, seven areas scored, every count and caveat published.
-
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
-
Model routing and the refusal tax: a pre-registered study
On short, checkable tasks we measured no Opus 4.8-to-Fable 5 capability separation at either effort we ran, and the premium tier’s refusal ‘rescue’ silently served the cheaper model on 20 of 28 calls.
Analyses
Multi-agent failures
-
Limitations of multi-agent debate: what the paper found
What arXiv 2509.05396 found about the limitations of multi-agent debate: accuracy fell below a first-answer vote, models flipped correct answers to agree, and a correctness reward made it worse.
-
AdaMAST: an adaptive taxonomy for multi-agent failures
AdaMAST induces a failure taxonomy from an agent system's own traces: named codes on three fixed axes, no hand-authored labels. What the paper measured, and how much of the gain adaptivity supplies.
-
Multi-agent failure vocabulary: the eight terms, defined
Eight multi-agent failure terms, each keyed to the MAST failure mode underneath it, the boundary where the fault becomes visible, and the quantity you count once you instrument that boundary.
-
Multi-agent orchestration patterns and the failures they amplify
The five multi-agent orchestration patterns, supervisor, sequential-pipeline, swarm, debate, and blackboard, mapped to how errors cascade in each and the failure modes each one amplifies.
-
Multi-agent LLM failure modes, and how to contain error propagation
Why multi-agent LLM systems fail, grounded in the MAST failure taxonomy and mapped to how each failure propagates across agent topologies and the containment levers that bound the propagation radius.
-
Multi-agent systems, defined by how they fail
A multi-agent system is defined by its failure surface: agent, orchestration, coordination, shared state, and topology, each defined through the failure it enables, then routed to the research.
Propagation & containment
-
Do microservices resilience patterns port to AI agents?
Five proven microservices resilience patterns, from circuit breaker to timeout budget, mapped to their AI agent equivalents, with a judgment on how far each analogy actually holds.
-
Cascading failures in agent systems, from trigger to containment
OWASP's ASI08 files cascading failures under security. Reframed as error-propagation engineering, one fault becomes two measurable quantities, propagation radius and containment rate, per topology.
Reliability testing
-
Tool output truncation: per-framework limits and what breaks
Tool output truncation cuts a tool's return value before the model reads it. A per-runtime table of the defaults, each read from its own docs or code, plus the downstream failures a silent cut causes.
-
Why AI agents hang: timeouts, stalls, and stops that don't
Agent runs hang in three shapes: an unbounded wait, an inactivity timeout a slow stream keeps alive, and a cancellation the work declines. Verified framework defaults, and how to tell them apart.
-
How coding agents fail: a trajectory study of CLI agents
Across 1,794 annotated CLI coding-agent trajectories, the decisive error landed at a median of step 7, locked in one step later, and stayed invisible until around step 16. What that means for evals.
-
Silent failures: when agents report success and are wrong
Four 2026 papers on silent failure in AI agents: how often failed runs carry explicit success claims, why LLM judges barely beat chance at catching them, and what detects false success instead.
-
Structured output failures across models and frameworks
Structured output breaks four ways: a schema rejected at compile, the framework swapping enforcement methods, generation truncating mid-object, and output that validates while carrying a wrong value.
-
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
-
Make AI agents reliable in production by budgeting for failure
Reliable AI agents in production start with the compounding math: 0.95 per step over 20 steps is ~36% end to end. A topology-aware playbook of levers: retry, fallback, checkpoint, human-gate, degrade.
-
Agentic AI testing beyond a single eval run
Single-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.
-
How to measure agent reliability past a single pass rate
How to measure agent reliability with metrics that capture the consistency a single pass rate cannot: pass@k versus pass^k, a reliability@k suite aggregate, and a confidence interval on every rate.
-
AI agent reliability, from consistency to containment
AI agent reliability is a discipline of five properties: consistency, robustness, predictability, safety, and error propagation, with a map of where each is measured.
Eval statistics
-
Benchmark contamination: what you can check and what you cannot
Benchmark contamination is two problems under one name. Leakage inside your own splits is checkable in a browser in minutes. Corpus-level contamination is a research problem.
-
Eval statistics: which number needs which test
The statistics an agent eval rests on, routed by the question in front of you: sizing before the run, the interval on a rate, a paired test on a delta, and agreement on the labels.
-
Is your eval difference statistically significant?
Two eval runs a few points apart. Separate a real gain from run-to-run noise with a paired McNemar test on the same items: a p-value and a confidence interval on the pass-rate delta.
-
How many runs a reliable eval needs to catch a regression
How many runs a reliable eval needs is a power calculation set by the regression you must catch, your target power, and the baseline pass rate. Includes a runs-needed table and the formula behind it.
-
AI agent evaluation that follows the whole trajectory
AI agent evaluation breaks when it scores the final answer and skips the path. Evaluate the trajectory, catch early-step corruption, and report pass rates with intervals.
-
What LLM evals are, and what each type can certify
LLM eval covers four instruments: offline benchmark, LLM-as-judge, human, and online, each answering a different question, plus the benchmark-vs-product line and the rigor behind a trustworthy score.
-
LLM evals: which methods to trust and where they lie
LLM evals report whether a model passed. Whether that score is valid is a separate question. This hub maps the eval methods and the four ways an eval number lies, each routed to its fix.
Judge reliability
-
Kappa thresholds for LLM judges, and who published each one
Five published kappa bands from four sources, side by side, each with the author who wrote it, the date we read it, and the coefficient it was written for. They are conventions, and they disagree.
-
LLM judge bias vocabulary: six terms and their tests
Six terms name what goes wrong with an LLM judge: position bias, verbosity bias, self-preference, calibration, rubric drift, and the pairwise or pointwise choice beneath them. Each has its own test.
-
The LLM-judge bias checklist that gates your ranking
The LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.
-
Bias-correct your LLM-as-a-judge eval before reporting it
An LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.
-
LLM-as-a-judge bias, and the tests that catch it
LLM-as-a-judge bias is systematic, measurable distortion in an evaluator. A per-bias map pairs each bias with a detection test and a correction, so you can tell when a judge's ranking would flip.
-
Is your LLM-as-a-judge reliable? Test the evaluator
An LLM-as-a-judge is a fallible evaluator. Its reliability breaks along three axes, agreement, calibration, and bias, each with a test and a correction. This hub routes to all three.
RAG & retrieval
-
Is grep all you need? Agentic search vs vector retrieval
The PwC paper behind the grep debate: lexical search won every inline pairing on LongMemEval, lost five of ten once results moved to files, and the harness shifted accuracy as much as the retriever.
-
RAG evaluation metrics, and what each one certifies
Each RAG evaluation metric certifies one narrow property. What context precision, context recall, faithfulness, groundedness, answer relevancy and retrieval drift answer, and what each cannot see.
-
Agentic RAG architecture, and where each pattern breaks
Agentic RAG wraps retrieval in an autonomous agent that plans, routes, and self-corrects. Here is the architecture, the failure modes each pattern adds, and when classic RAG is the safer call.
-
Where RAGAS wins at RAG evaluation, and where it stops
A positioning read on RAGAS for RAG evaluation: its real strengths, a limitation its own authors flag, and the rigor gaps around confidence intervals, significance, and propagation-aware attribution.
-
How to evaluate a RAG pipeline beyond a single score
Evaluate a RAG pipeline by scoring retrieval and generation separately, putting a confidence interval on every metric, and attributing each failure to the stage that produced it.
-
RAG pipeline failure modes, and the gate that stops each one
RAG pipeline failure modes are containment failures at the retrieval-to-generation boundary: five modes mapped to how each propagates, its detection signal, and the gate that holds it.