Topic
Reliability testing
Testing agents like software: harnesses, regression suites, and what a passing run does and does not prove.
Research
- Analysis
Tool output truncation: per-framework limits and what breaks
Tool output truncation cuts a tool's return value before the model reads it. A per-runtime table of the defaults, each read from its own docs or code, plus the downstream failures a silent cut causes.
- Analysis
Why AI agents hang: timeouts, stalls, and stops that don't
Agent runs hang in three shapes: an unbounded wait, an inactivity timeout a slow stream keeps alive, and a cancellation the work declines. Verified framework defaults, and how to tell them apart.
- Analysis
How coding agents fail: a trajectory study of CLI agents
Across 1,794 annotated CLI coding-agent trajectories, the decisive error landed at a median of step 7, locked in one step later, and stayed invisible until around step 16. What that means for evals.
- Analysis
Silent failures: when agents report success and are wrong
Four 2026 papers on silent failure in AI agents: how often failed runs carry explicit success claims, why LLM judges barely beat chance at catching them, and what detects false success instead.
- Analysis
Structured output failures across models and frameworks
Structured output breaks four ways: a schema rejected at compile, the framework swapping enforcement methods, generation truncating mid-object, and output that validates while carrying a wrong value.
- Study
Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
The full three-way benchmark behind our builder guide. Claude Fable 5, GPT-5.6 Sol and Kimi K3 on identical tasks, eight areas scored, every interval and caveat published.
- Analysis
The agent reliability testing checklist your green eval skips
A pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.
- Analysis
Make AI agents reliable in production by budgeting for failure
Reliable AI agents in production start with the compounding math: 0.95 per step over 20 steps is ~36% end to end. A topology-aware playbook of levers: retry, fallback, checkpoint, human-gate, degrade.
- Analysis
Agentic AI testing beyond a single eval run
Single-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.
- Analysis
How to measure agent reliability past a single pass rate
How to measure agent reliability with metrics that capture the consistency a single pass rate cannot: pass@k versus pass^k, a reliability@k suite aggregate, and a confidence interval on every rate.
- Analysis
AI agent reliability, from consistency to containment
AI agent reliability is a discipline of five properties: consistency, robustness, predictability, safety, and error propagation, with a map of where each is measured.
Guides
- Guide
We scored Fable 5 vs Opus 5 vs Opus 4.8 and only the old one answered every call
While Claude Fable 5 and Claude Opus 5 are tied on our reliability benchmarks, Opus 4.8, surprisingly, is still the better choice for two specific kinds of work.
- Guide
We scored Fable 5 vs Sol vs Kimi K3 and the winner refused the most calls
Claude Fable 5 wins on points, GPT-5.6 Sol comes last and answers everything, and Kimi K3 is quick until it hangs. Pick by the failure your pipeline can absorb.
- Guide
Model fallback swaps the model unseen, and refusals log as success
When Claude Fable 5 refuses, it returns HTTP 200 with an empty content array, so error dashboards read it as a success, and with fallback on the model can swap to Opus 4.8 unseen. Instrument now.
- Guide
The OWASP AI agent security cheat sheet, read as a reliability spec
OWASP's AI Agent Security Cheat Sheet and the 2026 Agentic Top 10 name the same agent failures. Each maps to a reliability control you can test: least-privilege tools, isolated memory, bounded loops.
- Guide
AI observability proves the run finished. An eval proves it was right.
LangChain's State of Agent Engineering survey of 1,340 practitioners: 89% run observability, only 52.4% run offline evals. An offline eval is what scores whether the answer was right.
- Guide
How benchmarks get gamed, and how to check yours
BenchJack, a Berkeley auditing tool, found 219 flaws in ten popular agent benchmarks and gamed nine to near-perfect scores. Why a benchmark number is a claim about the harness, and how to check it.
Terms
-
Chaos engineering for AI agents
Chaos engineering for AI agents is the disciplined injection of controlled faults into an agent system to measure how far each fault propagates and what fraction of it the architecture contains, each result reported with a confidence interval.
-
Fault injection (agents)
Fault injection (agents) is a controlled experiment that introduces a chosen fault at a known point in an agent topology and measures how far it propagates, so its reach and containment become recorded quantities a system can be scored on.
-
reliability@k and pass^k
pass^k is the probability an agent solves all k runs of one task (closed form p^k). reliability@k is the lane's suite-level aggregate of pass^k: the mean across a representative task suite. It is the consistency counterpart to pass@k (best-of-k capability), not its inverse.