Sitemap
Every page on the site, in one place: the research library, the guides, the glossary, the topic pages, the tools, and the pages behind them. Looking for the machine-readable version? Crawlers use the XML sitemap.
Main pages
12 pagesResearch
40 pages- Benchmark contamination: what you can check and what you cannot
- Kappa thresholds for LLM judges, and who published each one
- Limitations of multi-agent debate: what the paper found
- AdaMAST: an adaptive taxonomy for multi-agent failures
- Is grep all you need? Agentic search vs vector retrieval
- Tool output truncation: per-framework limits and what breaks
- Why AI agents hang: timeouts, stalls, and stops that don't
- Eval statistics: which number needs which test
- How coding agents fail: a trajectory study of CLI agents
- LLM judge bias vocabulary: the terms and their tests
- Multi-agent failure vocabulary: the terms, defined
- RAG evaluation metrics, and what each one certifies
- Silent failures: when agents report success and are wrong
- Structured output failures across models and frameworks
- Claude Fable 5 vs Opus 5 vs Opus 4.8 reliability benchmark
- Claude Fable 5 vs GPT-5.6 Sol vs Kimi K3 reliability benchmark
- Agentic RAG architecture, and where each pattern breaks
- The agent reliability testing checklist your green eval skips
- The LLM-judge bias checklist that gates your ranking
- Model routing and the refusal tax: a pre-registered study
- Make AI agents reliable in production by budgeting for failure
- Where RAGAS wins at RAG evaluation, and where it stops
- How to evaluate a RAG pipeline beyond a single score
- RAG pipeline failure modes, and the gate that stops each one
- Do microservices resilience patterns port to AI agents?
- Cascading failures in agent systems, from trigger to containment
- Multi-agent orchestration patterns and the failures they amplify
- Agentic AI testing: the five dimensions one eval run cannot reach
- How to measure agent reliability past a single pass rate
- Is your eval difference statistically significant?
- How many runs a reliable eval needs to catch a regression
- AI agent evaluation that follows the whole trajectory
- Bias-correct your LLM-as-a-judge eval before reporting it
- AI agent reliability, from consistency to containment
- LLM-as-a-judge bias, and the tests that catch it
- What LLM evals are, and what each type can certify
- Is your LLM-as-a-judge reliable? Test the evaluator
- Multi-agent LLM failure modes, and how to contain error propagation
- LLM evals: which methods to trust and where they lie
- Multi-agent systems, defined by how they fail
Guides
34 guides- Evaluation case schema, field by field
- Evidence quality for AI claims: eleven dimensions, no score
- Golden set blueprint: coding agents
- Golden set blueprint: customer support
- Golden set blueprint: RAG and knowledge assistants
- Golden set blueprint: regulated content
- Golden set blueprint: research agents
- Golden set blueprint: structured extraction
- Golden set blueprint: summarization
- Golden set blueprint: transactional agents
- Hamming vs Cekura: voice agent eval platforms compared
- Datadog LLM Observability vs Langfuse for agent evals
- Helicone alternatives after the Mintlify acquisition
- We scored Fable 5 vs Opus 5 vs Opus 4.8 and only the old one answered every call
- We scored Fable 5 vs Sol vs Kimi K3 and the winner refused the most calls
- Every tool behind one MCP gateway. One breach reaches all of them.
- The best Claude model for coding is rarely the one at the top.
- Langfuse vs LangSmith: the real split, and what all three still miss
- Opus vs Sonnet: the routing question your task already answered
- OpenAI Evals is winding down. The alternatives skip the statistics.
- Amazon v. Perplexity: the agent-blocking ruling that didn't survive appeal
- Do you need Fable 5? A task-by-task verdict, every number sourced
- SEP-2567 deletes Mcp-Session-Id, and sticky routing with it
- Buy down, not up: what reaching for the top Claude tier really costs
- Model fallback swaps the model unseen, and refusals log as success
- The OWASP AI agent security cheat sheet, read as a reliability spec
- AI observability proves the run finished. An eval proves it was right.
- What your AI remembers, and whether it trains on your chats
- The prompt wording is a hyperparameter you never swept.
- Is Chrome's agentic browser safe when it clicks Buy for you?
- Whose side is your AI shopping agent on?
- How benchmarks get gamed, and how to check yours
- Why AI sounds confident when it's wrong
- Prompt injection by calendar invite, and how to stop it
Glossary
60 terms- Answer coverage
- Answer relevancy
- Backpressure (agent pipelines)
- Bootstrap resampling (eval intervals)
- Bulkhead isolation (agent systems)
- Capability tier (model routing)
- Cascade resistance
- Chaos engineering for AI agents
- Circuit breaker (AI agents)
- Cohen's kappa (eval agreement)
- Consensus and voting reliability
- Construct validity (benchmarks)
- Containment rate
- Context handoff loss (agent to agent)
- Context precision (RAG evals)
- Context recall (RAG evals)
- Conversation reset (agent sessions)
- Coverage conditioning
- Effect size (eval deltas)
- Error budget (agent systems)
- Error propagation (multi-agent)
- Eval confidence interval
- Eval reproducibility
- Exponential backoff with jitter
- Failure attribution (agents)
- Failure domain (agent topology)
- Failure to ask for clarification
- Faithfulness (RAG evals)
- Fallback chain (model routing)
- Fault injection (agents)
- Graceful degradation (AI agents)
- Groundedness (LLM answers)
- Ignored agent input
- Information withholding (multi-agent)
- Inter-agent misalignment
- Judge calibration (LLM evals)
- Krippendorff's alpha (eval agreement)
- Model router
- Multi-agent debate failure
- Orchestrator-worker reliability
- Pairwise vs pointwise scoring
- Position bias (LLM judges)
- Premature termination (AI agents)
- Propagation depth
- Propagation radius (agent systems)
- Reasoning-action mismatch
- Refusal rate (LLM models)
- reliability@k and pass^k
- Retrieval drift
- Retry storm (agent systems)
- Role drift (multi-agent systems)
- Rubric drift (LLM judges)
- Self-preference bias (LLM judges)
- Specification violation (agent tasks)
- Statistical power (eval design)
- Task derailment
- Task verification failure (agents)
- Timeout budget (agent runs)
- Variance decomposition (eval runs)
- Verbosity bias (LLM judges)
Topics
6 topicsTools
62 pages- Evaluation and reliability calculators
- Agent Loop Cap Calculator
- Acceptable Error Rate Calculator: The Reliability Bar Automation Needs
- Eval Run Register: Preregistration and Run Cards for LLM Evals
- Expected Cost of Failure Calculator for AI Agents
- Reproducibility Checklist and Manifest Validator
- Agent Action Risk Matrix: Which Actions Need an Approval Gate
- Paired vs Independent: Choosing an Eval Comparison Design
- pass^k, pass@k and reliability@k Estimator: Chance That All k Runs Pass
- Irreversible Action Inventory: What Your Agent Cannot Take Back
- Golden Set Size Planner: How Many Examples an Eval Set Needs
- Wilson and Clopper-Pearson Pass-Rate Confidence Interval Calculator
- Repeated-Run Variance Planner: How Many Trials per Eval Case
- Eval Sample Size and Power Calculator
- Judge Validation Report Builder: Report, Version Card, Drift Plan
- Eval Budget Calculator: What an LLM Evaluation Run Will Cost
- McNemar Test Calculator for Paired Eval Runs
- Harness Migration Score Continuity Checker
- Multi-Step Agent Reliability Calculator (Independent Steps)
- Cohen's Kappa Calculator (Two Raters): Inter-Rater Reliability
- LLM Judge Bias Correction Calculator
- LLM Judge Position Bias Calculator (Swap-Consistency Test)
- RAG Chunk Size Calculator: Index Size and Context Budget
- LLM Token Counter: Context Budget and Cost per Attempt
- LLM Judge Calibration Calculator: Brier Score and ECE
- LLM Eval A/B Comparator: Paired Difference with Bootstrap CIs
- Pairwise Win Rate Calculator with Elo and Bradley-Terry Ratings
- Human Review Threshold Optimizer for Confidence-Gated Automation
- Fleiss' Kappa, Krippendorff's Alpha and ICC Calculator
- Judge Agreement Tracker: Log Kappa Over Time
- Retry, Verification, and Fallback Reliability Optimizer for LLM Agents
- Your judge disagrees with your humans. Which bias explains it?
- AI SLO Error Budget Calculator: Split a Failure Budget Across Classes
- Benchmark Rank Uncertainty Calculator: Score Intervals and Rank Ranges
- Prompt Robustness Analyzer: Perturbation Stability for LLM Evals
- RAG Retrieval Quality Calculator: Precision@K, Recall@K, nDCG, MRR, MAP
- Multi-Step Agent Reliability with Shared Failures (Cascade Simulator)
- Eval Dataset Schema Validator: Golden-Set Checks and Export Profiles
- Exact Duplicate Row Checker for Eval and Golden Datasets
- Near-Duplicate Row Detector: Find Eval Rows That Differ Only in Formatting
- Cross-Split Overlap Checker: Find Eval Items in Both Train and Test
- N-Gram Overlap Checker: Find Contamination Between Two Eval Files
- Slice and Class Balance Analyzer: See What an Eval Set Contains
- Cost per Successful Task Calculator: What One Acceptable Result Costs
- Evaluation Card Generator: Markdown and JSON on a Published Schema
- Golden Set Version Tracker: Dataset Diff and Changelog
- Release Gate Designer: Go/No-Go Rules and Decision Log
- Change Impact Worksheet: What to Rerun Before You Ship
- AI Model Deprecation Dates, Replacements and Breaking Changes
- Independent Two-Proportion Test Calculator for Eval Arms
- Context-of-Use Profiler: Turn a System Description into Evaluation Consequences
- Agent Trace Analyzer: Step Counts, Repeated Calls, Handoff Edges
- LLM Latency Budget Calculator: TTFT and Token Time Across a Chain
- Evaluation Blueprint Builder: Write the Argument Before You Run the Eval
- MCP Tool Surface Auditor
- Risk-to-Test Mapper: Which Failures Need Which Cases
- Evaluation Coverage Matrix: Which Cells Have No Cases
- Evaluator and Metric Navigator: Which Grader, Which Number
- What Does This Evaluation Prove?
- Acceptance Criteria Builder: Set the Bar Before the Run
- Golden Set Blueprint Builder: Seed a Test Set From a Task Family
- Claim-Evidence Matrix: What Backs Each Deployment Claim