Independent research on AI agent reliability: how multi-agent failures cascade and propagate, and whether the evals meant to certify them hold up statistically.https://latenteval.aienhttps://latenteval.ai/analysis/agentic-rag-architecturehttps://latenteval.ai/analysis/agentic-rag-architectureAgentic RAG wraps retrieval in an autonomous agent that plans, routes, and self-corrects. Here is the architecture, the failure modes each pattern adds, and when classic RAG is the safer call.Fri, 10 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/mcp-gatewayhttps://latenteval.ai/building-ai/mcp-gatewayAn MCP gateway fronts many MCP servers as one endpoint, brokering credentials and policy. It buys one audit trail and concentrates every tool into one process. Gateway vs proxy, when to run one.Fri, 10 Jul 2026 00:00:00 GMThttps://latenteval.ai/analysis/agent-reliability-testing-checklisthttps://latenteval.ai/analysis/agent-reliability-testing-checklistA pre-flight agent reliability testing checklist: six checks with pass criteria and calculators, from failure-mode coverage and statistical power to pass^k, cascade budget, and judge calibration.Fri, 10 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/which-claude-modelhttps://latenteval.ai/building-ai/which-claude-modelWhich Claude model should you use? A task-by-task routing guide across Opus 4.8, Sonnet 5, Haiku 4.5, and Fable 5, with a checkability rule for coding, writing, research, and agents.Fri, 10 Jul 2026 00:00:00 GMThttps://latenteval.ai/analysis/llm-as-a-judge-bias-checklisthttps://latenteval.ai/analysis/llm-as-a-judge-bias-checklistThe LLM-judge bias checklist is eight pass/fail gates you run before trusting a ranking. Each gate pairs a detection test with a numeric pass line and the calculator that computes it.Fri, 10 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/langfuse-vs-langsmithhttps://latenteval.ai/building-ai/langfuse-vs-langsmithA verified comparison of Langfuse, LangSmith, and Braintrust on licensing, self-hosting, tracing, evals, and pricing shape, plus the statistical eval layer none of the three computes for you.Thu, 09 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/opus-vs-sonnethttps://latenteval.ai/building-ai/opus-vs-sonnetClaude Opus 4.8 lists at 1.67x Sonnet 5, yet on checkable work our routing eval measured no separation between them. A table for routing Opus, Sonnet, Haiku, and Fable 5 by task risk and cost.Thu, 09 Jul 2026 00:00:00 GMThttps://latenteval.ai/research/model-routing-refusal-taxhttps://latenteval.ai/research/model-routing-refusal-taxOn short, checkable tasks we measured no Opus 4.8-to-Fable 5 capability separation at either effort we ran, and the premium tier’s refusal ‘rescue’ silently served the cheaper model on 20 of 28 calls.Tue, 07 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/openai-evalshttps://latenteval.ai/building-ai/openai-evalsOpenAI is deprecating its hosted Evals platform and steering users to Promptfoo, which it is acquiring. What OpenAI Evals, DeepEval, Ragas, TruLens, and Promptfoo each do, and the rigor none ship.Sat, 04 Jul 2026 00:00:00 GMThttps://latenteval.ai/using-ai/amazon-perplexity-agent-rulinghttps://latenteval.ai/using-ai/amazon-perplexity-agent-rulingYou told your shopping agent yes, and Amazon wants a court to say no; it won that order against Perplexity's Comet, and the Ninth Circuit has not yet ruled on the appeal.Fri, 03 Jul 2026 00:00:00 GMThttps://latenteval.ai/using-ai/do-you-need-fable-5https://latenteval.ai/using-ai/do-you-need-fable-5Claude Fable 5 is included on Pro, Max, and Team through July 7, then metered at twice Opus 4.8's price. Which tasks justify switching it on, and which are better left on Opus 4.8.Fri, 03 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/mcp-goes-statelesshttps://latenteval.ai/building-ai/mcp-goes-statelessRoots, Sampling, and Logging are only annotation-deprecated; SEP-2567 deletes Mcp-Session-Id and sticky routing on July 28 with no grace period. Audit the session layer first.Fri, 03 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/log-refusals-and-model-fallbackshttps://latenteval.ai/building-ai/log-refusals-and-model-fallbacksWhen Claude Fable 5 refuses, it returns HTTP 200 with an empty content array, so error dashboards read it as a success, and with fallback on the model can swap to Opus 4.8 unseen. Instrument now.Thu, 02 Jul 2026 00:00:00 GMThttps://latenteval.ai/building-ai/routing-claude-models-by-taskhttps://latenteval.ai/building-ai/routing-claude-models-by-taskOn short work you can check, our routing eval found no Opus-to-Fable capability separation at both effort levels. Buy down: Fable 5's 2x buys a refusal tax and a fallback to the cheaper model.Thu, 02 Jul 2026 00:00:00 GMThttps://latenteval.ai/analysis/reliable-ai-agents-in-productionhttps://latenteval.ai/analysis/reliable-ai-agents-in-productionReliable AI agents in production start with the compounding math: 0.95 per step over 20 steps is ~36% end to end. A topology-aware playbook of levers: retry, fallback, checkpoint, human-gate, degrade.Tue, 30 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/ragas-rigorous-rag-evaluationhttps://latenteval.ai/analysis/ragas-rigorous-rag-evaluationA positioning read on RAGAS for RAG evaluation: its real strengths, a limitation its own authors flag, and the rigor gaps around confidence intervals, significance, and propagation-aware attribution.Tue, 30 Jun 2026 00:00:00 GMThttps://latenteval.ai/building-ai/owasp-ai-agent-security-cheat-sheethttps://latenteval.ai/building-ai/owasp-ai-agent-security-cheat-sheetOWASP's AI Agent Security Cheat Sheet and the 2026 Agentic Top 10 name the same agent failures. Each maps to a reliability control you can test: least-privilege tools, isolated memory, bounded loops.Tue, 23 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/how-to-evaluate-rag-pipelinehttps://latenteval.ai/analysis/how-to-evaluate-rag-pipelineEvaluate a RAG pipeline by scoring retrieval and generation separately, putting a confidence interval on every metric, and attributing each failure to the stage that produced it.Mon, 22 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/rag-pipeline-failure-modeshttps://latenteval.ai/analysis/rag-pipeline-failure-modesRAG pipeline failure modes are containment failures at the retrieval-to-generation boundary: five modes mapped to how each propagates, its detection signal, and the gate that holds it.Mon, 22 Jun 2026 00:00:00 GMThttps://latenteval.ai/building-ai/observability-is-not-evalshttps://latenteval.ai/building-ai/observability-is-not-evalsLangChain's State of Agent Engineering survey of 1,340 practitioners: 89% run observability, only 52.4% run offline evals. An offline eval is what scores whether the answer was right.Wed, 17 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/cascading-failure-microservices-to-agentshttps://latenteval.ai/analysis/cascading-failure-microservices-to-agentsFive proven microservices resilience patterns, from circuit breaker to timeout budget, mapped to their AI agent equivalents, with a judgment on how far each analogy actually holds.Wed, 17 Jun 2026 00:00:00 GMThttps://latenteval.ai/using-ai/what-your-ai-remembershttps://latenteval.ai/using-ai/what-your-ai-remembersYou flipped the switch you could see. The other one is still on.Wed, 17 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/cascading-failures-agent-systemshttps://latenteval.ai/analysis/cascading-failures-agent-systemsOWASP's ASI08 files cascading failures under security. Reframed as error-propagation engineering, one fault becomes two measurable quantities, propagation radius and containment rate, per topology.Wed, 10 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/multi-agent-orchestration-patternshttps://latenteval.ai/analysis/multi-agent-orchestration-patternsThe five multi-agent orchestration patterns, supervisor, sequential-pipeline, swarm, debate, and blackboard, mapped to how errors cascade in each and the failure modes each one amplifies.Wed, 10 Jun 2026 00:00:00 GMThttps://latenteval.ai/building-ai/the-lazy-prompter-problemhttps://latenteval.ai/building-ai/the-lazy-prompter-problemRewording the same task swings a model's pass rate: format, option order, even a 'please'. A one-phrasing eval samples one point from a spread you never measured. Pin the prompt and measure it.Wed, 10 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/agent-reliability-testinghttps://latenteval.ai/analysis/agent-reliability-testingSingle-run eval samples agent reliability once. Rigorous testing measures it across many runs with confidence intervals, statistical power, pass^k, and fault injection for cascade propagation.Wed, 03 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/how-to-measure-agent-reliabilityhttps://latenteval.ai/analysis/how-to-measure-agent-reliabilityHow to measure agent reliability with metrics that capture the consistency a single pass rate cannot: pass@k versus pass^k, a reliability@k suite aggregate, and a confidence interval on every rate.Wed, 03 Jun 2026 00:00:00 GMThttps://latenteval.ai/analysis/is-my-eval-statistically-significanthttps://latenteval.ai/analysis/is-my-eval-statistically-significantTwo eval runs a few points apart. Separate a real gain from run-to-run noise with a paired McNemar test on the same items: a p-value and a confidence interval on the pass-rate delta.Wed, 03 Jun 2026 00:00:00 GMThttps://latenteval.ai/using-ai/chrome-ai-browsing-safetyhttps://latenteval.ai/using-ai/chrome-ai-browsing-safetyGoogle's auto browse lets an AI click and type across your tabs, in a US-only paid preview. The same agent obeys instructions hidden on the pages it reads. How to keep a hand on it.Wed, 27 May 2026 00:00:00 GMThttps://latenteval.ai/using-ai/whose-side-is-your-shopping-agent-onhttps://latenteval.ai/using-ai/whose-side-is-your-shopping-agent-onYour AI shopping agent has your card and picks what you buy. Google, Amazon, and Perplexity all ship one now, and when a model carries a sponsor's incentive, it steers. Learn whose side yours is on.Wed, 27 May 2026 00:00:00 GMThttps://latenteval.ai/building-ai/how-benchmarks-get-gamedhttps://latenteval.ai/building-ai/how-benchmarks-get-gamedBenchJack, a Berkeley auditing tool, found 219 flaws in ten popular agent benchmarks and gamed nine to near-perfect scores. Why a benchmark number is a claim about the harness, and how to check it.Sat, 16 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/how-many-runs-for-a-reliable-evalhttps://latenteval.ai/analysis/how-many-runs-for-a-reliable-evalHow many runs a reliable eval needs is a power calculation set by the regression you must catch, your target power, and the baseline pass rate. Includes a runs-needed table and the formula behind it.Sat, 16 May 2026 00:00:00 GMThttps://latenteval.ai/using-ai/why-ai-sounds-confident-when-wronghttps://latenteval.ai/using-ai/why-ai-sounds-confident-when-wrongA language model's confidence reads like clean handwriting: the page stays just as neat whether the claim underneath is solid or hollow. Why right and wrong arrive in the same voice.Sat, 16 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/ai-agent-evaluationhttps://latenteval.ai/analysis/ai-agent-evaluationAI agent evaluation breaks when it scores the final answer and skips the path. Evaluate the trajectory, catch early-step corruption, and report pass rates with intervals.Sat, 09 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/reporting-llm-as-a-judge-evaluationshttps://latenteval.ai/analysis/reporting-llm-as-a-judge-evaluationsAn LLM judge is an imperfect classifier, so its raw pass rate is biased. Correct it with the judge's sensitivity and specificity, then report a calibration-aware confidence interval.Sat, 09 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/ai-agent-reliabilityhttps://latenteval.ai/analysis/ai-agent-reliabilityAI agent reliability is a discipline of five properties: consistency, robustness, predictability, safety, and error propagation, with a map of where each is measured.Sat, 02 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/llm-as-a-judge-biashttps://latenteval.ai/analysis/llm-as-a-judge-biasLLM-as-a-judge bias is systematic, measurable distortion in an evaluator. A per-bias map pairs each bias with a detection test and a correction, so you can tell when a judge's ranking would flip.Sat, 02 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/what-are-llm-evalshttps://latenteval.ai/analysis/what-are-llm-evalsLLM eval covers four instruments: offline benchmark, LLM-as-judge, human, and online, each answering a different question, plus the benchmark-vs-product line and the rigor behind a trustworthy score.Sat, 02 May 2026 00:00:00 GMThttps://latenteval.ai/analysis/llm-as-a-judgehttps://latenteval.ai/analysis/llm-as-a-judgeAn LLM-as-a-judge is a fallible evaluator. Its reliability breaks along three axes, agreement, calibration, and bias, each with a test and a correction. This hub routes to all three.Sat, 18 Apr 2026 00:00:00 GMThttps://latenteval.ai/analysis/multi-agent-failure-modeshttps://latenteval.ai/analysis/multi-agent-failure-modesWhy multi-agent LLM systems fail, grounded in the MAST failure taxonomy and mapped to how each failure propagates across agent topologies and the containment levers that bound the propagation radius.Sat, 18 Apr 2026 00:00:00 GMThttps://latenteval.ai/using-ai/calendar-invite-prompt-injectionhttps://latenteval.ai/using-ai/calendar-invite-prompt-injectionYour AI reads your calendar, and a stranger's invite can quietly hand it orders you never see.Sat, 18 Apr 2026 00:00:00 GMThttps://latenteval.ai/analysis/llm-evalshttps://latenteval.ai/analysis/llm-evalsLLM evals report whether a model passed. Whether that score is valid is a separate question. This hub maps the eval methods and the four ways an eval number lies, each routed to its fix.Tue, 07 Apr 2026 00:00:00 GMThttps://latenteval.ai/analysis/multi-agent-systemshttps://latenteval.ai/analysis/multi-agent-systemsA multi-agent system is defined by its failure surface: agent, orchestration, coordination, shared state, and topology, each defined through the failure it enables, then routed to the research.Tue, 07 Apr 2026 00:00:00 GMT