Datadog LLM Observability vs Langfuse for agent evals

Type “datadog llm observability vs langfuse” into a search engine and read the bylines. As of early August 2026, the first page of results carried two vendor-authored head-to-heads, one from Langfuse itself and one from an observability vendor selling against Datadog, plus a documentation review on a personal engineering blog and a row of directory sites that publish a page for every product pair.¹ Datadog has not published one. So the most detailed account of this matchup available to a buyer comes from one of the two vendors in it.

This choice is rarely made from a clean slate. The team asking it usually already pays Datadog for APM, and is deciding whether agent evals belong inside that contract or in a separate open-source stack. Route the question through one vendor’s homework and you inherit that vendor’s framing of it.

Pick Datadog when your organization already runs on Datadog and the requirement is agent traces and evals correlated with the APM, infrastructure, and session data you monitor there. The product now ships as Agent Observability, free to 40k LLM spans a month and $160 a month for the first 100k billed annually.²³ Pick Langfuse when self-hosting, data custody, or eval workflow depth decides it: its core is MIT licensed and free to run on your own infrastructure, with cloud tiers running from a free 50k-unit month through $29 Core and $199 Pro to a $2,499 Enterprise plan.⁴⁵ Neither platform documents a confidence interval or a significance test on the scores it stores, so whichever you pick, the statistics live outside it. Part of the evidence for that comes from earlier work: our three-way comparison of Langfuse, LangSmith, and Braintrust (AI evals) is where we read Langfuse’s eval documentation in July 2026 and re-checked it on 2026-08-02, and how Langfuse fares against those two is a question that page already owns. This one covers only the suite-versus-specialist split.

Datadog’s product is now called Agent Observability

Source-checking turns up a rename. Datadog’s docs, product page, and price list now call the product Agent Observability (the documentation still lives under the llm_observability URL path).⁶ None of the third-party comparisons in that August 2026 result set led with the new name, and our own URL keeps the old one too, because it is still what people type.

Under either name, the shape is an enterprise suite move. Agent traces land next to the APM services, infrastructure signals, and real-user sessions the rest of the company already monitors in Datadog, and the product page sells exactly that correlation.³

The eval layer has grown well past tracing: two managed evaluations run out of the box (Language Mismatch and Sensitive Data Scanning), nine LLM-as-a-judge templates cover the common checks, custom judges are defined in natural language, and external scores push in through the API.⁷ Three of the nine templates score agent behavior: Tool Selection, Tool Argument Correctness, and Goal Completeness. The other six read a single prompt and response. Tool Selection assesses whether “the tools chosen by the LLM align with the user’s request and the set of available tools”, and Goal Completeness judges whether the agent resolved the user’s intent across a full session, though Datadog’s own write-up frames it for chatbots too, calling it “especially valuable for LLM chatbot applications”.⁸ Datasets and experiments exist too, for testing prompt and model changes before they ship.⁹

Ingest keeps the decision reversible. Agent Observability accepts OpenTelemetry traces following the GenAI semantic conventions (1.37 and later) or the OpenInference conventions, sent directly over OTLP with no Datadog SDK or Datadog Agent required, alongside its own SDKs for Python, Node.js, and Java.¹⁰ Langfuse ingests the same OpenTelemetry instrumentation.¹¹ Instrument once, and the exporter endpoint is the only thing that has to change.

That covers what Datadog ships. The public record of how it stands next to Langfuse is thinner, and worth reading closely.

We checked Langfuse’s numbers on Datadog, and they held

A vendor writing the fullest public comparison of itself against a rival is a reason to check the homework, so we checked. Langfuse’s engineering piece on this exact matchup makes concrete claims about Datadog: hosted SaaS only, with no self-hosted option; a free tier of 40k LLM spans a month; $160 a month for the first 100k spans billed annually; $3.50 per additional 10k; and billing that counts LLM inference spans only, with tool, workflow, and retrieval spans free.¹¹ Against Datadog’s own price list and product page, every one of those numbers held on 2026-08-02.²³ Datadog’s own wording is two items wider than Langfuse’s summary of it: agent and embedding spans ride free as well. For anyone pricing an agent workload, the agent span is the one worth knowing about.

The authorship shows in the choice of axes. The comparison leans on self-hosting, runtime prompt serving, and code-based evaluators, which are the rows Langfuse wins, and spends less time on cross-stack correlation, which is the row Datadog exists to win.

Both halves are true at once. The framing decides which half you see first.

Langfuse itself is the open-source specialist in this pairing. Its core product capabilities (tracing, evaluations, prompt management, experiments, annotation, and the playground) are MIT licensed without usage limits; the commercial exceptions are enterprise governance modules such as SCIM, audit logging, and data-retention policies, which sit in marked ee directories behind a license key.⁴ Self-hosting is free and first-class, and the hosted cloud meters usage in units, a unit being any tracing data point: a trace, an observation, or a score.⁵

The two meters read the same agent turn differently

Pricing pages get compared on the headline number. What each platform counts moves the bill further than the headline does.

Datadog bills LLM inference spans only, so tool, workflow, agent, embedding, and retrieval spans all ride free, and retention beyond the default is a separate meter at $1.50 to $4 per 10k spans a month depending on the window you keep.² Langfuse Cloud counts every tracing data point as a unit, from $8 per 100k units past a tier’s allowance.⁵ Take a schematic agent turn with three model calls and eleven tool calls. Datadog meters the three model calls. Langfuse meters all fourteen observations, plus the trace containing them, plus every score you attach. Neither shape is cheaper in the abstract; which is cheaper for you depends on how tool-heavy your traces run and how many scores you write, so pull that ratio from your own last week of traffic before either sales conversation.

Neither meter covers the judge. Datadog’s LLM-as-a-judge evaluations, templates and custom ones alike, run on an LLM provider account you connect yourself: the setup flow has you “select the LLM provider and corresponding account to use for your LLM judge” across OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, or Vertex AI.¹² Langfuse routes its own judge the same way, through an LLM connection you configure with your own key.¹³ Both platforms orchestrate the evaluation; your model provider’s invoice pays for the tokens it burns. Both document that plainly, and it still means a budget built from either price list leaves the eval spend out.

Six stages of an agent eval, and only two have a decisive winner

Walk an agent eval from instrumentation to decision and ask who is stronger at each stage. Four of the six come out a wash.

The two decisive rows point in opposite directions, so there is no ranking to hand you.

A team that needs custody cannot buy it from Datadog at any tier, and a team that needs agent latency on the same screen as the database underneath it cannot get that from Langfuse alone. Which of those two constraints binds harder is the whole decision.

Neither platform documents the statistics under the score

Both products will store a pass rate per experiment run and chart it over time. Datadog’s experiments documentation describes creating and versioning datasets and comparing results to evaluate impact, and documents no confidence interval on that comparison and no significance test (checked 2026-08-02).⁹ We hit the same absence across Langfuse, LangSmith, and Braintrust (AI evals) when we read all three platforms’ eval documentation side by side in July 2026, and re-checking all three on 2026-08-02 left the finding unchanged. A pass rate that ticks up 3 pp between runs is, on either platform, a chart, and whether that delta is statistically significant is a question you answer somewhere else.

Somewhere else is not far. Put a confidence interval on each run’s pass rate, so a rate measured over 40 runs stops looking as solid as one measured over 400, and run a McNemar test on the paired delta when two runs scored the same items. The wider map of which number needs which test lives in our eval statistics hub. Why a tracing layer was never going to answer this sits in the line between observability and evals: a trace can certify that every step completed and still say nothing about whether the answer was right.

Who should pick which, and where this page stops

If your organization already runs Datadog and the platform team owns this decision, Datadog is the path of least resistance, and the correlation is real: agent traces beside the services and infrastructure underneath them, one vendor relationship, one bill. Its weak spots are visible right on the price list. There’s no self-hosted deployment at any tier, retention past the default is a separate meter, and the $160 rate needs an annual commitment: the same 100k spans cost $200 month-to-month and $240 on demand.²

If custody, audit, or an air-gapped deployment appears anywhere in your requirements, Langfuse is the defensible pick, and the MIT grant on the code outlasts any pricing page. Its weak spots are operational instead: self-hosting means you run the database, the upgrades, and every scaling question yourself, and the governance modules (SCIM, audit logs, retention policies) carry the commercial license. Plenty of teams split the difference and feed both platforms from one OpenTelemetry exporter, Datadog for the estate, Langfuse for the eval loop.¹¹

We verified this page against both vendors’ documentation, product pages, and price lists on 2026-08-02. That date is also the limit of it. We did not push a production workload through either platform, and docs and prices move, so treat every number here as a snapshot with a date attached. If you arrived at this question mid-migration, where Helicone users went after the Mintlify acquisition maps that adjacent decision. If the agents you’re evaluating talk out loud, the shortlist changes entirely, and our Hamming vs Cekura comparison covers that lane. Before any platform choice hardens, though, the method question comes first: what an agent eval should measure across the whole trajectory decides more than the vendor does.