OpenAI Evals is winding down. The alternatives skip the statistics.

You standardized your CI gate on OpenAI Evals, and in June the vendor pulled the rug. On June 3, 2026, OpenAI posted that its hosted Evals platform is being deprecated: read-only for everyone on October 31, and the dashboard and API shut down on November 30.¹ The migration guide it hands you points at Promptfoo, a competing open-source eval tool that OpenAI announced it was acquiring three months earlier, on March 9.² The company that wrote the eval product you standardized on now recommends a different one it just bought.

So the practical question lands on your desk this quarter: if not the hosted OpenAI product, then what. And underneath it a quieter question that outlasts any acquisition, which is whether any of the replacements actually tell you your number is real.

Pick your eval framework by the surface you are testing: Promptfoo for a declarative CI gate and red-teaming, DeepEval for Pytest-native assertions, Ragas or TruLens for RAG. But every one of them is a harness that runs tests and grades outputs. None of them puts a confidence interval on your pass rate or tells you a run-to-run difference is significant. Choose the harness for fit, then wrap it with the statistics yourself. Skip that second step and you will ship a model swap on a pass-rate move that was inside the noise the whole time.

What is OpenAI Evals in 2026?

OpenAI Evals is two things wearing one name, and the deprecation only kills one of them. The open-source openai/evals framework is a Python project, MIT-licensed, that describes itself as a framework for evaluating LLMs plus a registry of benchmarks; it still sits on GitHub at roughly 18.9k stars.³ Read the commit log before you build on it, though: the work since 2024 is housekeeping (pinning CI hooks, dropping suites with dead dependencies), and the last tagged release, 3.0.1, dates to May 2024. It is a benchmark registry you can still run, not a harness under active development.

The hosted platform, the Evals dashboard and API you configure at the OpenAI console, is the piece being retired: read-only October 31, 2026, shut down November 30.¹ The API models an eval as a task definition with a data_source_config schema and testing_criteria, a run that executes it against a model, graders like string_check that score outputs, and a report_url back to the dashboard.⁴

Read that structure closely and you have the shape of every framework below. A place to define cases, a way to run them, a grader, a report. That is a harness. It is the necessary substrate for evaluation, and it stops one step short of knowing whether the result means anything, a distinction we draw out in what an LLM eval actually is.

The alternatives: five frameworks, one shared blind spot

Here is the field as it stands in July 2026, on axes that decide which harness fits, plus the one axis that separates all of them from a statistical verdict.

The Ships-significance column is the whole argument. Every framework scores outputs. Not one of them, in its own docs, computes whether your pass-rate difference clears sampling noise. That column marks a shared boundary, drawn at the same place by all five, and it is where your own work starts.

What is DeepEval, and when is it the right harness?

DeepEval is an open-source (Apache 2.0), Pytest-native LLM evaluation framework from Confident AI. It ships more than fifty metrics covering RAG, agents, and multi-turn chat, including its own implementation of G-Eval, the criteria-in-natural-language judge.⁶ If your team already lives in pytest, writing assert_test(case, [FaithfulnessMetric()]) next to your unit tests is a real ergonomic win, and running the suite in CI is one command.

The caveat is the same one that applies to every judge-based metric, DeepEval included. G-Eval, the technique underneath many of these scorers, correlates with human ratings at a Spearman 0.514 on summarization in the paper that introduced it, better than the older BLEU and ROUGE metrics and still far from a settled measurement.¹⁰ Making the judge one line to call leaves its accuracy exactly where the paper measured it, which is exactly why the judge’s own numbers need reporting rigor and why construct validity, whether the metric measures what you think, is the question that outranks the framework choice.

Ragas vs DeepEval: specialist scores against a general harness

The two get compared constantly, and the split is clean once you name it. Ragas is a RAG-specialist metric set; DeepEval is a general Pytest harness that includes RAG metrics among fifty others. Reach for Ragas when the thing under test is a retrieval pipeline and you want its native decomposition into faithfulness, context precision, context recall, and answer relevancy, the retrieval-versus-generation split that tells you which half of the pipeline failed.⁷ Reach for DeepEval when RAG is one surface among many and you want retrieval scoring to sit in the same pytest run as your agent and conversation tests.

Neither choice resolves the harder question. Both hand you a per-example score; both leave you to decide how many examples make the average trustworthy and whether last week’s 0.82 and this week’s 0.86 are a real gain. That is a sample-size and significance problem, one the metric library leaves entirely to you and the subject of our rigorous read on Ragas and the fuller how to evaluate a RAG pipeline walkthrough.

How does TruLens evaluate RAG without a ground-truth answer?

TruLens is an MIT-licensed evaluation and tracing library whose RAG Triad scores retrieval quality when you have no gold answer for every query, the pain the other frameworks hand back to you. The triad is three reference-free feedback functions, context relevance, groundedness, and answer relevance, that together flag most RAG failure modes without a ground-truth label on each row.⁸

Context relevance asks whether the retrieved chunks fit the question, groundedness whether the answer is supported by those chunks, and answer relevance whether the answer addresses the question. TruLens stays open source and self-hostable, stewarded inside Snowflake since Snowflake acquired its originator TruEra in May 2024.⁹

The reference-free trick is genuinely useful and it is also the ceiling. Each feedback function is itself an LLM judge, so the triad inherits the judge’s calibration problem: a groundedness score of 0.9 is a model’s opinion with its own error bar, which the judge-bias literature treats as a measurement to validate, not a truth to trust.

What every framework leaves on your desk

Line the five up and the shared blind spot is stark. Each gives you a place to define cases, a runner, a grader, and a report. That is the harness, and a good harness is worth having. What none of them ships is the statistics that turn a grade into a claim.

None of them, out of the box, will tell you:

• How many runs you need before a pass rate is stable enough to gate on. A twenty-item eval and a two-hundred-item eval produce very different error bars, and the framework happily reports both to two decimals. The number of runs is a power calculation you make before the eval starts, and the harness treats it as your problem.
• Whether a difference is significant. Your suite went 0.82 to 0.86 after a prompt tweak. Real, or noise on the same items? That is a paired test, and whether your eval delta is statistically significant is a question the harness never asks.
• A confidence interval on the number. A pass rate is a point estimate from a sample. Reporting it without an interval is the single most common eval mistake, and the reason the lane treats an eval confidence interval as the unit of a defensible result.

This is the same gap that makes a green observability dashboard feel like proof when observability is not evaluation. A harness that runs and grades tests, like a trace that shows a run finished, answers a real question and stops one question short of the one you care about. Our stake: the site points toward a reliability profiler still ahead of launch that would report each reliability number with a confidence interval against field norms. That is design intent and nothing more, and because no run has produced such a number yet, this page claims none of its own.

The step no harness takes for you

Choose the harness for the surface, then own the statistics it does not.

1. Match the tool to what you are testing. Promptfoo for a declarative CI gate and red-teaming, DeepEval for Pytest-native assertions across many surfaces, Ragas or TruLens when the surface is RAG. If you are on the hosted OpenAI platform, plan your exit before October 31, 2026: Promptfoo is the migration path OpenAI itself points to, and treat the frozen openai/evals repo as a benchmark registry only.¹
2. Fix the eval set size and phrasing before you trust a number. The framework will not tell you the sample was too small; the runs-needed math will.
3. Report every pass rate as an interval, and gate model or prompt changes on a significance test, not a raw delta. That is the step none of the five frameworks do for you, and the one that stops you from shipping a swap on noise. Start with which eval methods to trust and is my eval statistically significant.

The harness question has five decent answers and one changing vendor. The rigor question has exactly one answer, and it was always going to be you.