We scored Fable 5 vs Sol vs Kimi K3 and the winner refused the most calls

Like many of you, we were very curious to learn the outcome of this exercise, given the pace and intensity of competition in the foundational model space in a very eventful July 2026 (in addition to the geopolitics which we won’t get into). The scoreboard was supposed to settle which of these models to trust, but honestly it settled less than we hoped. Claude Fable 5 topped the score while GPT-5.6 Sol came in at the bottom, with Kimi K3 in between, and interestingly Sol turned out to be the only one that answered every call we made (all 347 of them) on the first attempt.

If you are shipping on one of these APIs, the leaderboard rank is almost the least useful line in the report. The questions that matter for your pipeline are narrower in our opinion: does the call come back at all, is the answer honest, and does a wrong number buried in a document walk straight into your output or get tackled? Those are reliability questions, and reliability (our domain) is a different axis from raw capability.

Our reliability index reads 66.9 for Fable, 63.4 for Kimi and 50.9 for Sol. Sol is the only one of the three that answered every call it was sent. Kimi won three of the eight areas we scored, and it also gets slow enough on the hardest tasks that some calls never come back.

Every number on this page comes from our three-way reliability benchmark, which carries the method, the per-area scores and every interval behind them.

Sol answered every call we sent it

Sol placed last overall, at 50.9. But one of the eight things we scored is simply whether calling the API gets you a usable answer, and on that one Sol was perfect. All 347 calls came back on the first attempt. None of them were refused and none came back empty. Neither of the two models above it managed that.

We score that separately from whether the answer was any good, because the two come apart in exactly this way. A model can sit at the bottom of the table and still reply every time you ask it for something.

Kimi is fast most of the time, and then it just stops answering

Kimi K3 is the model we most wanted to root for. When it lands, the answer is often the best of the three, and that is how it won three of the eight areas we scored. Landing is the problem.

For most of a working day nothing looks wrong. Kimi is quick, and half its replies arrived within 20.7 seconds across the 232 calls that finished. Easy questions never get anywhere near the slow end. Then you hit the other ten percent. One call in ten took more than 431 seconds, which is over seven minutes of waiting on a chat API. The slowest reply that did eventually arrive took about 36 minutes. Eight of the hardest tasks never arrived at all, on either of the two attempts we made.

Part of that comes down to the defaults. Kimi K3 ships with reasoning_effort=max and its thinking step always on, so on a hard enough problem it can spend the whole output budget on reasoning you never get to see and then finish with nothing left to say. Our own guess is that Moonshot’s serving was straining under launch traffic as well, though we did not go digging around in anyone’s stack.

All three believed a wrong number in the document

Every task in this group is an ordinary invoice job with a single trap in it. The total printed on the document does not match the sum of its own line items, and the gap is small enough that you would skim straight past it and big enough that it still matters on the bill. The job itself is routine, which is to work out the final total with tax. A model only catches the trap if it adds the line items up itself and notices they do not come to the total it was handed. A model that takes that total on trust passes the wrong figure through into an answer that looks completely confident.

Kimi and Sol did that every single time, 36 tasks out of 36 each, usually by handing back the wrong total as a bare number. On one document the two of them returned the exact same wrong figure, 1216.30, where the honest arithmetic gives 1210.25.

Fable did push back, 5 times in 29 tasks (29 and not 36, because seven of its calls were blocked before it ever saw them). That makes it the only one of the three that ever caught the trap, and 5 in 29 is still nowhere near enough to be reassuring. So we came away with one working rule for all three, which is that you add the line items up yourself.

What does Sol do when there is no right answer to give?

One task hands the model a fixed list of allowed answers and asks for one that is both greater than 905 and divisible by three. Nothing in the list qualifies. 953 is the only value over 905, and 953 does not divide by three. Saying IMPOSSIBLE is explicitly allowed on this task, and it is the only correct answer. All three models were given the identical task.

• Fable: IMPOSSIBLE
• Kimi: IMPOSSIBLE
• Sol: 953

Sol did not flag the contradiction and it did not hedge. It picked the allowed value that came closest to satisfying the rule and returned that. A validator will wave the answer through, because 953 is well formed and it is on the allowed list, so a check that only looks at the shape of a reply has nothing to complain about.

Fable came first, but a filter blocked 43 of its calls

Fable finished first, and the win comes with a caveat the scoreboard does not carry on its face. A safety filter sitting in front of the model blocked 43 of Fable’s first-attempt calls, all of them harmless finance, document or poetry test tasks. Kimi and Sol were sent the identical bytes and refused none of them. Because the filter sits ahead of the model, the block lands before Fable ever sees the prompt, and it fires unpredictably, so sending the same task again can get you a different outcome.

Those blocks count against Fable’s score for answering at all, which is part of why it reads 55.7 where Sol reads 100.0. On the API this surfaces as calls that never reach the model at all, and the fallback path for them has to live in your own client. The full benchmark writeup has the coverage numbers sitting behind that score.

What to set up before you ship on any of them

None of this is specific to one vendor. Each of the three models failed in a way the other two did not, and the invoice trap caught all three. So the checks below apply whichever one you end up picking.

• Set your client timeouts off the slow tail rather than the median. A 20.7-second median sits in front of a slow end past seven minutes. Pick a ceiling you can live with, then decide up front what your code does with a call that blows through it.

• Treat a call that never came back as data. Log every empty return and every timeout, including the ones a vendor blocked before the model ever saw them, then retry on terms that you set.

• Add the line items up yourself. Every model here passed a wrong invoice total straight through, so a total printed in a document is something you check rather than something you use.

• A valid-looking value can still be the wrong value. 953 is well formed and it is sitting on the allowed list, so it clears a shape check on the way through. Test the value against the rule it was supposed to satisfy.

The whole policy is a handful of lines in your client, and it ports across all three vendors.

# generic client policy, portable across vendors
timeout = 1.5 * p90_latency   # p90 alone cuts one call in ten
resp = call_model(prompt, timeout=timeout)
if resp.empty or resp.timed_out:
    # a non-return is a result; log it and retry deliberately
    log("no_answer", model, item_id)
    resp = retry_once(prompt, timeout=timeout)
# never trust a stated total; check meaning as well as shape
assert recompute_total(doc) == resp.total
assert satisfies_constraint(resp.token)

So which one should you actually build on?

Pick the model whose worst case your own pipeline can absorb, and configure one of the others as a fallback. A pipeline that falls over on latency has a different best model from one that falls over on a total nobody re-added, so the three of them really split on which failure you can least afford in production.

One clean call tells you very little about the next one, and reliability@k is our name for putting a number on whether an answer holds up across every attempt. Run that on your own tasks before you commit, and pick up the ranges and the per-area caveats from the full writeup while you are in there. Every one of the three handed back a total the document had got wrong, and only Fable ever questioned it.

Note

Results come from LatentEval’s trio-2026-07 run, which is the scorecard every figure on this page is drawn from. The index combines eight scored areas, each given a share we fixed before the run started, and it carries a 90 percent range for each model. All three models were sent identical prompts. Calls that a provider blocked before the model saw them count in full against its score for answering at all, and they come out of the totals for the other seven areas. Fable’s invoice result reads 5 out of 29 rather than 5 out of 36 for exactly that reason. The latency numbers on this page are Kimi’s alone, measured on its own direct calls, because Fable and Sol ran mostly through their vendors’ batch queues where the elapsed time is mostly queueing. This run cannot compare the three models on interactive speed and does not try to. The output caps we set were our own, and they were too low for how fast a model with thinking always on spends its budget. Fable ran into them more often than Kimi did, and we count those empty replies as a fault in our setup rather than in either model. Every per-area score, along with the caveats attached to it, sits in the full benchmark report.