AI benchmarks vs. real-world use: why the scores and the reports differ

Almost every model launch includes a benchmark claim. It ranks first on an index, matches a bigger model, or gains a set number of points on a test suite. These claims are easy to print and hard for users to check, so on Day 7 receipts they usually stay marked unclear until real use says something.

That isn’t because benchmarks are useless. It is because a benchmark answers a narrower question than the one most people are asking. This article uses five receipts to show where the gap opens up, and what to look at instead.

What a benchmark claim actually says

Here are the benchmark-style claims from our current receipts, quoted exactly from the launch pages:

• Gemini 4 Argon: “Its depth and precision across multi-step tasks leads Argon to outperform other frontier models on benchmarks across domains, ranking #1 on the Vals Index”
• GPT-6.1 Sol: “an upgrade to GPT-6 Sol that nearly matches GPT-6 Astra’s intelligence on agentic coding, computer use, and professional work at one-fifth of Astra’s standard input and output token prices.”
• DeepSeek-V4.1-Flash-1017: “Stronger benchmark performance, faster task completion, and shorter outputs than the previous version”
• EmbeddingGemma 2: “a significant 9.92-point improvement on code performance (in MTEB Code, from 68.76 to 78.68)”

Each is a statement about a fixed set of tasks, run under the vendor’s conditions. None of them tells you how the model behaves inside your agent harness, on your codebase, on your plan, with your bill.

Gap 1: users test tasks, not test suites

Gemini 4 Argon’s #1 ranking is the clearest example. We found two hands-on coding reports. One user found its coding underwhelming and only slightly better than Gemini 3.8 Flash in their experience. Another developer said it found real bugs in a large mixed-language codebase, but asking it to write a long plan file stalled in repeated retries of about a minute each. Two reports pointing in different directions is not enough to judge a benchmark claim, so it stays unclear.

For GPT-6.1 Sol, we found no solid user comparison against GPT-6 Astra at all. People did report on other things: the app updates needed to see the model, the plans that got the faster tier, and a forum thread about refusals and sanitised rewrites on allowed work. Those are the questions users actually ran into, and the benchmark claim doesn’t touch any of them.

Gap 2: cost per token isn’t cost per task

Benchmark scores are usually reported alongside prices per million tokens. But what you pay to finish a job depends on how many tokens the model spends getting there.

DeepSeek’s claim bundles a benchmark gain with “faster task completion, and shorter outputs”. Users split on it. One tester who compared the new weights with the old ones measured fewer output tokens at every reasoning level, with the same intelligence and tool-calling pattern. Another, after 48 hours on dozens of tickets, found it beat GLM 5.3 and ran faster on the same hardware. Against that, a team using it in their own coding agent said it turns simple “describe this” requests into long investigations and was still investigating after GLM 5.3 had fixed the same problem. A new user said their spend climbed far faster than expected in about half an hour. Artificial Analysis ranks it near the top of open models its size but calls it very verbose.

Nobody has rerun the vendor’s benchmarks against the previous version, so that part of the claim is untested. The speed and length parts depend on how you use it, which is why the claim is mixed.

Gap 3: the harness matters as much as the model

Many reports on DeepSeek came from people running it as a worker inside another tool: Claude Code, the Pi harness, their own agent. One user ran three hard questions with it and two Claude models as workers and found it cheaper and more correct than the mid-tier Claude model. Another did about three hours of heavy development with it and said it got everything done cheaply, though a Claude model might have needed less guidance. A third, decompiling old games, found it capable but error-prone.

Same model, different harnesses, different tasks, different results. A benchmark fixes the harness so the comparison is fair. Real use doesn’t, which is exactly why you can’t read real use off the benchmark.

Gap 4: a score needs hardware and tooling to matter

Muse Spark Open shows a different kind of gap. Meta’s headline claim was that it is “the first model in its class to run on your desktop using two Apple M5 Ultra Mac Studios with the latest MLX framework.” Apple published its own measurements on launch day, about 16 tokens a second with two machines in pipeline mode and about 25 in tensor mode. But Apple’s page still listed the 512GB M5 Ultra option as coming in late October, so users couldn’t buy the pair, and no independent two-machine run had been reported. A measured number on hardware you can’t buy yet is not a result you can use.

When a benchmark claim does hold

EmbeddingGemma 2 is our one example where a benchmark-style claim lined up with independent testing. Google claimed the new model “matches the strong multilingual text performance of EmbeddingGemma” while improving code retrieval. The creator of an independent 550-task multilingual retrieval benchmark found both versions scored about the same overall, with the new one ahead on code retrieval. One user’s first tests found no gain for text-only retrieval, which is what the claim implies.

Nobody has rerun MTEB Code itself, but an independent test pointing the same way is strong evidence. Two things helped: the claim was narrow and modest (match on text, gain on code), and the open weights meant anyone could test it. The receipt is stamped Delivered. We look at what separates it from the rest in Delivered vs. Partly.

How to read a benchmark claim on launch day

1. Ask what the test measures. A retrieval benchmark, a coding benchmark and a general index answer different questions.
2. Look for an independent rerun. One outside test pointing the same way is worth more than the vendor’s chart.
3. Estimate cost per task, not per token. Watch for reports of longer reasoning or more output.
4. Check it runs where you need it. A score on hardware or in a runtime you can’t use doesn’t help you yet.
5. Wait for reports in harnesses like yours. Results inside one agent tool don’t always carry over to another.

This is why our method leaves benchmark claims unclear until users report on them, and why the verdict on a launch rests on the claims people can test. Browse every stamp in the archive, or start from the latest receipts.