How to compare state-of-the-art language models without fooling yourself

Start with the decision, not the leaderboard

A model choice is a decision with constraints, and the constraints usually narrow the field before capability does. Before comparing anything, write down four things: the tasks the model must do, the latency the product can tolerate, the budget per request or per month, and the deployment constraints — whether data may leave your network, which regions you must serve, and whether you need the weights.

It is common to discover at this point that the decision is already made. If data cannot leave your infrastructure, hosted models are out regardless of how they score; if the product must answer in a few hundred milliseconds, the largest reasoning models are out. Doing this first saves weeks of benchmarking models you were never going to ship.

Capability axes worth separating

“Capability” is not one thing, and models are not uniformly ordered. A model that writes excellent prose may be mediocre at following a rigid output schema. A model that is strong at competition mathematics may be unremarkable at reading a messy contract. Separate the axes and test them separately.

Two of these are routinely conflated. Faithfulness is not accuracy: a model can be right about the world and unfaithful to the document you gave it, which is the failure that matters in retrieval systems. And structured-output reliability is a tail property — almost always valid is still unusable if the remaining failures arrive unhandled.

What public benchmarks can and cannot tell you

Public benchmarks are genuinely valuable: they are how the field compares methods, and a model weak across many of them is unlikely to surprise you. The mistake is treating a benchmark as a measurement of your workload.

• Contamination. Benchmarks are public text. Training corpora are large. The older and more famous a benchmark, the more likely parts of it are memorized, and the effect is invisible from the outside.
• Saturation. When the strongest models cluster near the ceiling, the remaining gap is noise, ambiguous items and mislabeled answers rather than capability.
• Prompt sensitivity. Reported results depend on the prompt template, the number of examples shown, the decoding settings and the answer parser. Two honest evaluations of the same model can disagree substantially.
• Single-number reporting. An aggregate over many subtasks hides the subtask you care about, which may move in the opposite direction.
• Selection effects in arenas. Human preference rankings measure what raters prefer in a short exchange: often formatting, confidence and length rather than correctness.

Use benchmarks as a reference check: a filter that removes unsuitable candidates, not the interview. The open evaluation harnesses are more useful than the published numbers, because running one teaches you how sensitive the result is.

Latency, cost and context: the axes that decide shipping

Capability differences between the leading models are often smaller than the operational differences, and operational properties are much easier to measure correctly.

• Time to first token, separately from tokens per second. A streaming interface lives or dies on the first; a batch job lives or dies on the second.
• Tail latency. Measure the slow end of the distribution under your real concurrency, not an average from an idle afternoon.
• Cost per unit of work, not per token. A model with a higher token price that finishes a task in fewer attempts can be cheaper per completed task.
• Caching. Where a provider supports reusing a long stable prefix, the effective cost of a system prompt or a document changes substantially — often the largest single lever on cost.
• Effective context length, not advertised context length. Test where in a long input the model still reliably uses what it is given.

Reasoning modes versus direct answering

Most current families offer some form of extended deliberation: the model spends additional computation before answering, either always or when asked. The trade is consistent. Deliberation buys accuracy on problems with many dependent steps — mathematics, planning, hard debugging — and costs latency and money on everything else.

• Classify your tasks. Extraction, classification, routing, summarization and formatting rarely benefit from extended reasoning; multi-step derivation usually does.
• Where the mode is adjustable, treat the reasoning budget as a tunable parameter and measure it like one, rather than leaving it at whatever the default is.
• Route rather than choose. Many production systems send most traffic to a fast model and escalate the hard minority. The routing rule is usually simple and worth more than a model upgrade.
• Remember that intermediate reasoning is not an explanation: it is generated text that correlates with the answer, not a faithful trace of how the answer was produced.

Open-weight versus hosted

This is a deployment decision more than a capability decision, and it is best made on the constraints you wrote down at the start.

One point is easy to miss: “open weights” is not “open source”, and licenses here range from genuinely permissive to conditional on company size or use. If the license matters, read it rather than the announcement.

The landscape, described qualitatively

What follows is a characterization of families, not a ranking, and deliberately contains no version numbers or scores. Specific models within each family differ from each other more than the families differ on average, and all of it moves.

• GPT (OpenAI). A broad hosted lineup spanning small fast models to large deliberative ones, with a mature tool-calling and structured-output surface and wide third-party integration. Usually among the first to expose new capabilities.
• Claude (Anthropic). Hosted, with a strong emphasis on long-context work, careful instruction following and agentic tool use; frequently chosen for document-heavy and code-heavy workloads.
• Gemini (Google). Hosted, natively multimodal across text, images, audio and video, with very large context offerings and tight integration into Google's cloud and developer tooling.
• Llama (Meta). The reference open-weight family for much of the ecosystem: several sizes, extensive tooling support, and a community license worth reading rather than assuming.
• Mistral (Mistral AI). A European provider offering both open-weight releases and hosted models, with a consistent focus on efficiency and on smaller models that are cheap to serve.
• Qwen (Alibaba). A large open-weight family covering many sizes and modalities, with notably strong multilingual coverage, particularly across Asian languages.
• DeepSeek. Open-weight releases that pushed efficient training and inference architectures and explicit reasoning modes into general availability, and that are widely used as self-hosted baselines.

There are strong models from other labs too, and the useful habit is not to memorize the list but to re-derive it: read the model cards of whatever is current, note the context limit, modalities, license and stated training cutoff, and put the two or three plausible candidates through your own evaluation set.

Building a private evaluation set

1. Take real requests from your own logs or your own domain, including the awkward ones. Thirty is a usable start.
2. For each, write the check: an exact match, a schema validation, a test suite, a regular expression, or a short human rubric. If you cannot state the check, the task is underspecified and worth fixing before you blame a model.
3. Fix the prompt, the temperature and the parsing across candidates. Changing two things at once is how comparisons become folklore.
4. Run each candidate several times. Sampling variance between runs of the same model is frequently larger than the difference between two models.
5. Re-run the set when a provider updates a model you did not pin. Silent upgrades are real and they move behavior.

The purpose of a comparison is to make a decision you can defend in six months, not to be right on the day the model launched.

Summary

Comparisons go wrong in predictable ways: one aggregate number stands in for a workload it does not resemble, operational properties are ignored until launch, and a figure copied from an article ages into a wrong assumption. The corrective is unglamorous. Write down the constraints first. Separate capability into axes and test the ones you depend on. Use public benchmarks as a filter and your own tasks as the interview. Measure latency and cost per completed task, not per token. And when a specific number matters, look it up at the source rather than trusting anyone's recollection, including your own.

Further reading

Named rather than linked, on purpose: specifications move and URLs rot, while a title and an author survive a search.

• Model Cards for Model Reporting — Mitchell et al., the paper that established the documentation format every vendor now publishes.
• HELM (Holistic Evaluation of Language Models) — Stanford CRFM's multi-metric evaluation framework, and a good argument against single-number reporting.
• EleutherAI's lm-evaluation-harness — the open harness behind many reported results; running it yourself is the fastest way to learn how sensitive scores are.
• Chatbot Arena (LMSYS) — human-preference rankings, best read alongside an understanding of what preference measures.
• Each vendor's own model card and pricing documentation — the only current source for context limits, modalities, training cutoffs and cost.