A model comparison written in March is a historical document by September. The benchmark table that justified it is stale. The model that won has been superseded by its own vendor. And the reasoning that produced the choice was never written down anywhere it could be re-run.
What survives is the contract, the integration, and a quiet assumption nobody wants to re-litigate.
That is the real failure mode in model selection, and it is not solved by picking better. It is solved by deciding differently.
So the thing worth building is not the choice. It is the procedure that produces the choice, and the evidence it ran on. Which criteria actually resolve GPT versus Claude versus Gemini versus open weights, how to weight them against your own workloads, where the money genuinely changes hands between an API and a GPU, and why most organisations that run this exercise honestly stop looking for a winner and start routing. The models on today's leaderboard will not survive the year. The criteria will.
The question that decays, and the one that does not
"Which model is best" has a short half-life. Between February and April 2026 the three leading labs shipped seven frontier models between them, roughly one new state-of-the-art system every 11 days (Digital Applied, 2026), and each release reshuffles the leaderboards the previous decision was built on. Open-weight models have compressed the gap on a similar cadence, trailing the closed frontier by an average of about four months since January 2026 (GMI Cloud, 2026), with other trackers putting the lag closer to seven. Any article that ranks specific models, this one included, has an expiry date.
"How should we decide" behaves differently. The criteria that mattered two years ago still matter. Whether the model does your task well. Whether it responds fast enough under your concurrency. What it costs at your volume, whether your data can legally sit where the model runs, and how expensive it is to leave. Those did not churn. Only the values plugged into them did.
The practical consequence deserves blunt phrasing. If your selection lives in a benchmark screenshot, re-deciding means starting over. If it lives in a weighted criteria sheet with your own evaluation set attached, re-deciding means an afternoon of tests. One of those organisations adopts new models faster than the other, and adopts them for reasons it can defend to an auditor.
What goes on that sheet is the rest of this article.
State your evaluation method before you compare anything
A comparison without a stated method is a preference with a table attached.
Seven criteria cover almost every real decision. Write them down, with how each is judged, before looking at a single provider.
Task fit on your own evaluations. Not leaderboard rank. A set of real inputs from your own traffic, with pass criteria defined before you look at any output. This is the one criterion where public benchmarks mislead, because a model tuned to score well on a public set can behave differently on your domain language, your document formats, and your edge cases.
Latency and throughput at your concurrency. Median latency in a quiet test tells you very little. What matters is p95 under the request pattern you generate, including the burst shape. A model that is fast in isolation and rate-limited at your peak is slow.
Total cost at your volume. Per-token pricing is one input among several. The number that decides is monthly spend at your projected request volume and token profile, against the fully loaded cost of the alternative.
Data residency and compliance posture. Where inference runs, what is retained, what contractual commitments exist about training on your inputs, and which certifications the provider holds. This one is usually pass or fail rather than weighted.
Context window and retrieval fit. How much context the model can use well, which is not always the advertised maximum, and how that interacts with your retrieval design. A larger window sometimes replaces retrieval engineering. Sometimes it just makes each call more expensive.
Operational maturity. Rate limits and how they are raised, uptime history, deprecation and migration policy, support responsiveness, roadmap stability. This is the criterion that never appears in a comparison table and causes the most production pain.
Exit cost. How much work it is to move off this provider in eighteen months. Prompt portability, output-format coupling , dependence on provider-specific features, and how much evaluation you would need to redo.
How you judge each one belongs in a sentence next to it. "Task fit is judged by pass rate on the fifty-item golden set, minimum eighty percent, scored blind" is a method. "We evaluated task fit" is a note to self.
Weighting the criteria for your situation
The criteria are stable. The weights are not, and this is the part almost nobody publishes.
Two of the seven are pass-or-fail rather than weighted. Data residency is a threshold: a provider that cannot run inference where your data is legally allowed to be processed is out, and no amount of capability compensates. Apply it first, before any scoring, because scoring a disqualified option wastes the exercise. Operational maturity behaves the same way when the workload is customer-facing.
The rest get weighted by workload profile. Three profiles cover most of what a mid-market or enterprise organisation runs.
| Criterion | Regulated internal tooling | High-volume customer-facing | Low-volume, high-stakes reasoning |
|---|---|---|---|
| Task fit on your evals | Medium | High | Highest |
| Latency and throughput | Low | Highest | Low |
| Total cost at volume | Low | Highest | Lowest |
| Data residency | Disqualifier | High | High |
| Context and retrieval fit | High | Medium | High |
| Operational maturity | Medium | Disqualifier | Medium |
| Exit cost | High | Medium | Low |
Read the columns rather than the rows. A regulated internal tool tolerates latency nobody would accept in a product, so speed stops differentiating and residency does the deciding. A high-volume customer-facing workload inverts that: a fractional cost difference per call compounds into the largest line on the bill, and rate limits become an availability risk rather than an inconvenience. Low-volume high-stakes work, the contract analysis and the incident diagnosis, is where paying several times more per call is obviously worth it and quality is the only thing being bought.
Fill this in for your own workloads before you look at any provider. Weights chosen after seeing the options are rationalisation with a table attached.
Where each provider actually wins right now
This section ages faster than anything else here. Read it as a snapshot, then re-run the criteria above.
The frontier proprietary models from OpenAI, Anthropic and Google have converged more than the marketing suggests. On broad general capability the practical differences for most business tasks are smaller than the difference between a good prompt and a careless one. Where they diverge is specialisation and ecosystem: coding and long-form agentic work, multimodal breadth, integration with an existing cloud estate, and the unglamorous operational details of rate limits, enterprise agreements and regional availability. (That last group is where most of the real friction lives, and it is the part nobody demos.)
Open-weight families have changed the shape of the decision more than any single frontier release. The relevant fact is a narrower one than "open has caught up". A mid-sized open model now clears the quality bar for a large share of production workloads, particularly classification, extraction, summarisation and routing. Those workloads are also the high-volume ones, which is exactly where cost and residency pressure is highest.
Smaller models deserve more attention than they get. A great deal of enterprise work does not need frontier reasoning, and the industry has been rediscovering that a smaller model, fine-tuned or well-prompted on a narrow task, is often faster, cheaper and more predictable than a large general one.
Three habits keep this section from becoming a liability:
- Attach a date to every capability claim you record internally, so it is obvious when it went stale.
- Never let a provider comparison outlive its evaluation set. If the evals have not been re-run, the comparison is a memory.
- Distinguish "wins on our evals" from "wins on a leaderboard". They agree less often than you would like.
Open source, open weight, and source-available are three different things
Almost everything called an open-source LLM is not one, and the distinction has legal consequences rather than merely semantic ones.
Open source in the sense the term normally carries would mean the training code, the training data, and the weights are all available under a licence that permits use, modification and redistribution without restriction. Very few widely used models meet that bar.
Open weight means the trained parameters are downloadable and runnable, while the training data and often the training pipeline are not. This is what most people mean when they say open-source LLM. You can self-host it and fine-tune it. You cannot reproduce it, and you cannot fully audit what it learned from.
Source-available means the weights come with a licence that restricts something material: commercial use above a scale threshold, particular application categories, competitive use, or redistribution terms. The model is downloadable and the licence is not open in any conventional sense.
Why this matters on a Tuesday afternoon in a procurement review:
- Legal will ask which licence, and "it is open source" does not survive the question. The licence name does.
- Some licences impose obligations that attach to your product, not just your infrastructure.
- Acceptable-use terms in several open-weight licences restrict application categories that a regulated business might be in.
- The audit story differs sharply. Open weights let you inspect behaviour but not provenance, which matters for some compliance regimes and not others.
Record the actual licence name against every candidate model. That single field prevents most of the downstream surprises, and it costs nothing to capture while you are already evaluating.
What self-hosting actually costs
The open-versus-proprietary argument is usually conducted on principle and settled by arithmetic. The arithmetic is not complicated. It is rarely finished, because the API side of the comparison has one line item and the self-hosted side has seven.
The table below is a structure rather than a quotation. The values depend entirely on your volume, your model size, your region and your existing team, so fill it in with your own numbers before drawing any conclusion.
| Cost line | Managed API path | Self-hosted path |
|---|---|---|
| Inference compute | Per-token price, scales linearly with usage | Reserved or on-demand GPU capacity, paid whether or not it is used |
| Idle capacity | None, you pay per call | Real and often large, since capacity is sized for peak |
| Serving and scaling infrastructure | Included | Your responsibility, plus the engineering to run it |
| Model operations headcount | Minimal | The dominant line for most organisations |
| Evaluation infrastructure | Needed either way | Needed either way, plus regression testing across model updates you now own |
| Fine-tuning and iteration | Priced per job, if offered | Cheaper per experiment, more expensive to set up |
| Upgrade path | Provider ships it | You plan, test and execute every upgrade |
Three observations hold regardless of the specific numbers.
The crossover is a question of volume rather than philosophy. At low and moderate request volumes, the managed API is almost always cheaper once headcount is counted honestly, because the fixed costs of self-hosting are large and the variable cost of the API is small. At sustained high volume with a predictable load shape, the economics invert, because reserved capacity running near full utilisation is efficient and per-token pricing is not.
Utilisation is the hidden variable. Self-hosting looks cheap when modelled at full GPU utilisation and expensive when modelled at the utilisation you will achieve. Model both, and treat the gap as the real risk in the estimate.
Headcount is the line people leave out, and it is usually the one that decides. Running your own inference is an ongoing operational commitment rather than a deployment. If that capability does not already exist in the organisation, the honest cost of self-hosting includes building it.
Write the assumptions next to every input. A cost model whose assumptions are invisible cannot be challenged, and one that cannot be challenged will not survive its first surprise.
Managed API, VPC, or on-premises
Deployment topology is where compliance resolves, and it gets confused with the open-versus-proprietary question constantly. They are separate axes.
Managed API. Lowest operational burden, fastest to start, least control over where inference happens. Region selection and contractual data commitments do a lot of work here, and the enterprise tiers of the major providers have moved substantially toward what regulated buyers ask for.
Cloud VPC deployment. The model runs inside your own cloud tenancy, whether that is a proprietary model offered through a hyperscaler's platform or an open-weight model you deploy yourself. Data stays inside your network boundary. Operational burden sits between the other two options.
On-premises. Full control, full responsibility, highest fixed cost. Necessary for a narrow set of requirements: air-gapped environments and certain sovereignty regimes.
Here is the misconception that costs the most money. "We need open source for data privacy" is usually false. The underlying requirement is almost always that data must not leave a boundary, or must not be retained, or must not be trained on. A proprietary model deployed inside your own cloud tenancy satisfies that, and so does a contractual commitment plus regional pinning for a large share of use cases. Open weights are one route to a residency requirement among several. Choosing them for a constraint a VPC deployment would have satisfied means paying the self-hosting bill indefinitely for nothing.
Separate the requirement from the implementation. Write down the actual constraint, ask which topologies satisfy it, and only then ask which models are available in those topologies.
Why the answer is usually hybrid
Run the weighting exercise across a real portfolio of workloads and a single provider rarely wins every column. That is the finding, not a flaw in the exercise.
A typical shape looks like this. Frontier proprietary models handle the low-volume, high-stakes reasoning where quality is the only thing that matters and cost per call is irrelevant. A cheaper model, often open-weight and often smaller, handles the high-volume extraction, classification and routing that makes up the bulk of request traffic. Anything touching data that cannot leave a boundary runs in a VPC or on-premises deployment, whichever the constraint demands. Each workload gets the model its own weighting selected, rather than every workload inheriting one organisational preference.
This is also the honest answer to the vendor-lock-in question. Multi-provider capability buys an option more than it buys a discount. An organisation that already routes to two providers can move a workload in days. An organisation coupled to one moves in quarters, knows it, and therefore tends not to move at all.
The cost of routing is real, and any argument for hybrid that skips it is selling something:
- An abstraction layer over provider APIs, which someone has to own and keep current as provider interfaces change.
- Per-provider evaluation, because a prompt that performs well on one model frequently does not transfer cleanly to another.
- Observability that works across providers, so cost, latency and quality are comparable rather than reported in three incompatible shapes.
- Prompt and output-format portability as an explicit design constraint rather than an afterthought.
- More surface area to secure, monitor and keep compliant.
That overhead is why hybrid is a decision rather than a default. It pays off when the portfolio is heterogeneous, which for most organisations past the pilot stage it is. It does not pay off for a single workload, and building a routing layer before there is anything to route is a common and expensive mistake.
Design for portability from the start. Add providers when a workload justifies one.
Test it on your own workload
Everything above is a framework until it meets real inputs. The evaluation set is what converts it into a decision, and it does not need to be elaborate.
Build a golden set from actual traffic. Thirty to a hundred real inputs, covering the common cases and the edge cases you already know hurt. Real inputs, not invented ones, because invented inputs are unconsciously written to be answerable.
Define pass criteria before you look at any output. Write down what a correct response must contain and what disqualifies one. Doing this afterwards means grading against whatever the first model happened to produce.
Score blind where you can. Strip provider identity from outputs before review. Expectation shapes scoring more than most engineers believe it does.
Run every serious candidate through the same set, including the smaller and cheaper models. The frequent and useful surprise is that a cheaper model clears the bar on a workload where the expensive one was assumed to be necessary.
Re-run quarterly and whenever a provider ships a major release. This is the step that makes the framework compound rather than decay. The golden set is the asset, and it gets more valuable every time it is used.
Record the result, the date, and the criteria weights alongside it. That record is what turns the next decision into an afternoon instead of a project.
Red flags we keep running into
Patterns from multi-model deployments, in rough order of how often they cause damage.
The decision was made on a benchmark and never tested on real traffic. Nearly universal in first deployments. The model that ranks highest on general capability is frequently not the best fit for a narrow, repetitive production task, and nobody discovers this until the quality complaints arrive.
Rate limits found in production. Provider quotas surface during the first real traffic peak rather than during evaluation. Ask about limits and the process for raising them before committing.
Portability treated as a later problem. Prompts written against one provider's quirks, outputs parsed in a provider-specific shape, provider-only features used casually in core paths. Each is small. Together they are the migration.
Cost modelled on a demo rather than a load shape. Estimates built from a handful of test calls miss retries, long contexts, system prompts on every request, and the multiplication that comes from chaining calls together.
Self-hosting chosen for a requirement a VPC deployment would have met. The most expensive version of this mistake, because the operational cost is paid indefinitely for a constraint that had a cheaper solution.
No owner for the evaluation set. The set is built during selection and never re-run. Six months later nobody can say whether the choice is still right, and the next decision starts from zero.
A pilot that quietly became production. The temporary single-provider integration hardens, the abstraction that was going to be added never is, and the organisation discovers its architecture by trying to change it.
Where to start
Pick one real workload. Write the seven criteria and their weights for it before looking at any provider. Apply the disqualifiers first, build a small golden set from real traffic, and run every candidate including the cheap ones. Model the cost with the assumptions written next to the inputs. Record the result with a date on it.
That takes days rather than months, and it produces the one thing a benchmark screenshot never will: a decision you can re-run, defend, and hand to someone else.
If it would be useful to run this against your own workloads with people who have done it before, that is what our Discovery Sprint is for. One week, your actual traffic and constraints, and a written selection framework plus a roadmap at the end of it.
