Almost every agentic architecture diagram has a box labelled LLM Runtime, with provider logos on one side and open models on the other. It looks like a catalogue preference. It is the decision that moves the most money in the whole diagram, and the only one most teams make for a reason that does not apply.
The short answer: self-hosting an open model almost never comes out cheaper than calling a provider, because those same open models are sold by API at close to the raw cost of the GPU. What dominates cost is not the model, it is the shape of the traffic. Self-hosting is justified by control, and there are four cases where it genuinely is.
What is the runtime and why decide it separately?
It is where and how the model executes. And it is decided separately because it answers questions no other layer answers.
The orchestration layer decides what to do, with what context, in what order and under what controls. We covered that in the 13 layers of a real AI system. The runtime is a different thing: it is the infrastructure that executes inference, and its questions are cost, latency, control and data residency.
Mixing it with the rest is what makes the decision go wrong. The runtime gets chosen at the start of the project, when the one piece of data that settles it does not exist yet: what your real traffic will look like.
What actually drives the cost?
The shape of the traffic. Not the model, not the provider, not the negotiation.
This is the point that changes the conversation. A GPU is paid by the hour, whether it is working or waiting. If your traffic arrives in short bursts separated by dead hours, you pay for the full day to use the machine for a few minutes. A published analysis of the same model served on the same GPU shows the size of the effect.
Data
| Sustained concurrency | Cost |
|---|---|
| 1 request per second | 15.3 |
| 25 requests per second | 0.9 |
Seventeen times the difference without changing the model or the hardware. All that changed was how many people were using it at once.
From which the uncomfortable rule follows: your own infrastructure rewards even traffic and punishes scattered traffic. And most companies' traffic is scattered, because it follows one country's working hours.
When does self-hosting come out cheaper?
When you sustain high utilization, and that is more demanding than it sounds.
Published estimates agree on the order of magnitude: self-hosting competes when the GPU stays above roughly two thirds utilization much of the time, with dozens of simultaneous requests in flight on a sustained basis [1]. Not at Tuesday's peak: sustained.
And that leaves out the cost that never makes it into the spreadsheet. One platform engineer can operate only a handful of GPUs, and the total cost ends up a multiple of the raw hardware rental once you add monitoring, upgrades, on-call and model replacement [1].
There is one last figure almost nobody considers, and it closes the argument: those same open models are sold by API. A specialist provider serves Llama, Qwen or Mistral at close to the raw GPU cost, with utilization averaged across all its customers instead of yours alone. If you wanted the open model for the model, you can already buy it without buying the operational problem.
Does the law require keeping the model in house?
Almost never, and this confusion costs entire projects.
It is the reason I hear most often for self-hosting in the region, and in most cases it is misstated. Neither Colombia nor Brazil requires processing to happen on local hardware.
In Colombia, Law 1581 and Decree 1377 permit international transfer of personal data through contractual clauses verified by the supervisory authority [2]. In Brazil, the LGPD imposes no strict localization: it permits cross-border transfers to countries with an adequate level of protection, or covered by consent, contracts or specific legal provisions, while requiring appropriate technical and administrative measures [3].
What the rules ask for is a legal basis and demonstrable protection, not a server in the building. A properly drafted processing agreement with a provider that does not train on your data complies; buying GPUs exempts you from nothing on its own, and additionally makes you responsible for the security that used to be the provider's.
This does not say the requirement never exists. It says the requirement has to be read before buying hardware, and that it often comes from a customer's contract rather than from the law.
So when is it worth it?
Four cases, and none of them is price per token.
High, even volume. If your traffic is sustained, with real concurrency for much of the day, the math starts to close. It is the only legitimate economic case.
A customer who will not allow their data to leave. Not the law: the contract. It is a valid commercial reason and perfectly common in banking, health and the public sector.
Latency a remote provider cannot meet. If the case lives inside a plant, a point of sale or a real-time system, physical distance is the argument.
Needing the model not to change underneath you. Providers deprecate versions and adjust behavior. If your system depends on stable output and you have evaluations that prove it, pinning the model has real value.
How do you decide without gambling?
With three numbers you do not have today, and which take a quarter to get.
| Step | What you do | What you get |
|---|---|---|
| 1 | Start on the provider's API | Reach production without committing capital |
| 2 | Isolate the runtime behind your own interface | Changing it becomes a module swap |
| 3 | Measure volume, concurrency and hourly distribution | The three numbers that settle the math |
| 4 | Only then compare against self-hosting | A decision, not a gamble |
Step 2 is the one most often skipped and the cheapest to do at the start. If the code that talks to the model is separate from the business logic, switching runtimes in six months costs days. If it is spread across the system, it costs a project, and that is exactly why many teams keep the decision they made on day one without data.
How we apply this at MasterDragon
We always start on a provider, and we leave the door open from day one.
In practice that means the runtime lives behind an interface of ours, with the model declared in configuration rather than in code. It has let us switch models per task without touching the logic, which is also the largest cost lever available before anyone thinks about hardware.
And when a client raises self-hosting, the first question is not technical: it is whether the requirement comes from the law, from one of their customers' contracts, or from an intuition about savings. In the third case, which is the most common, the answer is usually to measure for three months before buying anything.
If you are defining the architecture and want to know which runtime fits you, talk to our engineers. You can start with how we build your software and review our portfolio of shipped AI-native products. The full AI spend picture is in AI cost is consumption, not licences, what sits underneath an agent that survives production in the 13 layers of a real AI system, and how to design so one piece can change without taking down the rest in composable enterprise architecture.
References
- Comparative cost analyses between self-hosting open-weight models with vLLM and provider APIs (2026), covering utilization thresholds, the effect of concurrency on cost per million tokens, and total operating cost against raw GPU rental.
- Congress of the Republic of Colombia. (2012). Law 1581 of 2012, and Decree 1377 of 2013, on personal data protection and international transfer through contractual clauses verified by the Superintendency of Industry and Commerce.
- Brazil. (2018). Lei Geral de Proteção de Dados Pessoais, Law 13.709/2018, in force since 2020, on international transfers to countries with an adequate level of protection and the technical and administrative measures required.