Self-hosting an open-weight model is not "free AI." It is trading an API bill for an infrastructure bill — and sometimes that trade is very good, sometimes terrible.
TL;DR
- Self-hosting buys control, privacy, and predictable cost at the price of operational burden.
- It pays off at high, steady volume or under strict data requirements.
- It rarely pays off for spiky, low-volume, or frontier-reasoning workloads.
- Be honest about the total cost, not just the per-token comparison.
What "self-hosting" actually means
Open-weight models — the DeepSeek-class releases and their peers — can be downloaded and run on hardware you control, rather than called through a vendor's API. That changes the cost structure from per-token API pricing to "whatever it costs to run the GPUs," plus the engineering to operate them.
When it pays off
Self-hosting tends to win in these conditions:
- High, steady volume. At enough sustained tokens per day, owned or reserved GPUs beat per-token API pricing. The fixed cost amortizes.
- Strict data requirements. Regulated industries or sensitive data that cannot leave your environment make self-hosting less a cost question than a compliance one.
- Latency control. Co-locating the model with your application can cut round-trip time and remove a third-party dependency.
- Predictability. A fixed infrastructure cost can be easier to budget than a usage bill that scales with success.
Self-hosting is a volume-and-control play. If you have neither high volume nor a hard data constraint, the API is almost always cheaper all-in.
When it does not
Be equally clear about when not to:
- Spiky or low volume. Idle GPUs are pure waste. APIs scale to zero; your hardware does not.
- You need frontier reasoning. The very best models are generally not the ones you self-host. If the task needs that tier, use the API.
- Small team, no ops appetite. Running inference in production means GPU provisioning, scaling, monitoring, updates, and on-call. That is real, ongoing work.
Count the whole cost
The per-token comparison that makes self-hosting look cheap usually ignores the rest of the iceberg:
- Hardware or reserved-capacity cost, including idle time.
- Engineering time to deploy, scale, and maintain serving infrastructure.
- Monitoring, observability, and on-call for a new production system.
- The opportunity cost of the team doing this instead of product work.
True cost = GPU/infra (incl. idle) + ops engineering + monitoring + opportunity cost
Compare THAT to the API bill — not just price-per-token.
A pragmatic path
Most teams should start on an API, prove the use case, and measure real volume. If sustained volume or a data constraint then justifies it, migrate the high-volume, well-bounded workloads to self-hosted open weights — and keep using APIs for the spiky and the frontier-reasoning tasks. Hybrid is usually the right answer, not religion in either direction.
Decide with the total cost in front of you, not the per-token number that flatters the choice you already wanted to make.