The Real Cost of Self-Hosting an LLM Is Utilization, Not GPUs

Self-hosting an LLM is usually more expensive than paying an API bill, even when the model weights cost nothing. The mistake is treating the comparison as “GPU rental versus tokens” instead of “an always-on inference operation versus an elastic service.”

I am not arguing that developers should give up on open models. Open weights are strategically valuable: they make model behavior inspectable, reduce dependence on one vendor, and let teams keep sensitive workloads inside their own boundary. But too many engineering teams hear “open” and silently translate it to “cheaper.” Then they deploy a model, admire the local endpoint for a week, and discover that their largest cost is a mostly idle GPU waiting politely for the next request.

A GPU is not a per-token resource once you run it yourself

Managed model APIs turn infrastructure into a variable cost. When your traffic goes quiet, much of the provider’s fleet—not yours—absorbs the idle time. Self-hosting flips that arrangement. Whether the machine is a workstation under a desk, a reserved cloud instance, or hardware in a rack, you have acquired capacity for a period of time. Its cost keeps accumulating while it generates zero tokens.

The arithmetic gets uncomfortable quickly. AWS lists an eight-H100 p5.48xlarge Capacity Block in major US regions at $34.608 per hour, or roughly $25,264 for 730 hours. Even its one-H100 p5.4xlarge listing works out to roughly $3,158 per month if kept running. Those are useful reference points, not a universal quote: contracts, regions, spot capacity, owned hardware, and alternative accelerators change the number. They do not change the economic fact that underutilization is expensive.

This is why a quick benchmark is such a dangerous planning tool. You may load a 32B model, send one prompt at a time, see an impressive tokens-per-second result, and conclude you have ample capacity. Production is not a benchmark loop. It has bursts, uneven prompt lengths, long contexts, streaming users who expect fast first tokens, retries, tool calls, and a peak-load target you cannot casually miss.

The hidden bill is the system around the model

A competent self-hosted deployment needs far more than weights and an OpenAI-compatible endpoint. You need an image and dependency policy, model-artifact storage, startup and warm-up procedures, authentication, rate limits, request-size limits, logging with a deliberate policy for prompt data, tracing, dashboards, alerting, backups for configuration, and a plan for rollbacks. If the endpoint matters to customers, you also need redundancy, health checks, capacity headroom, and someone responsible when a driver update turns a previously stable server into a 2 a.m. incident.

Those are not bureaucratic extras. They are the product. An inference server’s economics depend on scheduling requests efficiently and managing the key-value cache, not merely loading a checkpoint. Modern serving stacks offer continuous batching, streaming, quantization, tensor parallelism, and metrics precisely because a single-request-at-a-time server leaves expensive hardware idle. Long prompts consume cache and prefill time; generous context windows reduce how many concurrent requests fit; a model architecture that falls back to a less optimized execution path can invalidate the capacity assumptions in your spreadsheet.

I would put the real costs into four buckets:

• Capacity: GPUs, CPU and RAM, storage, networking, power, and the idle reserve required for peaks or failures.
• Utilization: queueing, batch composition, prompt lengths, output lengths, cache pressure, and the fraction of time your accelerators are truly doing useful work.
• Operations: upgrades, security patching, observability, incident response, model rollouts, evaluation, and on-call labor.
• Quality: the cost of serving a smaller or more aggressively quantized model when its errors create retries, human review, or worse downstream decisions.

The strongest counterargument—and why it is right sometimes

The best case for self-hosting is stronger than “we like control.” A company with steady, high-volume traffic can amortize hardware well. A workload with strict data-residency requirements, private documents that cannot leave a controlled environment, predictable latency needs, or a domain-tuned open model may be a poor fit for a general API. The API route can also create a strategically dangerous dependency: pricing can change, limits can appear, an important model can be retired, and proprietary behavior cannot be reproduced exactly. For a core product capability, owning the serving path can be rational insurance.

I agree. But that is a case for deliberate ownership after measurement, not for self-hosting as the default expression of engineering independence. The organizations that make this work have sufficiently steady demand, enough workload knowledge to choose a model and context policy, and people who can operate GPU infrastructure. They also compare against the fully loaded API alternative, including cache hits and routing simple requests to cheaper models—not against a single premium-model list price selected to make self-hosting look good.

Run the break-even test honestly

Before committing, I would run an API-backed pilot and instrument it aggressively. Measure input and output tokens separately; requests per minute by hour; p50, p95, and p99 latency; prompt and completion distributions; cacheability; failure and retry rates; and which requests genuinely require the chosen model. Then replay representative traces against a candidate self-hosted stack with the same context limits, streaming behavior, and quality tests.

Most importantly, calculate cost at required service levels, not average throughput. Add idle headroom, redundancy, engineering time, storage, observability, and the cost of a bad deployment. Divide that fully loaded monthly cost by the tokens you expect to serve at the utilization you can actually sustain. If you have to assume a permanently packed queue to win the spreadsheet, you do not have a self-hosting business case. You have a benchmark.

Teams should stop asking, “Can we run this model?” Almost every capable engineering team can. They should ask whether they can keep it busy, safe, observable, and good enough to replace what they buy today. Open-model projects should publish workload traces and operating assumptions alongside their launch posts, and engineering leaders should require utilization and quality gates before approving GPU spend. That change would make self-hosting less fashionable—and far more successful.