Self-hosted
$0.00208
TOOL | eval
At what load does serving your own model cost less than the API?
Enter your instance throughput, fleet parameters and API token prices. The calculator sizes a fleet for the peak, compares both monthly bills and finds the request volume where they cross.
Enter your instance throughput, API token prices and the load you serve. The calculator sizes a fleet, compares the monthly bills and reports the volume where they cross. Both sides price attempts: our refusal-tax study measured pass rates collapsing once refusals count, and the three-way reliability benchmark shows the cheaper attempt can be the slower or less reliable one, which the cost-per-successful-task calculator prices. This page prices the attempt.
Showing your last valid result. Update the inputs above to recompute.
These figures are an illustration. Replace every field with your own measurements and prices.
Self-hosted saves
$11,315.00/mo
Self-hosted $1,825.00 | API $13,140.00 | Difference $11,315.00
Per-request cost
API
$0.015
Both sides divide the full monthly bill by total requests, so the two figures are directly comparable.
Per million output tokens
Self-hosted
$5.21
API
$37.50
Both sides divide the full monthly bill by total output tokens. Input tokens are not given their own denominator on either side.
Fleet
| Instances | Average utilization | Busy instances (average) |
|---|---|---|
| 1 | 14.8% | 0.15 |
Crossing load
| Requests per hour | Requests per month | Instances at that load |
|---|---|---|
| 167 | 122,000 | 1 |
The continuous request rate where the two monthly bills meet. The real bill steps in whole instances, so the crossing you will actually observe sits at the next instance boundary.
What would have to change
| Instance price | Output tokens/s per instance |
|---|---|
| $18.00 | 30.9 tokens/s |
The instance price or throughput at which this month's bills would be equal, computed on the continuous relaxation of the fleet curve.
Load sensitivity
| Load | Requests per hour | Instances | Self-hosted monthly | API monthly | Cheaper |
|---|---|---|---|---|---|
| 0.25x | 300 | 1 | $1,825.00 | $3,285.00 | Self-hosted |
| 0.5x | 600 | 1 | $1,825.00 | $6,570.00 | Self-hosted |
| 1x (entered) | 1,200 | 1 | $1,825.00 | $13,140.00 | Self-hosted |
| 2x | 2,400 | 1 | $1,825.00 | $26,280.00 | Self-hosted |
| 4x | 4,800 | 1 | $1,825.00 | $52,560.00 | Self-hosted |
Both monthly bills at 0.25x, 0.5x, 1x, 2x and 4x the entered request rate, with everything else held fixed.
w = out ÷ tok/s (+ in ÷ prefill); rho = R · w ÷ 3600; N = max(min, ceil(rho · peak ÷ utilization))How?
How this is calculated
The request footprint. Every request occupies one serving instance for the time it takes to generate the output tokens. That time is output tokens divided by the output throughput seconds. When a prefill throughput is entered, the prefill pass adds input tokens divided by the prefill throughput seconds to the capacity accounting. The sum is the instance-seconds per request, labeled w in the formulas.
Busy instances. Multiply w by the request rate in requests per second (that is, requests per hour divided by 3,600) and the result is the average number of instances busy at any moment. Call it rho. This figure assumes perfectly flat traffic; the peak-to-average ratio accounts for the fact that it is not.
Fleet sizing. The fleet must cover the peak, not the average, and it must leave room at the peak so that a small spike does not queue every request. The fleet size N is the smallest whole number of instances at or above rho times the peak-to-average ratio divided by the target utilization, or the minimum instance count, whichever is larger. Average utilization over the month is then rho / N, and it will be lower than the target because the fleet is an integer and the peak is not all day. The gap between the two is what you pay for.
The two bills. The self-hosted monthly bill is straightforward: N instances times the hourly price times the hours per month, plus the fixed monthly cost. The API bill multiplies total monthly requests by the effective per-request price. That per-request price blends input and output rates, applying the cached-input price to the cached share when prompt caching is in use. Both bills are divided by the same request count to get a per-request comparison, and by the same output-token total to get a per-million-output-token comparison, so neither side carries an attribution rule the other does not.
Why the throughput must come from a load test. Modern serving systems use iteration-level batching: one instance interleaves many requests in flight, so aggregate tokens per second rises with concurrency rather than equaling the single-stream decode speed. The number this page asks for is the throughput measured at the concurrency you will actually serve, with your own prompt and output lengths. A benchmark that decodes one stream at a time will undercount it, and a vendor datasheet number measured at a concurrency you do not serve will overcount it. Orca (Yu et al., OSDI 2022) introduced iteration-level batching for transformer serving, and the throughput it produces is what the output throughput field expects.
The crossing. The self-hosted cost curve has two segments: a flat region where the minimum-instance floor binds, and a linear region where each extra request per hour adds a fraction of an instance. The API cost is a straight line from the origin. Where the two meet is the crossing load, reported in requests per hour and requests per month. The page computes it on the continuous relaxation of the fleet curve (dropping the integer ceiling), so the real crossing sits at the next whole-instance boundary above the reported one. When the marginal self-host cost per request is at or above the API's per-request price, no crossing exists at any volume, and the page says so.
This page sizes a fleet for cost. A fleet sized for a latency deadline is a different fleet, and the latency budget calculator handles that.
The what-if figures. Two numbers answer "how wrong could my measurement be and still hold this verdict." The first is the instance price at which this month's bills would be equal. The second is the output throughput per instance at which they would be equal. Both are computed on the continuous fleet curve with every other field held at the entered value. If either is close to the entered value, the verdict rests on the load test.
Formula: w = out ÷ tok/s (+ in ÷ prefill); rho = R · w ÷ 3600; N = max(min, ceil(rho · peak ÷ utilization))
Questions
What if the self-hosted model and the API model have different pass rates?
This page prices attempts on both sides and says which side is cheaper per attempt. When the model you self-host and the model behind the API pass at different rates, the cheaper attempt is not necessarily the cheaper result. Take either side's per-request cost into the cost-per-successful-task calculator as the cost per attempt, one run for each side, and compare the cost per result you can actually ship.
Where do I get the input and output token counts?
Count them from a real payload rather than estimating. The token counter takes a prompt or a batch and returns the exact count for any model with a known tokenizer. Include the system prompt and any retrieved context in the input count, and include hidden reasoning tokens in the output count if your provider bills for them.
Does a cheaper fleet mean faster responses?
A fleet sized by this page is sized for cost. A fleet sized for a latency deadline is a different fleet. Concurrency appears on both pages, but here it is the condition under which a throughput figure was measured and the divisor that sizes a fleet; on the latency budget calculator it is queueing against a deadline. A fleet that wins the cost comparison here can still miss a latency target.
How do I measure the throughput number this page asks for?
Run a load test at the serving concurrency you expect in production, using prompts and output lengths drawn from your actual traffic. Record the sustained output tokens per second that one instance delivers under that load. A single-stream decode benchmark, a vendor datasheet at a different concurrency, or a number measured with a different prompt length is a different number and will give a different fleet size. The serving concurrency field keeps the throughput figure anchored to the measurement conditions.
Why does the page sometimes say no crossing exists?
The self-hosted marginal cost per request is the cost of the fractional instance each additional request requires. When that marginal cost is at or above the API's per-request price, every additional request costs more on the self-hosted side, so adding volume never closes the gap. The three fields that can change this are the instance price, the output throughput and the target utilization. If the crossing does exist but sits at a volume far above the entered load, the sensitivity table shows how the comparison moves at 2x and 4x.
Does the API side account for rate limits?
The API bill assumes the provider absorbs every request at the stated rate. If your workload pushes against the provider's rate limit, some requests queue or fail, and the effective per-request cost changes. The rate-limit backoff planner models the drain time and retry overhead that rate limiting adds.
Sources
- Orca: A Distributed Serving System for Transformer-Based Generative Models. Introduces iteration-level (continuous) batching, in which one serving instance interleaves many requests in flight, so aggregate tokens per second rises with concurrency instead of being fixed per stream. That is why this page asks for a throughput measured at a concurrency rather than a single-stream decode speed
- Efficient Memory Management for Large Language Model Serving with PagedAttention. Shows that the concurrency one instance can sustain is bounded by KV-cache memory rather than by compute alone, which is why the page requires a measured concurrency instead of letting a reader raise batch size for free