The economics of running a model yourself

What inference costs

An inference system has several main costs:

• model memory
• compute
• power
• storage
• network transfer
• hardware
• system operation

Model weights are only one part of the cost. During inference, the system also needs memory for the runtime, activations, and KV cache. For a language model, the KV cache grows as the context and generated sequence grow. Model size alone does not define total inference memory.

The model must fit in memory

A basic estimate for model weight memory is:

memory = parameters × bytes per parameter

These numbers describe the weights only. The runtime and KV cache require additional memory.

Quantization can reduce weight memory and make local inference possible on smaller hardware. It can also reduce model quality. The effect depends on the model, quantization method, and task.

Hardware changes the cost

Cloud inference gives you access to hardware without requiring you to buy it. You pay for the service instead. With local inference, you own or rent the hardware. The cost then includes:

• hardware
• electricity
• storage
• maintenance
• software

A local system can have a high initial cost. After that, each request can cost little when the hardware is already available. Cloud inference has lower initial hardware cost and can provide flexible capacity. The better choice depends on the workload.

Utilization matters

Hardware cost also depends on utilization. If a GPU processes one request at a time, much of its capacity can remain unused. If many requests use the same GPU, the hardware cost can be shared across more requests.

Batching can increase throughput. It can also increase the waiting time for individual requests. Interactive applications usually care more about latency. Batch workloads usually care more about throughput.

Tokens affect inference

Language-model inference has two main stages. The first stage processes the input context. The second stage generates output tokens.

Longer inputs and outputs require more computation. A large context also increases KV-cache memory. This is why token usage and context length matter when comparing inference systems.

Small models change the calculation

A smaller model normally needs less memory than a larger model at the same precision. It also needs less compute. This can reduce the hardware required for inference and make local deployment possible.

A smaller model is not always cheaper. If it needs more retries, or cannot complete the task reliably, total cost can increase. Compare the cost to complete the task.

Small and large models can work together

A production system does not always need one model.

request → small model → large model when required

A small model can handle simple requests. A larger model handles requests that need more capability. For example, a small model can perform classification, routing, extraction, or simple tool selection. This can reduce the number of requests sent to the larger model. The routing system adds complexity, so it should be measured against the benefit.

When local inference makes sense

Running a model yourself can make sense when:

• traffic is stable
• the model fits the hardware
• hardware utilization is high
• data should remain local
• latency is important
• network access should not be required
• you already have suitable hardware

Cloud inference can be better when traffic is unpredictable, hardware needs are large, or you do not want to operate the system yourself.

The real question

The question is not: “Is local inference cheaper?”

The better question is: “What is the lowest total cost for the required task?”

That includes:

• model quality
• hardware
• memory
• latency
• throughput
• power
• network
• utilization
• operations

A small model can reduce several of these costs. A cloud service can reduce hardware and operations work. A local system can reduce network dependence and recurring inference charges. The best inference system is the one that matches the workload.