Deployment fit
Headroom.
Fitting is the easy part. Headroom is what keeps it running.
Set the device, the model and the context. The estimate shows weights, KV cache and a runtime reserve against the memory you have.
WHAT THE ESTIMATE ASSUMES
- Weights Model size times bytes per weight. A 4-bit file is not exactly half of an 8-bit file, so the values in the tool are rounded up.
- KV cache Two tensors per layer, at 16-bit, for every token in the context and every concurrent session. Encoder-only models have no KV cache.
- Runtime reserve 22% of weights and KV cache, with a 350 MB floor, for the runtime, buffers and fragmentation.
- Tokens per second Memory bandwidth divided by the planning total, at 70% of peak. A rough ceiling, not a measured rate.
This is a planning estimate, not a benchmark. Measure peak memory on the target device before you commit. Every model release changes the real numbers, so treat the result as a first filter.