Before you download a model or rent a GPU, you need to know whether it fits. Inference memory has three parts: the model weights, the KV cache that grows with every token of context, and runtime overhead. This calculator estimates each, so you can choose a precision, context length and GPU that work together.
How it works
Weights: parameters × bytes per parameter. An 8B model at FP16 (2 bytes each) is 16 GB; at 4-bit it's about 4 GB. This is usually the biggest part, and quantization is the biggest lever for shrinking it.
KV cache: for every token in the context, each layer stores a key and a value vector for each key/value head. Per token that's 2 × layers × KV heads × head dimension × bytes. Multiply by context length and by the number of sequences served at once. Long contexts and large batches can make it rival the weights.
Models with grouped-query attention use fewer key/value heads than attention heads, which cuts the KV cache a lot. Read num_hidden_layers, num_key_value_heads and the head dimension from the model's config.json for an accurate figure.
Overhead covers activations, the CUDA context and framework buffers. 10% to 20% is a reasonable planning margin. Serving engines that pre-allocate memory may reserve more than this, so leave headroom.
A worked example
An 8B model with 32 layers, 8 KV heads and a head dimension of 128, at FP16 with an 8,192-token context: weights take 16 GB and the KV cache 1.07 GB (131 KB per token). With 10% overhead that's about 18.8 GB, which fits on a single 24 GB GPU. At 4-bit, weights drop to 4 GB.
Questions people ask
How much VRAM do I need to run a 7B or 8B model?
About 14 to 16 GB for the weights at FP16, plus the KV cache and overhead, so a 24 GB GPU is comfortable. At 4-bit quantization, the weights fit in roughly 4 to 5 GB, so 8 GB GPUs can run them with shorter contexts.
How much VRAM does a 70B model need?
About 140 GB for FP16 weights alone, which means multiple 80 GB GPUs. At 4-bit, weights are around 35 GB, which can fit on a single 48 GB GPU with a modest context, or be split across two 24 GB cards.
Why does longer context use so much more memory?
The KV cache stores keys and values for every token in the context, in every layer. Doubling the context doubles the KV cache, and serving several users at once multiplies it again.
Does quantization hurt quality?
8-bit is usually close to full quality. 4-bit loses a little more, depending on the method and model, but it's the standard way to run large models on consumer GPUs. Test on your own tasks.
Is this the same for training?
No. Training also stores gradients and optimizer state, which typically needs several times the memory of the weights alone. This calculator is for inference.