Whether a language model runs on your GPU comes down to one question: does it fit in video memory? If it doesn't, it either won't load or will spill into system memory and slow to a crawl. The good news is that the estimate is simple arithmetic, and the biggest lever is one you control.
Part 1: the weights
The model's parameters have to sit in memory. Their size is the parameter count × bytes per parameter. At 16-bit precision (FP16 or BF16) each parameter takes 2 bytes, so an 8-billion-parameter model needs 16 GB for its weights alone. At 8-bit it's 1 byte per parameter, and at 4-bit about half a byte.
Part 2: the KV cache
While generating, the model stores keys and values for every token in the context so it doesn't recompute them. Per token that's 2 × layers × key-value heads × head dimension × bytes. For a typical 8B model with 32 layers, 8 key-value heads and 128-dimension heads at 16-bit, that's about 131 KB per token. It grows with context length and with the number of requests served at once.
Part 3: overhead
The runtime, activations and memory fragmentation take extra. Around 10% is a reasonable planning margin, more for some serving frameworks.
Weights
8B × 2 bytesequals16 GB
KV cache per token
2 × 32 layers × 8 heads × 128 × 2 bytesequals131 KB
KV cache
131 KB × 8,192 tokens × 1equals1.07 GB
Total with overhead
(16 + 1.07) × (1 + 10%)equals18.8 GB
So an 8B model at full 16-bit precision needs about 18.8 GB and fits on a 24 GB card, not a 16 GB one.
Quantization is the big lever
Dropping precision shrinks the weights in proportion. The same 8B model needs about 10 GB at 8-bit, which fits a 12 GB card, and about 5.6 GB at 4-bit, which fits an 8 GB card.
Weights
8B × 0.5 bytesequals4 GB
KV cache per token
2 × 32 layers × 8 heads × 128 × 2 bytesequals131 KB
KV cache
131 KB × 8,192 tokens × 1equals1.07 GB
Total with overhead
(4 + 1.07) × (1 + 10%)equals5.6 GB
Quality usually drops a little with heavier quantization, and more for small models than large ones. Test on your own tasks before settling on a precision.
Long context and bigger models
Context is the part people forget. Raising the same 8B FP16 model's context from 8,192 to 32,768 tokens quadruples the KV cache to about 4.3 GB and takes the total to about 22.3 GB, which is tight on a 24 GB card. At the other end, a 70B model with 80 layers at 4-bit needs about 41.5 GB with an 8K context: a 48 GB card, or two 24 GB cards with the model split across them.
Worked examples for an 8-billion-parameter model
Using our LLM VRAM Calculator with a typical 8B model (32 layers, 8 key-value heads):
- FP16 weights, 8K context: about 18.8 GB. Fits a 24 GB GPU.
- 8-bit weights, 8K context: about 10 GB. Fits a 12 GB GPU.
- 4-bit weights, 8K context: about 5.6 GB. Fits an 8 GB GPU.
- FP16 weights, 32K context: about 22.3 GB, of which 4.3 GB is the KV cache.
- FP16 weights, 128K context: about 36.5 GB. The KV cache alone is 17.2 GB, more than the model itself.
The last line is the one that surprises people. At long contexts, memory is dominated by the KV cache, not the weights. Quantising the weights to 4-bit at 128K context only brings the total to about 23.3 GB, because the 17.2 GB cache is unchanged.
Serving several users at once
Each simultaneous request needs its own KV cache. A 4-bit 8B model serving 8 requests at 8K context needs about 13.9 GB: 4 GB of weights and 8.6 GB of cache. If you plan to serve many users, size the GPU for the cache at your expected concurrency, not just the model.
A practical buying guide
- 8 to 12 GB: small models (around 8B) at 4-bit or 8-bit with modest context, for personal use.
- 16 to 24 GB: 8B models at full precision, or larger models heavily quantised.
- 48 GB: 70B-class models at 4-bit (about 40 GB at 8K context), or long contexts on smaller models.
- 80 GB and up: 70B-class models at higher precision, or serving many users.
Tip: If a model almost fits, reduce the context length or quantize the KV cache to 8-bit before buying a bigger GPU. Both cut memory with little effort.
Questions people ask
- Can I run a 70B model on a 24 GB GPU?
- Not fully on the GPU. Even at 4-bit, a 70B model needs about 40 GB with an 8K context. On 24 GB you would need to offload part of the model to system memory, which is much slower, or use a smaller model.
- How much VRAM does an 8B model need?
- About 18.8 GB at 16-bit with an 8K context, about 10 GB at 8-bit and about 5.6 GB at 4-bit, including the KV cache and a 10% overhead margin.
- What is the KV cache?
- Stored keys and values for every token in the context, kept so the model doesn't recompute them. It grows with context length and with the number of simultaneous requests.
- Can I run a 70B model on one GPU?
- At 4-bit with a modest context, it needs roughly 41 to 42 GB, so a 48 GB card. On 24 GB cards it has to be split across two or more.
- Why does long context use so much VRAM?
- The KV cache grows with every token of context. For a typical 8B model it is about 17 GB at 128K tokens, more than the FP16 weights.