How Much VRAM Do You Need to Run an LLM Locally?

Whether a language model runs on your GPU comes down to one question: does it fit in video memory? If it doesn't, it either won't load or will spill into system memory and slow to a crawl. The good news is that the estimate is simple arithmetic, and the biggest lever is one you control.

Part 1: the weights

The model's parameters have to sit in memory. Their size is the parameter count × bytes per parameter. At 16-bit precision (FP16 or BF16) each parameter takes 2 bytes, so an 8-billion-parameter model needs 16 GB for its weights alone. At 8-bit it's 1 byte per parameter, and at 4-bit about half a byte.

Part 2: the KV cache

While generating, the model stores keys and values for every token in the context so it doesn't recompute them. Per token that's 2 × layers × key-value heads × head dimension × bytes. For a typical 8B model with 32 layers, 8 key-value heads and 128-dimension heads at 16-bit, that's about 131 KB per token. It grows with context length and with the number of requests served at once.

Part 3: overhead

The runtime, activations and memory fragmentation take extra. Around 10% is a reasonable planning margin, more for some serving frameworks.

So an 8B model at full 16-bit precision needs about 18.8 GB and fits on a 24 GB card, not a 16 GB one.

Quantization is the big lever

Dropping precision shrinks the weights in proportion. The same 8B model needs about 10 GB at 8-bit, which fits a 12 GB card, and about 5.6 GB at 4-bit, which fits an 8 GB card.

Quality usually drops a little with heavier quantization, and more for small models than large ones. Test on your own tasks before settling on a precision.

Long context and bigger models

Context is the part people forget. Raising the same 8B FP16 model's context from 8,192 to 32,768 tokens quadruples the KV cache to about 4.3 GB and takes the total to about 22.3 GB, which is tight on a 24 GB card. At the other end, a 70B model with 80 layers at 4-bit needs about 41.5 GB with an 8K context: a 48 GB card, or two 24 GB cards with the model split across them.

Tip: If a model almost fits, reduce the context length or quantize the KV cache to 8-bit before buying a bigger GPU. Both cut memory with little effort.