[Official FAQ] Share your experience or follow-up questions below.

Short answer: it depends on the quantization and the context length, but you can estimate it with a simple rule of thumb.

VRAM use is roughly: model weights + KV cache (grows with context) + runtime overhead.

• Weights (approximate): at 4-bit quantization such as Q4_K_M, a model takes very roughly 0.55–0.65 GB per billion parameters. At 8-bit, roughly 1 GB per billion. At FP16, roughly 2 GB per billion.
• KV cache: grows with context length, and with the number of layers and attention heads. A long context can add several GB, so do not ignore it.
• Overhead: leave around 1–2 GB spare for buffers and the display.

Very rough, approximate planning numbers for 4-bit models:

• 7B–8B: comfortable on an 8 GB card with a modest context; 12 GB is more relaxed.
• 13B–14B: usually wants 12–16 GB.
• 30B–34B: usually wants 24 GB-class cards.
• 70B: typically needs 40 GB+ in total, so multiple GPUs, a large unified-memory Mac, or partial CPU offload (which is slower).

If the model does not fit, you can offload some layers to system RAM. It works, but token speed drops noticeably. MoE models need enough memory to hold all the weights, even though only some experts are active per token.

These are estimates, not guarantees. Real usage varies by architecture and software, so check the file size of the GGUF you plan to download and add headroom.

What GPU do you have and what model are you eyeing? Post it below and we can help you estimate.