Guides · Updated 2026-10-17

VRAM and quantisation explained: what actually fits on your GPU

Why a 9B model fits in 8GB, what Q4_K_M means, how context length eats memory, and why bandwidth sets your speed.

Two numbers decide your local AI experience: how much memory you have (which models fit) and how fast that memory is (how quickly they reply).

Parameters and memory

A model’s size is measured in parameters (“8B” = 8 billion). At full 16-bit precision each parameter takes 2 bytes, so an 8B model needs about 16 GB. That’s why almost everyone runs quantised models.

Quantisation in one paragraph

Quantisation stores each weight with fewer bits. At 4-bit, the same 8B model shrinks to roughly 5 GB with only a small quality loss. Ollama’s default tags are 4-bit, usually Q4_K_M, a llama.cpp format that keeps the most sensitive weights at higher precision. Rules of thumb:

Some models ship in their own formats: gpt-oss uses MXFP4, and Gemma offers QAT (“quantisation-aware trained”) tags that hold up better at 4-bit.

Better to run a bigger model at Q4 than a smaller model at Q8.

Context length costs memory too

On top of the weights, the model keeps a KV cache for the conversation. It grows with context length. Ollama’s docs list a default context of 4,096 tokens; going to 32K or 128K can add several gigabytes. Our picker adds ~1.5 GB for a short context. If you need long documents, leave more headroom or enable KV-cache quantisation (OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention).

When it doesn’t fit

Ollama can split a model between GPU and system RAM. It still works, but speed drops sharply, because system RAM is many times slower than VRAM. ollama ps shows the split.

Why bandwidth sets speed

Generating each token means reading (nearly) all the model’s weights. So a rough ceiling is: tokens/s ≈ memory bandwidth ÷ model size. A 5 GB model on a 448 GB/s card tops out near 90 tokens/s; real-world numbers land lower. That’s why a used RTX 3090 (936 GB/s) still beats many newer mid-range cards.

Mixture-of-experts (MoE) models

Models like Qwen3.6 35B-A3B, Gemma 4 26B-A4B and gpt-oss only use a few billion “active” parameters per token. They need memory for the whole model, but run at the speed of a much smaller one. That makes them great for unified-memory machines and for GPU + RAM splits.

Next: see what fits your card in 8GB, 12GB, 16GB or 24GB.