VRAM: the one number that decides which AI models you can run
If you're shopping hardware for local AI, one spec matters before all the others: VRAM — the memory on your graphics card. It decides which models you can run at all. A faster GPU with less memory loses to a slower GPU with more, because a model that doesn't fit in memory doesn't run slightly worse — it runs dramatically worse, or not at all. Get the VRAM question right and every other spec decision gets easier.
TL;DR
- VRAM (video RAM) is your GPU's onboard memory. For local AI, it's the hard ceiling: the model's weights plus its working memory must fit inside it for full-speed generation.
- Rough fit guide at the standard 4-bit quantization (Q4): an 8B model wants ~6–7 GB, a 14B ~10–12 GB, a 32B ~22–24 GB, and a 70B ~40+ GB — which is why 8 GB, 16 GB, and 24 GB cards land in very different tiers. ⏱️ verified 2026-07-03
- Quantization shrinks models to fit smaller cards. Q4 roughly halves memory versus Q8 with a quality drop most people can't notice in normal chat.
- Speed follows memory too: how fast a model generates text tracks your card's memory bandwidth divided by the model's size. Same card, smaller model = faster output.
- Compute (how "powerful" the GPU is) matters far less than the spec sheets suggest. Capacity first, bandwidth second, compute third.
What is VRAM, and why does AI care so much about it?
VRAM is the dedicated memory chip-set on your graphics card — separate from your computer's regular RAM. When you run a local AI model, the entire model gets loaded into VRAM so the GPU can read it at full speed.
Here's the part that makes it the number: to generate each word, the GPU has to read through essentially all of the model's weights. Not some of them — all of them, every single token. That means two things:
- The whole model needs to be in VRAM. If it doesn't fit, the overflow spills into regular system RAM, which the GPU reaches over a much slower connection. Generation speed doesn't dip — it falls off a cliff, often to a tenth of the speed or worse.
- Reading speed is the bottleneck, not math speed. Modern GPUs can do the arithmetic far faster than they can fetch the data. So for one user chatting with one model, generation is limited by memory, not compute.
That's why a used 24 GB card from 2020 is still prized for local AI in 2026 while newer, "faster" 8 GB cards are not. The old card holds more model.
How much VRAM does each model size actually need?
The quick answer: at the standard Q4 quantization, budget roughly 0.6–0.75 GB of VRAM per billion parameters, including working memory. Here's the real table (llama.cpp VRAM guide, llama.cpp hardware discussion, ⏱️ verified 2026-07-03):
| Model size | Q4 file size (weights) | Realistic VRAM needed | Fits comfortably on |
|---|---|---|---|
| 7–8B | ~4 GB | ~6–7 GB | 8 GB cards |
| 13–14B | ~7–8 GB | ~10–12 GB | 12–16 GB cards |
| 32B | ~19 GB | ~22–24 GB | 24 GB cards |
| 70B | ~39–40 GB | ~40+ GB | 48 GB (usually two cards) |
Why is "VRAM needed" bigger than the file size? Because the model's weights aren't the only tenant. The KV cache — the model's working memory of your conversation — takes space too, and it grows with how much text is in play. A long document pasted into the chat costs real gigabytes. That's the gap between "the file is 19 GB" and "you'll want 24 GB."
The tiers explain the used-GPU market: 8 GB cards are 8B-class machines, 16 GB cards handle 14B well, 24 GB is the entry ticket to 32B, and 70B is multi-GPU territory. There's no clever setting that moves a card up a tier — only quantization moves the models down.
What is quantization, in plain terms?
Quantization stores each of the model's numbers with fewer bits — like saving a photo as a smaller JPEG. The model was trained with high-precision numbers (16-bit); quantization rounds them to 8-bit (Q8) or 4-bit (Q4) versions, shrinking the file so it fits smaller cards.
What you give up is precision, and the honest numbers are smaller than you'd guess (quantization quality comparison, ⏱️ verified 2026-07-03):
- Q8 is nearly indistinguishable from the original full-precision model.
- Q4_K_M (the common default — the "K_M" means it keeps the most sensitive layers at higher precision) measures roughly 3–5% worse on quality benchmarks than Q8. In normal conversation that shows up as occasional different word choices, not wrong facts.
- Below Q4 — the Q3/Q2 territory — quality loss starts becoming something you can feel, especially in reasoning and code.
And there's a bonus most guides skip: Q4 isn't just smaller, it's faster. Since generation speed is limited by how much data moves per token, a model at half the size moves half the data — Q8 measured about 29% slower than Q4_K_M on the same hardware in recent testing. For most people, Q4_K_M is the right default: half the memory, most of the quality, more speed.
How fast will a model run on my card?
A rule of thumb gets you a real estimate before you buy anything:
Generation speed (tokens/second) ≈ memory bandwidth (GB/s) ÷ model size in memory (GB)
The logic: each token requires reading the whole model out of VRAM once, so tokens-per-second is just "how many times per second can this card read that many gigabytes." A token is roughly three-quarters of a word.
Two examples with real cards:
- A card with ~448 GB/s bandwidth running a 4 GB (8B Q4) model: ~448 ÷ 4 ≈ up to ~110 tokens/s theoretical — expect a healthy fraction of that in practice. Fast, chat feels instant.
- The same card on a 19 GB (32B Q4) model — if it even fit: ~23 tokens/s ceiling. Usable, noticeably slower.
The estimate runs optimistic (overhead eats some of it), but it gets the tier right every time, and it explains the market: people pay for high-bandwidth 24 GB cards because both terms of the fraction work in their favor.
So which spec should I actually shop for?
In order: capacity, bandwidth, then everything else. First decide the biggest model class you honestly want to run (be realistic — 8B and 14B models handle most hobbyist work well). That sets your minimum VRAM tier. Then, among cards at that tier, prefer higher memory bandwidth — it's the speed number. GPU compute specs (cores, TFLOPS) mostly matter for how fast the model reads your prompt, which is a secondary concern for chat-length inputs.
One warning shaped by this: a card with generous VRAM but weak bandwidth holds big models without delivering them at a satisfying speed. Check both numbers.
When this won't help
- Apple Silicon works differently. Macs use unified memory — the GPU shares the system's RAM pool — so "VRAM" isn't a separate number there. The capacity logic still applies (the model must fit in memory), but the shopping math is about total RAM and the chip's memory bandwidth.
- Mixture-of-experts (MoE) models bend the speed rule. They store many "expert" sub-networks but only read a few per token, so they generate faster than their total size suggests. The capacity rule still holds — all the experts must fit in memory — but the bandwidth-÷-size speed estimate runs pessimistic for them.
- Serving many users at once changes the bottleneck. With heavy batching (many simultaneous requests), compute starts to matter and the single-user rules above stop being the whole story. For a homelab serving you and your household, they hold.
- This won't pick your model for you. Fitting a 32B model doesn't make it the right tool — a good 14B often beats a heavily-squeezed 32B at Q2. Fit the quantization tier that keeps quality, not the biggest number that technically loads.
FAQ
How much VRAM do I need to start with local AI? 8 GB runs today's 7–8B models comfortably at Q4 — genuinely useful for chat, summaries, and light coding help. 16 GB opens up the 14B class, and 24 GB is the entry point for 32B models. Start with what you have; an 8B model on existing hardware teaches you more than a spec sheet.
What happens if a model doesn't fit in VRAM? Tools like Ollama and llama.cpp will still run it by spilling the overflow into system RAM, but generation slows drastically — often below the speed of comfortable reading. It works for experiments; it's not something you'd use daily.
Is Q4 quantization "good enough," really? For chat, writing, and general use — yes. Measured quality loss versus Q8 is in the low single digits and shows up as wording variation, not wrong answers. For precision-heavy work like code generation, stepping up to Q6 or Q8 is a defensible use of spare VRAM.
Does my CPU or system RAM matter at all? Some. System RAM stages the model during loading and catches any overflow, and the CPU handles everything around the model. But neither rescues a too-small GPU — for the model itself, VRAM capacity and bandwidth do the work.
Why do people recommend old 24 GB cards over new 16 GB ones? Capacity beats generation. A 2020-era 24 GB card runs a whole model class (32B) that no 16 GB card can hold, and its memory bandwidth is still competitive. Newer architecture doesn't move a card up a capacity tier.
Last updated 2026-07-03. Time-sensitive: model VRAM figures reflect current-generation Q4_K_M builds and typical context sizes; specific model families vary by a gigabyte or two — check the model's download page for its actual file size before assuming fit.
Sources (verified 2026-07-03): model file sizes and VRAM requirements — llama.cpp VRAM requirements guide, llama.cpp hardware specs discussion (ggml-org); quantization quality and speed trade-offs — Q4/Q5/Q6/Q8 quality loss comparison, GGUF quantization guide.