Basics

What is VRAM — and why a "7B" model doesn't weigh 7 gigabytes

Illustration: a model as cargo being measured against a graphics-card shelf

Here's the trap everyone falls into the first time they run a model on their own machine.

You see "Llama 7B" and think: 7 billion parameters, so probably about seven gigabytes. I've got 8 gigabytes of video memory — should just about fit.

It won't. Or it'll fit with room to spare. Depends which version — the same "7B" model can take 14 gigabytes or under five. That's a 4× difference, and it's not about the model. It's about what kind of numbers its weights are stored in.

What VRAM is, in plain words

VRAM (video RAM) is memory soldered directly onto the graphics card. Separate from your computer's regular RAM.

Why separate: a GPU is only fast when the data sits right next to it. Shuttling it back and forth from ordinary RAM over the bus is like cooking while running to the shop for every single ingredient.

So the rule is hard: the model has to fit in VRAM entirely. Not "mostly," not "almost." If it doesn't, some layers spill into regular RAM and speed drops several-fold. You'll notice immediately: instead of a stream of text, words start dripping out one at a time.

Apple's M-series chips share memory — there VRAM and RAM are the same thing, and the threshold is softer. But on a normal PC with a discrete card, do the math honestly.

How to calculate what you need

The formula is surprisingly simple. Take the parameter count and multiply by the number of bytes per parameter:

| Weight format | Bytes per parameter | A 7B model | |---|---|---| | FP32 (full precision) | 4 | ≈ 28 GB | | FP16 / BF16 (typical release format) | 2 | ≈ 14 GB | | 8-bit (Q8) | 1 | ≈ 7 GB | | 4-bit (Q4) | ≈ 0.5 | ≈ 3.5 GB |

That's where the confusion comes from. "7 gigabytes" is the answer in exactly one case — 8 bits per parameter. Models are usually released in FP16, i.e. 2 bytes each, and then the same 7B needs roughly 14 gigabytes.

Shrinking the bit count is called quantization — the numbers inside the model are coarsened, and it compresses several-fold while losing surprisingly little quality.

Let's check against a live example

Take a real number. Run Mistral through Ollama and ask it which models are loaded, and it reports something like: parameter size 7.2B, quantization Q4_0, video memory used — 5137025024 bytes.

That's about 4.8 gigabytes. Check it against the formula: 7.2 billion × 0.5 bytes ≈ 3.6 GB for the weights alone. So where does the extra gigabyte-plus come from?

That's the second piece people forget: context takes memory too. The model holds more than weights in VRAM — it also holds intermediate computations for the whole conversation, the so-called KV cache. And the longer the context, the bigger it gets.

Hence the rule of thumb: calculate the weights, then add 20–30% on top. A model that "just barely fits" will hit the ceiling on a long conversation and spill into regular RAM.

What to do if it won't fit

In order, cheapest to most painful:

  1. Grab a more coarsely quantized version. Q4 instead of Q8 halves the memory. This is usually the first and most sensible move: quality drops far less than you'd expect.
  2. Shorten the context length. Many runners reserve memory for a large context by default. If you don't need tens of thousands of tokens, set it lower and free up gigabytes.
  3. Take a smaller model. A good 3B that flies beats an excellent 13B that thinks for a minute.
  4. Go to the cloud. For a one-off task, paying for API calls is cheaper than buying a graphics card.

How all of this works in practice is in the step-by-step guide on running AI models locally.

How much VRAM is enough to start?

8 gigabytes is the working minimum: models up to 8B in 4-bit quantization live there comfortably. 16 gigabytes opens up 20B models and a roomy context. 24 gigabytes gets you into the 30B class.

A reference point from Ollama's own developers: they test 7–8B models against 6–8 GB, 20B against 16 GB, and 27–33B against 20–32 GB. Giant 120B models need around 70 GB — no home card is going to cover that.

Can I combine two graphics cards?

Sometimes — some runners can split a model's layers across multiple GPUs. But it isn't a clean "8 + 8 = 16": traffic between cards is slower than within one, and the setup is noticeably fiddlier. For a first local run, it's simpler to pick a model that fits in a single card.

Why does the model use more memory than the file size?

The file on disk is just the weights. In memory you add the KV cache for context plus working buffers. So a 4.4 GB file comfortably occupies 5–6 GB of video memory — that's normal, not a bug.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →