How Much VRAM Do You Need to Run Local AI (2026)

What are the VRAM requirements for running an LLM?

A local LLM needs enough VRAM to hold its weights, its KV cache, and its runtime buffers at the same time. At the community default quantization of Q4_K_M, the working rule is about 0.5 GB of VRAM per billion parameters plus context overhead, so a 7B model lands near 5 GB and a 32B model near 22 GB. Practitioner-reported totals run 1.2 to 1.5 times the model's stated size once the cache and buffers are counted.

VRAM matters more than any other number because it is a cliff, not a slope. When the whole model fits, inference on a consumer GPU generally runs at a practitioner-reported 15 to 50 tokens per second. When it does not, the runtime splits layers between GPU and system RAM and throughput collapses, by 5 to 30 times, with practitioners reporting 1 to 2 tokens per second in the worst offloading cases against 40 to 50 when fully resident. There is no partial credit.

The math, worked out on a 7B model

Start from bits per weight. Quantization is how many bits each parameter gets, and the site's compression table pins the common levels: FP16 is 16-bit, Q8_0 is 8-bit, and Q4_K_M is 4.8-bit in practice, not a clean 4.

• 7B at Q4_K_M. 7 billion weights at 4.8 bits is 4.2 GB. Practitioner-reported figures put a 4-bit 7B at approximately 4 to 5 GB, which matches. Add cache and buffers and the site's catalog reports 5 to 7 GB in real use for 7B and 8B class models, with Llama 3.1 8B at about 6.2 GB.
• 7B at FP16. The same weights at 16 bits is 14 GB, roughly 3.3 times the Q4_K_M footprint, and the practitioner-reported 1.2 to 1.5 overhead multiplier puts the realistic requirement at 17 to 21 GB. That is a 24 GB card doing work an 8 GB card handles at Q4_K_M, for a quality gap practitioner reports put at roughly 3 to 5 percent on most benchmarks.
• Context is the third term. KV cache grows with the context window. quantization.html publishes bands of plus 1 to 2 GB at 8K, plus 4 to 8 GB at 32K, and plus 20 to 40 GB at 128K, while the catalog states the same idea as adding 20 to 50 percent for 32K to 128K contexts. At 4K, below the smallest published band, a 7B at Q4_K_M stays inside 8 GB. At 32K it needs real headroom. At 128K the cache can cost more than the weights, which is why long-context work on a small card is a memory problem before it is a model problem.

Two structural details change the arithmetic. Models using grouped-query attention or multi-head latent attention, such as the DeepSeek and Llama 3 families, hold much smaller caches for the same context. Mixture-of-experts models are a trap in the other direction: every expert weight must be resident even though only a fraction activates per token, so a 236B MoE with 22B active still needs roughly 120 GB at Q4. MoE buys you speed, not memory.

All of the above is inference. Practitioner-reported training and fine-tuning requirements are two to four times more VRAM, and sometimes six to eight times, because gradients and optimizer states also have to live in memory.

How much VRAM do you need for a local LLM at each model size?

Use this table as the fast answer. Q4_K_M leads because it is the community default and the quantization every VRAM figure on this site assumes. Q8 roughly doubles the weight cost for quality the site describes as essentially indistinguishable from FP16.

The Q8 column is computed at roughly 1 GB per billion parameters plus overhead, the ratio implied by the site's Q8_0 row (a 70B at Q8_0 is about 70 GB on disk, about 74 GB in VRAM). Where the practitioner-reported figures and the catalog disagree, both figures are printed rather than averaged.

Above 70B the curve stops being friendly. GPT-OSS 120B MoE needs 60 GB or more, Qwen3 235B MoE needs 55 GB or more, Llama 4 Scout (109B total, 17B active) needs about 55 to 60 GB at Q4_K_M, and DeepSeek V3 class models at 671B need 300 GB or more, which is multiple data center GPUs and not a desk decision.

What is the best GPU for running an LLM locally in 2026?

There is no single best GPU, only the cheapest card that clears the model tier you want. For most people that is a 24 GB card, the point where a 32B model at Q4_K_M fits with room for context. Below that, 16 GB runs 14B comfortably and 12 GB runs 8B with headroom.

Read the market by capacity bracket rather than by product name:

• 12 GB. Runs 8B models at Q4_K_M with real context headroom, and 14B at a squeeze. The entry point where local AI stops feeling like a demo.
• 24 GB. The consumer sweet spot. The RTX 4090 sits here, and a 32B at Q4_K_M fits easily at about 22 GB. A 70B at Q4_K_M does not fit and needs offloading or a lower quant.
• 32 GB. The RTX 5090 brought 32 GB to consumer hardware, which widens context headroom at 32B rather than unlocking a new model tier.
• 48 GB and above. Professional territory. Capacities beyond 32 GB are generally restricted to professional cards or multi-GPU setups. The RTX PRO 6000 Blackwell at 96 GB is practitioner-reported as able to hold a 70B on a single card with headroom.
• Data center. An H200 with 141 GB of HBM3e is practitioner-reported at upward of $30,000, the tier where renting is usually the rational choice.

Two caveats most buying guides skip. Capacity is not the only axis: memory bandwidth determines whether the compute cores sit idle, and it feeds directly into response time. And multi-GPU is not free VRAM, because splitting a model across cards adds inter-card latency and configuration complexity and does not behave like one large card. NVIDIA remains the default recommendation because CUDA support is the widest, which is an ecosystem fact rather than a performance verdict.

What can you actually run on 8 GB of VRAM?

Quite a lot, as long as you stay at Q4_K_M and keep context modest. 8 GB is explicitly the comfortable floor for 7B and 8B class models, where a practitioner-reported 4-bit model needs 4 to 5 GB and the remaining budget absorbs the KV cache and runtime overhead.

From the site's catalog: Llama 3.1 8B at about 6.2 GB, Qwen3 8B at 6 to 7 GB, DeepSeek-R1 distilled 7B and 8B at 5 to 7 GB, Mistral 7B at the low end of a 5 to 9 GB range, Phi-4-mini 3.8B at about 3 GB, and the 1B to 4B tier of Llama 3.2, Gemma 3 and Qwen3 between 1 and 4 GB. An 8 GB card also hosts the supporting cast: an embedding model for retrieval at 0.5 to 7 GB depending on size, Distil-Whisper at about 3 GB, and Kokoro-82M for speech synthesis at under 1 GB.

What 8 GB does not do is long context on a mid-size model. Adding a 32K window to an 8B model pushes you into the band where the cache alone is measured in gigabytes. If you need both, take a smaller model at a higher quantization rather than a larger one at a lower quantization. The site's own comparison is blunt: a 32B at Q8 produces cleaner code than a 70B at Q3.

What GPU do you need for local AI if you are buying rather than upgrading?

Buy for the model tier you will actually use every day, not the one you want to benchmark once. In practice that means 12 GB if local AI is a convenience, 24 GB if it is part of your workflow, and unified memory or multi-GPU if you genuinely need 70B class output.

The published hardware tiers map onto that decision. Tier 1 at 4 to 8 GB covers laptops, integrated GPUs and entry Macs, running models up to about 8B for chat, summarization and simple automation. Tier 2 at 12 to 24 GB covers 14B to 32B models and is the sweet spot for coding, reasoning and agent work. Tier 3 at 32 to 80 GB is multi-GPU or data center silicon for 70B and above. Tier 4 is 80 GB and up, multi-node, for the 200B to 744B MoE giants.

Before committing, price the alternative honestly. Local hardware wins on privacy, offline operation and per-token cost at volume, and the counterweight is direct: high-VRAM GPUs are expensive enough to be a real barrier to entry, and the card can age out as model sizes climb. If your workload is occasional, renting compute is not a defeat.

Apple Silicon: how much unified memory do you need?

Apple Silicon changes the calculation because unified memory is shared between CPU and GPU, so a Mac can load models that would otherwise need multiple NVIDIA cards. Practitioner reports call M-series Macs a strong option offering competitive performance and value, particularly for models above 32B.

Sizing, in practice:

• 16 GB to 24 GB. Equivalent to a Tier 1 to low Tier 2 GPU. Comfortable for 7B and 8B at Q4_K_M, workable at 14B.
• 32 GB. Runs 32B class models well. A 70B at Q4 technically loads, but the site's stack notes record it fitting only with swap pressure and recommend a 32B at Q6 instead.
• 64 GB. The site's guidance is that 64 GB runs a 70B easily, with room for a retrieval model, speech-to-text and speech synthesis alongside it. This is the tier practitioner reports mean when they call 64 GB and 128 GB Macs a real alternative to multi-GPU NVIDIA builds.
• 128 GB and above. Opens MoE models and very long contexts that no single consumer GPU reaches.

Two Mac-specific notes: use MLX where available, since it is the recommended framework on Apple Silicon, and expect generation to be memory-bandwidth-bound rather than compute-bound. A Mac's advantage is what it can hold, not how fast it finishes.

How much VRAM do you need for local image generation (FLUX and SDXL)?

Image models are lighter than the LLM tiers above, and 8 to 12 GB covers the most widely used open model. SDXL 1.0, at 3.5B parameters in the UNet, runs in 8 to 12 GB. FLUX.1 at 12B spans 8 to 24 GB depending on precision: full FP16 wants 24 GB, GGUF and NF4 quants bring it to 6 to 8 GB.

The rest of the current field, from the site's catalog:

• Stable Diffusion 3.5. Medium fits in 12 GB, Large needs 24 GB.
• FLUX.2. Spans 13 to 24 GB across variants, with the 4B Klein variant fitting 13 to 16 GB and the 32B Dev variant needing considerably more.
• Stable Cascade. Its three-stage architecture compresses the latent space enough to use about 40 percent less VRAM than SDXL at comparable quality.
• PixArt. At 600M parameters it is competitive with SDXL and is the practical pick for constrained setups.

The planning mistake to avoid is treating image and text budgets as separate. If you want a chat model and an image model available at once, the footprints add. The site's 24 GB creative stack pairs FLUX.1 Schnell at about 8 GB with the rest of the pipeline inside the same card, and the 12 GB stack notes SDXL Turbo fits at about 8 GB only when the LLM is unloaded. On a single card, sequencing beats wishful thinking.

How do you calculate VRAM for a model yourself?

Three steps, no tooling required. Multiply parameters in billions by the bytes per weight for your quantization (0.6 for Q4_K_M at 4.8-bit, 1.0 for Q8_0, 2.0 for FP16), add the KV cache for your context length, then add 20 to 50 percent of the total as runtime overhead. For a fast mental estimate at the default quantization, the catalog's shorthand is VRAM ≈ (Params_B × 0.5) + KV_overhead.

Worked end to end for a 14B at Q4_K_M with a 32K context: 14 times 0.6 is 8.4 GB of weights, the published 32K band adds 4 to 8 GB, and the total lands between 12 and 16 GB. That is a 16 GB card, and it agrees with the site's card figures of about 10.7 to 11 GB for 14B models at shorter contexts.

If you would rather not do the arithmetic, the model catalog carries a live VRAM Estimator behind the floating calculator button: pick a model size from 1B to 671B, a quantization, and a context length from 2K to 128K, and it returns an estimate plus a fit check against 12 GB, 24 GB, 48 GB and 80 GB hardware. It is an estimator, not a benchmark. Treat its output as a planning figure and check it against the per-model VRAM figures in the catalog before buying anything.

Frequently asked questions

How much VRAM do I need to run a local LLM?

At Q4_K_M, budget about 0.5 GB per billion parameters plus context overhead. That puts 7B and 8B models at 5 to 7 GB, 14B at 10 to 11 GB, 32B at about 22 GB, and 70B at 45 to 50 GB. Practitioner-reported weights-only figures are lower, 4 to 5 GB for a 4-bit 7B and around 9 GB for a 14B, the difference being KV cache and runtime buffers. Buy for the higher number.

What happens if a model does not fit in VRAM?

The runtime offloads layers to system RAM for the CPU to process, which is practitioner-reported at 5 to 30 times slower, the difference between 40 to 50 tokens per second and 1 to 2. The model still runs, so this arrives as a usability failure rather than an error message, which is why VRAM gets underestimated so often.

Is a 24 GB GPU or a 64 GB Mac better for local AI?

They solve different problems. A 24 GB GPU is faster on the models it can hold and has the wider ecosystem through CUDA. A 64 GB Mac holds far larger models, including a 70B at Q4 with the supporting stack alongside it, but generation is memory-bandwidth-bound. Pick the GPU for speed at 32B and below, the Mac for 70B class output without a multi-GPU build.

Should I run a bigger model at lower quality or a smaller model at higher quality?

Smaller and higher, in most cases. Quality holds at roughly 95 to 97 percent of FP16 at Q4_K_M, practitioner-reported, and is essentially indistinguishable from FP16 at Q8, but below Q4 the losses land on the tasks people care about: math, structured code and multi-step reasoning. Practitioner reports put the cost of aggressive 2-bit quantization at 8 to 15 percent of quality. A 32B at Q8 will write cleaner code than a 70B at Q3.

Sources

1. Quantization Explained, local-ai-models.ai: compression table for Llama 3.3 70B across FP16 to IQ2_XS, quality-sensitivity table, RTX 4090 and 16 GB scenarios, and technical notes on MoE memory, context scaling and CPU offloading. Page updated August 3, 2026.
2. Model catalog, local-ai-models.ai: the VRAM formula bar, the four published hardware tiers, per-model VRAM badges across the 100-model catalog (Llama, Qwen3, Gemma, Phi-4, Mistral, DeepSeek, GPT-OSS, SDXL, FLUX.1, FLUX.2, Stable Cascade, PixArt, Whisper, Kokoro), the catalog FAQ entry "How much VRAM do I need to run a local LLM?", and the live VRAM Estimator widget.
3. Recommended stacks, local-ai-models.ai: six pre-vetted stacks by hardware tier, including the Mac M-series unified-memory stack, the 12 GB stack and the 24 GB creative stack. Page updated August 3, 2026.
4. Practitioner-reported figures: a web-grounded research pass run August 10, 2026 across community and vendor writeups. Publishers recorded in that pass include apxml.com, corsair.com, plugable.com, lenovo.com, virtualizationreview.com, cybernews.com, nano-gpt.com, dataoorts.com and opensourcesai.com. No durable per-result URL was available, so these figures are attributed by publisher rather than linked.