Before you buy parts, size the GPU first: on a local-AI box the model's weights and working memory have to fit in VRAM, and the CPU, system RAM, power supply, and cooling all exist to keep that GPU fed.
Graphics Cards
02 Models That Fit Your Build
Mapped against the 100 models in our index. Green = comfortably fits at Q4_K_M with 8K context · Yellow = tight fit, may need lower quantization · Red = won't fit.
03 Price & Spec Sources
Every part in the picker links to a real product. Prices reference these primary sources.
- NVIDIA GeForce / Workstation · official MSRP for RTX 4090, 5090, 6000 Ada, A6000
- AMD Ryzen / Threadripper · official launch pricing
- Newegg · live street pricing for memory, storage, PSU, case, cooling
- Micro Center · CPU/motherboard combo pricing reference
- PCPartPicker · cross-retailer compatibility checking
- Tom's Hardware reviews · power-draw measurements per GPU under sustained inference load
04 How to Size a Local-AI Workstation
A parts list is downstream of one decision: which models you want to run, and at what context length. Get that right and the rest of the build follows. This is an evergreen primer on the reasoning; use it alongside the picker above and the recommender for exact, per-model numbers.
VRAM is the binding constraint
For local inference, the model's weights and its working memory have to live in memory the accelerator can read at full speed. On a discrete GPU that memory is VRAM. When the weights fit in VRAM, the GPU streams through them at its native bandwidth and generation is fast. When they do not, layers spill over the PCIe bus into system RAM and onto the CPU, which are far slower, and throughput collapses. That is why capacity, not raw compute, is usually the first thing that decides whether a model runs well: teraflops set how fast a model that already fits will run, while VRAM capacity sets whether it fits at all. Choose the GPU to hold your target model first, then optimize for speed.
Model size and quantization set the footprint
Two numbers drive how much VRAM the weights need: the parameter count and the precision each parameter is stored at. At full 16-bit precision every parameter is about two bytes, so an 8-billion-parameter model's weights alone are on the order of 16 GB. Quantization stores each weight in fewer bits, shrinking the footprint in exchange for a small, usually acceptable quality cost. A widely used rule of thumb, best treated as approximate: 4-bit quantization uses roughly half a gigabyte of VRAM per billion parameters for the weights, plus overhead for the KV cache, activations, and the runtime itself. So a 7-8B model at 4-bit lands around 5-6 GB, and larger models scale up from there. These are ballpark figures; the exact number moves with the quant scheme, the context length, and the runtime. For a precise per-model figure, lean on the site's tools rather than a memorized value: the quantization guide explains the trade-offs, and the recommender and model index report fit against real hardware.
Every other part exists to feed the GPU
Once VRAM is settled, the rest of the build is about keeping that GPU fed and stable under a long, steady load. Here is how the supporting parts earn their place.
The GPU
Does the matrix math and holds the weights. VRAM capacity gates the model size you can load; memory bandwidth gates how many tokens per second you get out. It is the one part worth choosing first and spending the most on.
CPU & System RAM
The CPU handles loading, tokenization, sampling, and orchestration; system RAM stages the model file and runs the OS and everything else. Aim to exceed the model file size comfortably, especially if you plan to offload part of a model to the CPU when it will not quite fit VRAM. CPU-only inference works, but it is far slower than the GPU.
Power Supply
Must supply the sustained draw of the GPU or GPUs plus the CPU and the rest, with margin for transient spikes. An undersized or low-quality unit causes shutdowns under load. Multi-GPU builds especially want a generous, high-quality PSU; this is not the place to cut corners.
Cooling & Airflow
Inference is a long, steady load, not the short bursts a game produces, so the GPU and CPU sit hot for minutes at a time. Adequate CPU cooling and case airflow keep clocks from throttling and hardware healthy across long sessions. Plan thermals for continuous load, not a momentary peak.
Unified Memory vs Discrete GPU
Apple Silicon shares one high-bandwidth memory pool between CPU and GPU, so a large share of system memory acts as VRAM: a capacity-friendly, low-power way to hold big models. Discrete Nvidia and AMD GPUs give dedicated VRAM with very high bandwidth and the strongest raw throughput, but capacity is fixed per card and you scale by adding cards. If you are capacity-bound, unified memory stretches further per watt; if you are speed-bound, discrete GPUs pull ahead.
Context Headroom
Weights are not the whole story: the KV cache grows with context length and with concurrent requests, and it lives in VRAM too. A model that just fits at short context may not fit at long context. Size VRAM for the weights plus the context window you actually intend to use, and leave headroom. The token simulator helps you reason about context and throughput.
- Entry · single mainstream GPU Small models around 7-8B at 4-bit fit comfortably with room for modest context. Plenty for chat, coding assistants, and retrieval over a handful of documents. A sensible first local-AI box.
- Mid-range · higher single-card VRAM Opens up mid-size models, roughly the 13-14B class and toward 30B, at 4-bit, with more context headroom for longer prompts and documents.
- High-end · top single card or dual GPUs Brings the 30B-to-70B class into reach at 4-bit, depending on quant level and context length. Watch cooling and PSU headroom as you add cards.
- Frontier · very large / mixture-of-experts 70B-plus dense models and large MoE models generally want multiple GPUs or a large unified-memory machine.
- These tiers are rules of thumb, not guarantees. Exact fit depends on quantization and context length, so confirm against the workstation builder above and the recommender before you buy.
How much VRAM do I actually need?
It depends on the model size, the quantization level, and the context length you plan to use. As an approximate rule of thumb, 4-bit quantization needs roughly half a gigabyte per billion parameters for the weights, plus overhead for the KV cache and runtime, so a 7-8B model lands around 5-6 GB and larger models scale up from there. Add headroom for longer context, since the KV cache also lives in VRAM.
Rather than commit to one number, check the exact fit for a specific model with the recommender, the workstation builder, and the quantization guide.
Do I need a top-end GPU?
Not necessarily. The right GPU is the one whose VRAM holds the largest model and context you actually intend to run, at a speed you are happy with. Many genuinely useful models run well on mainstream cards.
Flagship cards earn their price when you need big models, long context, high throughput, or several concurrent users. Match VRAM to your target model rather than to the top of the product stack, then use the workstation builder and stack builder to sanity-check the rest of the parts.
Can I run big models on system RAM?
Technically yes, through CPU offload or full CPU inference, but it is far slower than running on the GPU because system RAM bandwidth and CPU compute are much lower than a GPU's. It is reasonable for occasional use, or for nudging a model that only just overflows VRAM by offloading a few layers, but not for responsive interactive use of large models.
Unified-memory machines such as Apple Silicon are a different case: their shared memory is high-bandwidth and directly GPU-accessible, so what looks like system RAM there behaves much more like VRAM.
Nvidia vs AMD vs Apple Silicon for local AI?
There is no single winner; it depends on what you value. Nvidia has the most mature software ecosystem and the broadest tool support, which usually means the smoothest setup. AMD works and keeps improving, and some cards offer strong VRAM per dollar, though the software path can take more effort. Apple Silicon's unified memory delivers large capacity in a quiet, low-power package and is well supported by popular local runtimes, while its raw compute and bandwidth differ from top discrete GPUs.
Pick based on the software you plan to run, the model sizes you are targeting, and how much setup effort you will tolerate.
How much does a starter build cost?
It genuinely depends, so any figure quoted here would be misleading. Cost is driven mostly by the GPU you choose, whether you buy new or used, and current market conditions, since GPU prices in particular move around a lot. The other parts matter far less to the total.
Instead of a fixed number, assemble real parts in the workstation builder above to see an estimated total for your exact configuration, and use the stack builder for the software side. Spend where it counts, which is VRAM.