Recommended Stacks — Complete Local AI Setups by Hardware

— Choosing a stack

How to choose a local-AI stack

The six stacks above are worked examples. This section explains the reasoning behind them, so you can adapt one to your own hardware and goals, or build a fresh one from scratch.

A stack is not a single download. It is the working combination of a few independent layers that each do one job. Swap any layer and the rest usually keeps working, which is exactly why thinking in stacks, rather than in individual models, is the practical way to run AI locally.

The four layers

Every local setup is some arrangement of the same four pieces. Understanding what each layer contributes is most of the decision.

Layer 1

The model

The weights themselves: a chat model, a coding model, an embedding model, or a speech or image model. The model sets the ceiling on quality and the floor on memory. Its quantization (how tightly the weights are compressed) decides how much of your hardware it consumes. Browse options in the model catalog.

Layer 2

The runtime

The inference engine that loads the weights and turns your prompt into tokens. It handles the math, the context cache, GPU offload, and how many requests run at once. The same model can run on several different runtimes, and the one you pick shapes speed, concurrency, and how the model is exposed to other software.

Layer 3

The hardware

GPU VRAM, or unified memory on Apple Silicon, is the hard constraint: the model plus its context has to fit. As a rough rule of thumb, a small 7B or 8B model at 4-bit needs on the order of 5 to 6 GB of VRAM, and larger models scale up from there. The stack builder computes the real figure for a given model and context length.

Layer 4

The agent framework optional

The orchestration layer on top: tool calling, memory, retrieval, and multi-step planning. You only need it when the model has to do things (call tools, read files, chain steps) rather than just answer. Plenty of stacks never include one.

How the pieces fit together

Think of it as a chain running bottom to top. Hardware sets the budget. The runtime spends that budget by loading a model and serving it, usually behind a local API endpoint. Applications (a chat window, an editor plugin, a retrieval pipeline, or an agent framework) connect to that endpoint and never touch the weights directly. Because the layers talk through stable interfaces, you can upgrade the model without changing your app, or move to faster hardware without rewriting anything. A stack is balanced when no single layer is wasted: a large model starved of VRAM will crawl, and fast hardware behind a tiny model leaves quality on the table. The token simulator helps you feel that balance before you commit.

Matching a stack to a use case

The job you have in mind narrows the choices more than the hardware does. A few common shapes:

Chat / assistant

A single general-purpose chat model and a runtime is enough. Favor a model that fits comfortably, with room for a long context window, over the largest one that barely fits. Speech in and speech out is an optional add-on that loads on demand.

Coding

Prioritize a model trained for code and a generous context window, so it can see whole files. Raw parameter count matters less than instruction-following and low latency, since you work in a tight loop. An editor integration replaces the chat window.

RAG / document Q&A

Now you need two models: a chat model to answer, and a small embedding model to index your documents, plus a vector store and often a reranker to sharpen results. The embedding model is cheap in memory but central to answer quality.

Image / audio

These are their own model families (diffusion for images, dedicated speech models for audio) and usually run one at a time rather than all resident at once. Memory planning is about the peak single model, not the sum, so sequential loading is normal.

Agents

Add the framework layer and pick a model with reliable tool calling. Agents make many short calls, so serving throughput and a runtime that handles concurrency matter as much as single-answer quality. Explore options in agent frameworks.

Three ways to run it: the deployment tradeoff

The same model can be served in very different packages, and the package you choose is mostly about how much control you want versus how much setup you are willing to do. The three broad styles overlap, and many tools blur the lines, but the tradeoffs are consistent.

All-in-one desktop app

A single install that bundles a runtime, a model downloader, and a chat interface. Easiest to start with and hard to misconfigure, at the cost of visibility and fine control, since you work within whatever the app exposes. Ideal for a first local model, a personal assistant, or anyone who wants a working chat window in minutes.

Server runtime

A runtime you run yourself, headless, tuned by config. More setup and more decisions, but full control over memory, concurrency, and performance, and it can serve many applications at once. This is the shape most multi-model stacks and shared workstations end up using.

API-compatible server

A server runtime that mimics a well-known cloud API surface, so existing tools written for that API can point at your local endpoint with a change of URL. The advantage is the ecosystem: editor plugins, agent frameworks, and scripts work unmodified, while the model underneath stays swappable.

None of these is strictly better. A sensible default is to start with an all-in-one app to confirm a model does what you need, then graduate to a server or API-compatible runtime once you want it available to other tools or running unattended. Because only the runtime layer changes, your model choice and hardware plan carry straight over.

Frequently asked

What's in a local-AI stack?

Four layers: a model (the weights), a runtime that loads and serves it, the hardware it runs on, and, only when the model needs to take actions, an agent framework on top. A pure chat setup uses the first three; a document Q&A setup adds a second, smaller embedding model. See the six stack cards above for concrete, VRAM-checked combinations.

How do I pick a runtime?

Start from how you will use the model. If you want a chat window with no fuss, an all-in-one desktop app bundles a runtime for you. If the model needs to serve several tools at once or run unattended, a headless server runtime gives you control over memory and concurrency. If you want existing editor plugins and agent tools to work as-is, choose a runtime that exposes a familiar API surface. Your hardware matters too: GPU, Apple Silicon unified memory, and CPU offload each favor different engines. The stack builder shows what fits.

Do I need an agent framework?

Only if the model has to take actions rather than answer questions. A chat assistant or a document Q&A system does not need one. Add a framework when you want the model to call tools, run multi-step workflows, read files, or coordinate with other agents, and pair it with a model that does tool calling reliably. The agent frameworks catalog covers the options.

Air-gapped / enterprise, what changes?

Two things move to the front. First, everything has to be downloaded and staged ahead of time, because there is no internet after deployment: models, runtime, and dependencies all travel with you, and versions get pinned for reproducibility. Second, licensing becomes a gating decision, since permissively licensed components (for example Apache 2.0 or MIT) are the ones you can audit and redistribute inside a regulated environment. The air-gapped stack card above is built entirely from permissive licenses as an example of that discipline.

Where do I see recommended stacks?

The six cards at the top of this page are ready-to-copy stacks, each VRAM-checked and grouped by hardware tier. To build your own, the stack builder lets you pick hardware and see which models fit, the model recommender suggests a model from your goals and constraints, and the workstation builder plans the hardware itself. If you are new to all of this, getting started walks through your first model end to end, and community data shows what people are actually running.

Complete stacks