Gemma 4 CPU Only: No GPU Needed, How Slow Is It

No discrete GPU? You're not locked out of Gemma 4. Every variant runs on CPU alone — the question is which size, and how much patience you'll need.

Quick Answer

Yes, Gemma 4 runs without a GPU. On a typical office PC or budget mini PC with 32 GB of RAM, the surprise pick isn't the smallest model — it's 26B A4B, the Mixture-of-Experts (MoE) variant. Because it only activates about 3.8B parameters per token, independent testing on a $150 used i5-8500 desktop (32 GB RAM, no GPU) clocked it at ~7 tokens/second — faster than the fully-dense 12B model on the same machine (TerminalBytes). With 16 GB or less, stick to E4B, which is the only realistic choice at that RAM level. Either way, expect several seconds per response, not the instant feel of a cloud chatbot.

What Fits in Your RAM (CPU-Only)

CPU inference has a different math than GPU inference. There's no separate VRAM pool — the model competes with your OS and every other running program for the same system RAM. Our hardware requirements guide puts the rule of thumb at roughly 2× the model's Q4 weight size in free system RAM — one copy for the weights, one for working memory during computation — on top of whatever your OS and background apps already claim (typically 2–4 GB).

System RAM	What Actually Works	Reality Check
8 GB	Nothing comfortably. E2B (the phone/edge model) at a pinch, short prompts only	After OS overhead you're usually left with 4–6 GB free — short of even E4B's ~10 GB CPU need
16 GB	E4B comfortably	12B dense will load and run, but it's tight — expect swapping if you have much else open
32 GB	E4B and 12B comfortably; 26B A4B (MoE) becomes usable	The CPU sweet spot — the MoE model's 15.6 GB weight footprint fits, and it runs close to as fast as 12B despite being roughly double the total parameters
64 GB	Everything, including 31B dense	The only tier where the full 31B is genuinely practical on CPU — it's fully dense, so there's no MoE shortcut to lean on

These are system totals, not "free" numbers — leave headroom for your OS and browser. If you're on the edge of a tier, favor a shorter context window (more on that below) over hoping it fits.

Why the MoE Model Is the CPU Pick

This is the counterintuitive part of running Gemma 4 without a GPU: the bigger model can be faster.

Gemma 4's 26B A4B uses a Mixture of Experts architecture. All 26 billion parameters have to sit in RAM — the router needs instant access to every expert — but for any single token, it only computes through about 3.8 billion active parameters. The 12B model, by contrast, is fully dense: every one of its 12 billion parameters gets touched on every single token, whether you're on a GPU or grinding through it on a CPU.

On a GPU that difference barely shows, since GPUs have bandwidth to spare. On a CPU, where memory bandwidth is the scarce resource, it matters a lot. TerminalBytes' benchmark makes the case directly: on the same $150 i5-8500 desktop with 32 GB RAM and no GPU, 26B A4B ran at ~7 tokens/second, while the dense 12B model on that same hardware dropped to "low single digits" (TerminalBytes). A model with more than double the total parameters ran faster, purely because CPU inference cares about active compute per token, not size on disk.

The catch, worth repeating from our hardware guide: you still need the RAM to hold all 26B parameters, even though you're only computing with a fraction of them. That's why this pick only shows up once you're at 32 GB — below that, you simply can't load it.

Real Speed Expectations

Numbers vary by CPU generation, RAM speed, and quantization, but here's what's actually been measured or reasonably estimated for CPU-only Gemma 4 inference:

Model	Typical CPU speed	Basis
E4B	~2–5 tokens/sec on a modern laptop CPU	Estimate, per our hardware guide
12B dense	Low single digits, up to ~7 tok/s on a strong Ryzen chip	TerminalBytes (i5-8500, "low single digits"); Peladn W04 Ryzen 5 7640HS ran the similarly-sized Gemma 3 12B at 7.1 tok/s (HowToGeek) — a reasonable proxy, not a Gemma 4-specific figure
26B A4B (MoE)	~7 tokens/sec	TerminalBytes, i5-8500, 32 GB RAM
31B dense	Well under 1 token/sec on most consumer CPUs	Estimate — fully dense, no MoE shortcut, consistent with our hardware guide

At 7 tokens/second, expect roughly 5–25 seconds for a typical short reply — fine for a question and answer, rough for a fast back-and-forth. Under 2 tokens/second, plan on tasks you can walk away from: batch summarizing, not live conversation.

DDR4 vs. DDR5 matters more than core count. CPU inference is memory-bandwidth-bound, not compute-bound — the CPU spends most of its time waiting on RAM, not crunching numbers. DDR5 roughly doubles typical desktop DDR4 bandwidth, and it shows up directly in tokens/sec: one test comparing a DDR5-6000 rig to an equivalent DDR4-3200 setup running Llama 3.3 70B at 4-bit found roughly 7.2 tok/s versus an estimated 3.8 tok/s — nearly double (DEV Community). Shopping for a CPU-only box? Prioritize RAM speed and channel count over an extra core or two.

Tuning Tips

A few settings actually move the needle on CPU-only setups:

• Stick with Q4 quantization. It's the standard tradeoff — a quarter the memory of full precision (BF16), with quality loss that's hard to notice in normal use. Going higher (SFP8, BF16) doubles or quadruples RAM and compute cost for a CPU that's already the bottleneck.
• Keep context length short. The KV cache — memory holding your conversation so far — grows with every token, and on CPU that's memory you don't have to spare. Pushing toward Gemma 4's full 128K–256K context can add several gigabytes on top of the base weights. Capping context at a few thousand tokens (plenty for most single questions) keeps both RAM and per-token latency down.
• Set thread count to your physical core count, not logical (hyperthreaded) cores. Ollama and llama.cpp both let you cap threads — OLLAMA_NUM_THREAD as an environment variable or num_thread in the API for Ollama, -t on llama.cpp's CLI. More threads than physical cores typically doesn't help and can cost 5–10% throughput to cache contention. Six physical cores and 12 logical threads via SMT? Try 6 first.
• Close memory-hungry background apps before loading a model. A browser with dozens of tabs can eat several gigabytes that would otherwise go to context — the same tip that applies to running Gemma 4 in Ollama, doubly true with no GPU memory to fall back on.

When to Give Up and Get a GPU or Mac

CPU-only inference is real, but it has a ceiling. If any of these describe you, it's worth budgeting for hardware with dedicated memory bandwidth for AI:

• You need conversational speed, not "ask and come back later" — even a modest discrete GPU or Apple Silicon Mac will beat CPU-only by 5–10× or more.
• You want the 26B A4B or 31B models routinely, not just when your RAM tier happens to allow it. See our hardware requirements guide for exact VRAM/unified-memory numbers per model.
• You're deciding what to actually buy. Our best mini PC for Gemma 4 guide covers real, sourced picks from a ~$300 CPU-only box up to a 16 GB Mac Mini M4 (Metal-accelerated, not CPU-bound) and beyond.
• You're ready to get started once you've picked hardware. Run Gemma 4 with Ollama walks through installation and model selection regardless of whether you end up on CPU, GPU, or Apple Silicon.

FAQ

Can Gemma 4 run without a GPU?

Yes. All five Gemma 4 models run on CPU alone through runtimes like Ollama and llama.cpp — the model weights load into system RAM instead of VRAM. It works on any modern laptop or desktop; the tradeoff is speed, not compatibility.

Is 16 GB RAM enough?

For E4B, yes — comfortably. Budget roughly 10 GB free RAM for the model itself, which a 16 GB machine has after normal OS overhead. 12B dense will technically run on 16 GB but is tight, since it needs closer to 13–14 GB free; expect occasional swapping if you're multitasking. For 26B A4B or 31B on CPU, you'll want 32 GB or more.

How slow is CPU inference, really?

Expect roughly 2–7 tokens/second depending on the model and CPU, translating to 5–25+ seconds per short reply. The surprise is that 26B A4B (MoE) can match or beat the dense 12B model in speed despite being larger, because CPU inference cost tracks active parameters per token, not total model size. Dense 31B is the slowest tier — well under 1 token/second on typical consumer hardware.