Gemma 4 Quantization Guide: Q4_0 vs QAT vs Higher Precision

Every Gemma 4 hardware guide on this site eventually points you toward "Q4_0" or "the official QAT build" without stopping to explain the decision itself. This article is that missing step: what quantization actually does, what you give up at each level, why Google's official QAT checkpoints beat a plain Q4_0 conversion, why one model in the lineup is the exception to that rule, and a decision path from your usable memory (from our memory formula guide) to an actual quant to download.

Scope note: this article is about choosing between quantization levels — Q4_0 vs. SFP8 vs. BF16, and plain Q4_0 vs. QAT. If you want to know what QAT is in more depth, Google's exact June 5, 2026 release details, or the precise Hugging Face repo names and Ollama tags to pull, our dedicated QAT guide covers that ground already and this article deliberately doesn't repeat it. Think of that guide as "what QAT is and how to get it," and this one as "which level should I even be targeting."

What Quantization Actually Does

A model's weights are just numbers, and every number is stored using some fixed number of bits. Quantization lowers that bit count. Fewer bits per number means a smaller file and less memory to hold it — but each individual weight also becomes a coarser, less precise approximation of its original value.

The image-compression analogy from our hardware guide still holds: a high-quality JPEG looks nearly identical to the source image at a fraction of the file size, but push the compression too far and you start seeing artifacts. Quantization is the same trade, applied to a model's weights instead of an image's pixels. The three levels the Gemma 4 documentation covers officially are:

Here are the actual weight sizes across all five Gemma 4 models, from gemma-engine.ts (mirroring Google's published numbers):

The pattern in every row is the same: BF16 → SFP8 roughly halves the size, SFP8 → Q4_0 roughly halves it again. That's the mechanical trade. The question this article actually answers is when it's worth taking.

The Real File Is a Bit Bigger Than the Table

One honest wrinkle before going further: the numbers above are Google's weight-loading minimums — the floor for what has to be in memory. The GGUF file you actually download tends to run a little larger. Our memory formula guide covers why in more depth, but the short version, per the Hugging Face GGUF format docs: a GGUF file bundles tokenizer data and metadata alongside the tensors, and quantization tools commonly keep a handful of sensitive tensors — embeddings, output projections — at higher precision even inside an otherwise "4-bit" model.

You can see the gap directly in our checker's own data (gemma-engine.ts): the E2B's Q4_0 weight-loading minimum is 2.5 GB, but the actual official QAT GGUF download is 4.3 GB — noticeably larger, not just rounding. Across the lineup, actual QAT GGUF sizes run somewhat above the weights-only figures:

None of this changes which quant level you should pick — it's the same file either way. It's just worth knowing that "Q4_0 needs 6.7 GB" is a lower bound on disk space and load-time memory, not the exact download size you'll see on Hugging Face.

Why an Official QAT Checkpoint Beats Plain Q4_0

Here's the part that isn't obvious from the file sizes alone, because QAT and regular Q4_0 land at essentially the same size — 4 bits is 4 bits either way. The difference is entirely in quality, and it comes down to when the precision loss happens:

• Regular Q4_0 (post-training quantization). Train the model at full precision, then compress the finished weights down to 4 bits afterward. The model never got a chance to adapt — it's rounded down after the fact.
• QAT (quantization-aware training). Google simulates the 4-bit rounding error during training, so the model learns weights that hold up once they're actually quantized. Same final bit-width, but the model was trained knowing that's where it would end up.

Our QAT guide has the full release story; the takeaway for this article is simpler: if an official QAT build exists for the model you want, there is essentially no reason to run the plain Q4_0 conversion instead. Same memory cost, meaningfully better output.

The One Exception: Dense Models vs. the MoE Model

This is where quantization choice stops being purely about memory and starts depending on architecture. Four of Gemma 4's five models are dense — every parameter activates on every token — and Google shipped official QAT checkpoints for all four (E2B, E4B, 12B, 31B). Our checker tracks this directly as a field on each model, qatIsSolid, and it's true for all four dense models:

{ id: "e2b",     qatSize: 4.3, qatIsSolid: true  },
{ id: "e4b",     qatSize: 6.1, qatIsSolid: true  },
{ id: "12b",     qatSize: 7.2, qatIsSolid: true  },
{ id: "26b-a4b", qatSize: 16,  qatIsSolid: false },
{ id: "31b",     qatSize: 19,  qatIsSolid: true  },

The 26B A4B model is the fifth model and the odd one out: it's a Mixture-of-Experts (MoE) architecture, with 128 total experts (8 active plus 1 shared per token) rather than one dense stack of weights. MoE experts are structurally narrow — each expert has a hidden dimension of only 704 — and narrow layers have little numerical redundancy left to absorb 4-bit rounding error. That's the specific, architectural reason 26B A4B is qatIsSolid: false: Google's own official w4a16 QAT release lineup skips this model, and Google's guidance for production serving of 26B A4B points to INT8 instead, at roughly 47% memory savings rather than 4-bit's ~72%.

There is a QAT-trained Q4_0 GGUF for 26B A4B floating around, but independent testing — notably Unsloth's Gemma 4 QAT writeup — found that converting it naively to plain Q4_0 measurably hurts accuracy, and recommends a dynamic / mixed-precision quant instead (Unsloth's UD-Q4_K_XL tag, for example), which keeps a few sensitive layers at higher bit-width while quantizing the rest. Same rough memory footprint, meaningfully better output — the same logic as choosing QAT over plain Q4_0 for the dense models, just implemented differently because the architecture demands it.

The practical rule: for the four dense models, grab the official QAT build without a second thought. For 26B A4B, skip the plain Q4_0 GGUF and look specifically for a dynamic/mixed-precision community quant, or run INT8 if you're serving via vLLM.

Decision Path: From Usable Memory to a Quant

Work out your usable memory first — the memory formula guide walks through the exact math (roughly: Mac total − 5 GB, GPU VRAM − 1 GB, CPU RAM ÷ 2). Then:

1. Find the largest model that's "comfortable" at Q4_0/QAT for your usable memory, using the weight table above. Bigger and better wins over smaller-but-lower-quality whenever it genuinely fits — our checker's own recommendation logic favors bigger, more capable models over quantization headroom.
2. If it's a dense model (E2B, E4B, 12B, or 31B) — get the official QAT build. Same memory cost as regular Q4_0, better quality, no downside. Ollama, LM Studio, and Hugging Face all carry the QAT tags (see the QAT guide for exact names).
3. If it's 26B A4B — get a dynamic/mixed-precision quant, not plain Q4_0. Same rough size, meaningfully better output, per Unsloth's testing above.
4. Have real headroom left over (comfortable at Q4_0 with GB to spare)? Consider stepping up to SFP8 for that model instead of moving to a bigger model — SFP8 sits much closer to BF16 quality, and might beat a bigger model at Q4_0 for tasks where precision matters more than raw parameter count. This is a judgment call our checker doesn't make for you: it always recommends the largest comfortable model, not the highest quant of a smaller one.
5. Only if memory is genuinely abundant (comfortable at SFP8 or BF16) does full precision start to make sense — mainly for research, fine-tuning workflows, or serving setups where quantization error needs to be ruled out entirely rather than just minimized.

What We're Not Confident About

In the interest of not overstating precision: we don't have a controlled, apples-to-apples benchmark of QAT vs. BF16 quality gaps specific to Gemma 4 — Google's own language is the general "even higher overall quality compared to standard PTQ baselines," not a published numeric gap per model. Our QAT guide cites a third-party estimate (roughly 1–1.5 percentage points on benchmarks like MMLU versus full BF16) explicitly as an independent report, not an official figure, and that caveat applies here too. Similarly, we haven't independently reproduced Unsloth's specific accuracy numbers for 26B A4B's dynamic quants — we're relaying their published testing, not our own.

Next Steps

• Work out your exact usable memory and verdict: Can You Run Gemma 4? checker
• The full mechanics behind that checker: How much memory does Gemma 4 need? The full formula
• Exact QAT release details, repo names, and Ollama tags: Gemma 4 QAT guide
• Once you've picked a quant, get it running: Run Gemma 4 with Ollama