Gemma 4 QAT: 31B Now Fits in 24GB VRAM

If you last checked Gemma 4's memory requirements a few weeks ago, it's worth checking again. On June 5, 2026, Google released official quantization-aware training (QAT) checkpoints for four of the five Gemma 4 models, and they change the math on what counts as "enough hardware." The headline: the 31B dense model, which needed 58 GB at full precision, now runs in about 18 GB — squarely inside a 24 GB RTX 3090, RTX 4090, or 24 GB+ Apple Silicon Mac, with quality that holds up far better than a typical 4-bit shrink.

This guide covers what QAT actually is, exactly what Google shipped, the real memory numbers per model, what it unlocks on common hardware, why one model was left out, and how to download and run the checkpoints yourself.

What Is QAT, Actually?

Every quantization method answers the same question — how do you make a model's weights smaller? — but QAT and the regular kind get there in a different order.

Post-training quantization (PTQ), the method behind the plain "Q4_0" numbers in most hardware guides (including our own Gemma 4 hardware guide), is the usual approach: train a model at full precision, then squeeze the finished weights down to 4 bits afterward. It works, but the model never got a chance to adapt to the lower precision — it's compressed after the fact, like shrinking a finished photo.

Quantization-aware training (QAT) builds the compression into the process itself. During training, Google simulates the rounding errors that 4-bit quantization will introduce, so the model learns weights that stay accurate even after they're actually quantized. It's the difference between shrinking a photo after it's taken versus shooting with the final, smaller resolution in mind from the start — you lose a lot less along the way.

The practical result: a QAT-trained model at 4-bit lands much closer to its original BF16 quality than a PTQ model at the same 4-bit size, at effectively the same memory footprint.

The June 5 Release, in Plain Terms

Google shipped QAT checkpoints for E2B, E4B, 12B, and 31B in its official w4a16-ct compressed-tensors format — built for optimized serving in vLLM — plus matching Q4_0 GGUF conversions for local runtimes like llama.cpp, Ollama, and LM Studio. The 26B A4B Mixture-of-Experts model was left out of the w4a16 release (more on exactly why below).

Google describes the approach as yielding "even higher overall quality compared to standard PTQ baselines" at the same bit-width — in other words, this isn't a new quantization level, it's a better way of reaching the 4-bit level you were probably already targeting.

Memory: QAT vs. Regular Q4 vs. BF16

Here's the number that actually matters for hardware planning. QAT and regular Q4 land at nearly identical file sizes — 4 bits is 4 bits either way — so the memory column doesn't change much. What changes is the quality you get at that size.

(Numbers for E4B, 12B, and 31B PTQ come from Google's Gemma 4 documentation; QAT figures are corroborated across Google's own release materials and independent write-ups.)

So no, QAT doesn't shrink the model further than Q4 already does. What it buys you is the confidence to actually run the 4-bit version daily instead of treating it as a "good enough for testing" compromise.

What This Unlocks on Your Hardware

For most people, this release matters less as a memory story and more as a quality-per-GB story:

• 31B at ~18 GB now fits comfortably on a 24 GB card — an RTX 3090 or RTX 4090 — or a 24 GB+ unified-memory Mac, with headroom left for the KV cache. Previously, running 31B well meant a 32 GB+ workstation GPU or a Mac Studio; now it's a strong 24 GB consumer card's job, and quality is noticeably better than the PTQ Q4_0 build most people were already running at that size.
• 12B at ~7 GB was already the 16 GB sweet spot (see our hardware requirements guide); QAT just means that 16 GB machine — including a base 16 GB MacBook Air or budget mini PC — is now running a meaningfully more accurate model at the same memory cost.
• E4B at ~5 GB gets the same treatment on 8 GB machines, where every point of quality matters because there's no room to step up to a bigger model.

The practical takeaway: if your hardware already ran the regular Q4_0 build of any of these three models, swapping to the QAT version costs you nothing in memory and should meaningfully reduce weird phrasing, dropped instructions, and other small-model quirks.

Why 26B A4B Doesn't Have an Official QAT Build

The 26B A4B model uses a Mixture-of-Experts (MoE) architecture, and its expert layers are narrow — each expert has a hidden dimension of only 704. That narrowness is exactly what makes 4-bit quantization risky for this model: there's very little redundancy left to absorb rounding error once you drop to 4 bits, so accuracy degrades more than it does for the dense models. That's the specific reason Google's official w4a16-ct compressed-tensors lineup skips 26B A4B — for production serving, Google's own guidance points toward INT8 quantization for this model instead, which gives roughly 47% memory savings (versus ~72% for the 4-bit models) at safer quality.

That said, "no official w4a16" isn't quite "no QAT option at all." Google also published a QAT-trained, Q4_0 GGUF checkpoint for 26B A4B for local runtimes — but community testing (notably from Unsloth) found that naively converting that checkpoint straight to Q4_0 in llama.cpp measurably hurts accuracy on their benchmark (roughly 70% top-1, versus about 86% with a mixed-precision "dynamic" quant that keeps a few sensitive layers at higher bit-width). If you want to run 26B A4B locally, look for a dynamic/mixed-precision quant (Unsloth's UD-Q4_K_XL tag, for example) rather than the plain Q4_0 GGUF — same rough memory ballpark, meaningfully better output.

How to Actually Download and Run These

Hugging Face. Google's official checkpoints live under the google/gemma-4-qat-q4-0 collection, which includes the w4a16-ct compressed-tensors repos (google/gemma-4-E2B-it-qat-w4a16-ct, -E4B-, -12B-, and -31B-, for vLLM) alongside Q4_0 GGUF versions of all five sizes, including google/gemma-4-26B-A4B-it-qat-q4_0-gguf.

Ollama. Pull any QAT tag directly:

ollama pull gemma4:12b-it-qat
ollama pull gemma4:31b-it-qat
ollama pull gemma4:26b-a4b-it-qat

Full lineup and sizes: e2b-it-qat (4.3 GB), e4b-it-qat (6.1 GB), 12b-it-qat (7.2 GB), 26b-a4b-it-qat (16 GB), and 31b-it-qat (19 GB). See our Ollama guide for the rest of the setup, including image input and the local API.

LM Studio. Search for google/gemma-4-12b-qat (or e2b-qat, e4b-qat, 26b-a4b-qat, 31b-qat) in LM Studio's model browser, or use the "Use Model in LM Studio" button on each Hugging Face model page.

Quality: How Close to BF16, Really?

Google hasn't published a detailed head-to-head benchmark table of QAT versus BF16 for Gemma 4 specifically — its own language is the more general "even higher overall quality compared to standard PTQ baselines." Independent deployment write-ups (for example, Spheron Network's cloud deployment guide) report gaps in the range of roughly 1–1.5 percentage points on benchmarks like MMLU, HumanEval, and GSM8K versus full BF16 — treat that as a third-party estimate rather than an official Google figure, but it lines up with what QAT has delivered for past model generations: a small, usually unnoticeable-in-practice gap, well inside the range where the memory savings are worth it for local use.

FAQ

Is QAT better than GGUF Q4? For the four models Google shipped it for (E2B, E4B, 12B, 31B), yes — QAT and a regular Q4_0 GGUF use essentially the same memory, but the QAT version was trained to be accurate at that precision rather than compressed into it afterward. There's no real reason to pick the older PTQ Q4_0 build over QAT once it's available for your model size.

Can I run 31B QAT on an RTX 4090? Yes. At roughly 18 GB for the weights, 31B QAT fits comfortably inside a 24 GB RTX 4090 with several gigabytes left over for the KV cache and normal desktop use — a configuration that simply couldn't run the 58 GB BF16 version or comfortably fit the older 17.4 GB PTQ Q4_0 with much context room to spare.

What about 26B A4B — should I wait for an official QAT build? Don't wait on Google specifically for a w4a16 version; the expert-dimension issue means one may never ship. If you want 26B A4B locally today, use a dynamic/mixed-precision GGUF quant rather than the plain Q4_0 conversion, or run it at INT8 if you're deploying via vLLM.