Gemma 3 is Google DeepMind's third open-weight Gemma release, and it ships in four dense sizes:1B, 4B, 12B and 27B parameters. Google positions the whole range as lightweight open models designed to run on consumer-grade hardware — phones, laptops and high-end GPUs. The report is unusually specific about where the parameters land: the 4B, 12B and 27B models each carry a shared 417M-parameter vision encoder, while the 1B has none. Net of embeddings, the non-embedding counts are 698M (1B), 3,209M (4B), 10,759M (12B) and 25,600M (27B). All variants use Gemini 2.0's SentencePiece tokenizer — split digits, preserved whitespace, byte-level encoding — with a vocabulary of about 262k entries.
Vision, bolted on cheaply
The vision tower is a 400M-parameter SigLIP encoder, a Vision Transformer trained with a variation of the CLIP loss, fed square images resized to 896 ×896. It is shared across the 4B, 12B and 27B models andkept frozen during training— only the language model learns. To control inference cost, the encoder's output is condensed into a fixed256 vision tokensper image (the 896-resolution encoder applies 4×4 average pooling to reach that size), and the language model treats images as that sequence of soft tokens.
Because a rigid 896×896 input mangles non-square or high-resolution images — text becomes unreadable, small objects vanish — Google addsPan & Scan: an inference-time-only, adaptive windowing algorithm that tiles an image into non-overlapping equal-size crops, each resized to 896 ×896. It runs only when needed and can be disabled for faster inference. The payoff is concentrated on text-in-image tasks: on the 27B instruction-tuned model, InfoVQA climbs from 59.4 to 76.4 and DocVQA from 85.6 to 90.4 with Pan & Scan enabled.
The 128K trick: five local layers per global layer
Long context breaks the KV cache — the stored keys and values that attention re-reads — which tends to explode as sequences grow. Gemma 3's fix is architectural, not just a bigger window. It interleaves local sliding-window self-attention with global self-attention in a5:1 pattern— five local layers for every one global layer — starting with a local layer as the model's first layer. The local layers get a sliding-window span of only1024 tokens, so only the sparse global layers attend across the full context.
The memory math is the whole point. In the report's ablation, a global-only transformer carries a KV-cache overhead of60%at a 32k pre-filled context; with a 1:3 local-to-global ratio and a 1024-token sliding window, that drops tounder 15%. Gemma 2 had used a 1:1 ratio with a 4096 window. And the quality cost is small: Google reports minimal perplexity impact even at a 7-to-1 ratio, and finds the sliding window can be reduced significantly without impacting perplexity.
Two more changes make long context hold. The RoPE base frequency on global layers jumps from 10k to 1M (local layers stay at 10k), and rather than train on 128K sequences from scratch, Google pre-trains at 32K and rescales RoPE at the end of pre-training with a scaling factor of 8. Every size supports 128K context except the 1B, capped at 32K.
Training: distillation, then a new post-training recipe
Every Gemma 3 model is trained with knowledge distillation, pre-training and instruction-tuning alike. The pre-training token budget scales with size: 14T tokens for the 27B, 12T for the 12B, 4T for the 4B and 2T for the 1B, including more multilingual data. The distillation mechanism is concrete: the student samples256 logits per token, weighted by teacher probabilities, and learns the teacher's distribution over just those samples via cross-entropy, with non-sampled logits zeroed out and renormalized.
Post-training is where Google says the biggest gains land. The recipe pairs an improved version of knowledge distillation from a large instruction-tuned teacher with a reinforcement-learning phase built on improved versions of BOND, WARM and WARP. Reward functions cover helpfulness, math, coding, reasoning, instruction-following and multilingual ability while penalizing harmfulness, drawing on weight-averaged reward models trained on human feedback, code-execution feedback, and ground-truth rewards for math. Google also filters personal information, unsafe outputs and duplicates, and deliberately includes data that encourages attribution, hedging and refusals to cut hallucinations.
The quality claims
- The report's headline: Gemma3-4B-IT is competitive with Gemma2-27B-IT, and Gemma3-27B-IT is comparable to Gemini-1.5-Pro across benchmarks.
- On Chatbot Arena, Gemma-3-27B-IT scoresElo 1338, placing it in the top 10 — above far larger open models such as DeepSeek-V3 (1318) and Qwen2.5-70B (1257), and well above Gemma-2-27B-it (1220). The score ignores vision, which those rivals lack.
- Pointwise, 27B-IT hits 89.0 on MATH (Gemini-1.5-Pro: 86.5) and 64.9 on MMMU (65.9) — though it trails on MMLU-Pro, 67.5 versus 75.8.
What it
The trade-off is the local/global interleaving. You give up some raw capacity — most layers can only ever see the last 1024 tokens — in exchange for a KV cache that grows slowly enough that long sessions fit on modest hardware. The numbers are dramatic: the 27B occupies 54.0 GB of bf16 weights, rising to 72.7 GB with a 32k KV cache; in int4 it drops to 14.1 GB and 32.8 GB respectively. The 4B sits at 8.0 GB bf16 (12.7 GB with KV) and 2.6 GB int4 (7.3 GB with KV) — genuinely laptop-scale. And multimodality is almost free to add, since the frozen vision tower and its 256-token compression ride along without a second inference model.
The catch, in plain terms: if your workload is long-context document or image-text work on a single consumer GPU, Gemma 3's sliding-window-heavy design is built for exactly that. But because most layers hold a hard 1024-token view, fine-grained global reasoning across a long document depends on the one-in-six global layers plus the RoPE-rescale context extension — which, as the report itself concedes, generalizes to 128K and then rapidly degrades as you continue to scale.