The model is modular

The full model has 740 million parameters. You rarely need all of them. The model has a 270M text core. A 170M image encoder and a 300M audio encoder join it only when a task asks for them.

Google measured this memory on a Pixel 11 Pro with the int4 builds. The text-only load fits in the memory that one phone app can use.

One warning from the model card: use bfloat16 or float32. Plain float16 breaks the output. This is rare, and it will surprise people.

The context is a shared budget

The context is 8,192 tokens. All media share it. That is room for about 29 images, 58 video frames, or 5.5 minutes of audio. The previous model had a quarter of that.

It is still a budget. A three-hour video does not fit. Split the media into parts, embed each part, and search the parts.

Cut the vector, keep the meaning

The vectors are Matryoshka. A full vector has 768 dimensions. You can cut it to 512, 256, or 128. The model stays usable. A cut vector costs less storage and less math. Google says the 128 setting fits text-only work. For an index of one million documents, the cut from 768 to 256 removes about two thirds of the store.

The scores

Google reports MTEB Code from 68.76 to 78.68. Code search improved a lot. The multilingual text score barely moved: 61.15 to 61.36. The English score went down a little. The vision, video, and audio scores are new, so there is no old number to compare them with.

Now the warning. Google ran every test itself, on the full-precision model. The MTEB project has not accepted the submission. No outside test of this model exists yet. I would treat the scores as a starting point, and test the model on my own data first.

The license

EmbeddingGemma 1 used the Gemma license with a manual gate. You asked Google for access. Google approved. Version 2 is Apache 2.0 with the gate off. A build server can pull the weights without a person in the loop. This is the biggest change from version 1.

Two small details. The repository has no LICENSE file at the root. The license lives in the metadata and a link. The model card still mentions the Gemma Prohibited Use Policy. Neither blocks a commercial use today. Check both again before you build a product on it.

Start it

pip install sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")

# text and code embed straight away
doc = model.encode(["a voice memo about a broken window"])

# the image and audio encoders load only when a
# modality asks for them — no extra setup

The weights sit on Hugging Face and Kaggle under google/embeddinggemma-2. Transformers 5.19 supports the model. Ollama, MLX, ONNX, and llama.cpp builds exist. There is no gated repo and no permission to ask for.

What inference costs

An inference system has several main costs:

• model memory
• compute
• power
• storage
• network transfer
• hardware
• system operation

Model weights are only one part of the cost. During inference, the system also needs memory for the runtime, activations, and KV cache. For a language model, the KV cache grows as the context and generated sequence grow. This means that model size alone does not define total inference memory.

The model must fit in memory

A basic estimate for model weight memory is:

memory = parameters × bytes per parameter

These numbers describe the weights only. The runtime and KV cache require additional memory.

Quantization can reduce weight memory and make local inference possible on smaller hardware. It can also reduce model quality. The effect depends on the model, quantization method, and task.

Hardware changes the cost

Cloud inference gives you access to hardware without requiring you to buy it. You pay for the service instead. With local inference, you own or rent the hardware. The cost then includes:

• hardware
• electricity
• storage
• maintenance
• software

A local system can have a high initial cost. After that, additional requests can have a low incremental cost when the hardware is already available. Cloud inference has less initial hardware cost and can provide flexible capacity. The better choice depends on the workload.

Utilization matters

Hardware cost also depends on utilization. If a GPU processes one request at a time, much of its capacity can remain unused. If many requests use the same GPU, the hardware cost can be shared across more requests.

Batching can increase throughput. It can also increase the waiting time for individual requests. Interactive applications usually care more about latency. Batch workloads usually care more about throughput.

Tokens affect inference

Language-model inference has two main stages. The first stage processes the input context. The second stage generates output tokens.

Longer inputs and outputs require more computation. A large context also increases KV-cache memory. This is why token usage and context length matter when comparing inference systems.

Small models change the calculation

A smaller model normally needs less memory than a larger model at the same precision. It also needs less compute. This can reduce the hardware required for inference and make local deployment possible.

But a smaller model is not automatically cheaper. If it needs more retries or cannot complete the task reliably, the total system cost can increase. The correct comparison is the cost of completing the task.

Small and large models can work together

A production system does not always need one model.

request → small model → large model when required

A small model can handle simple requests. A larger model can handle requests that need more capability. For example, a small model can perform classification, routing, extraction, or simple tool selection. This can reduce the number of requests sent to the larger model. The routing system adds complexity, so it should be measured against the benefit.

When local inference makes sense

Running a model yourself can make sense when:

• traffic is stable
• the model fits the hardware
• hardware utilization is high
• data should remain local
• latency is important
• network access should not be required
• you already have suitable hardware

Cloud inference can be more suitable when traffic is unpredictable, hardware requirements are large, or operating the system yourself is not worth the effort.

The real question

The question is not: “Is local inference cheaper?”

The better question is: “What is the lowest total cost for the required task?”

That includes:

• model quality
• hardware
• memory
• latency
• throughput
• power
• network
• utilization
• operations

A small model can reduce several of these costs. A cloud service can reduce hardware and operations work. A local system can reduce network dependence and recurring inference charges. The best inference system is the one that matches the workload.

What "small" means

A small model has fewer parameters than a large model. For language models, this means models with fewer than 3 billion parameters — this is my opinion, not a hard rule. For vision models, the same bar is roughly 3 billion. For audio models, roughly 1 billion. The shared principle is simple: a model that fits on limited hardware.

The memory math

Two numbers matter, and they are not the same number. The first one everyone quotes. The second one decides if the model runs.

1 — Model memory (weights)

memory = params × bytes per param. A 7B model at FP16 (2 bytes) is 14 GB. The same model at Q4_K_M (about 4.85 bits per weight, roughly 0.6 bytes) is 4.3 GB. A good mental model — it sizes the download.

2 — Actual inference memory

weights + KV cache + activations + runtime overhead. The KV cache grows with context length and with concurrent users. Overhead adds ~10% plus ~0.5 GB. Quantization shrinks the first term only.

The calculator does the real math:

The point stands either way. A 1.7B model needs about 1 GB at 4-bit. A 70B model needs a rack. Small wins the math.

Cost is lower

Small models use less compute power. From the API prices I compared, a 7-billion-parameter model costs around 10 to 30 times less per token than a 70- to 175-billion-parameter one. At high volume that difference changes the business case, and companies run the same workload for a fraction of the cost.

Latency is lower

Small models give faster answers. They reduce the time to the first token. Some systems need a response under 100 milliseconds. A small model running locally gets close, and there is no network round-trip to pay for. In the cloud a small model also serves faster than a big one: less compute per request, shorter waits.

Deployment is easier

Small models fit on edge devices. They run on phones, cars, factory sensors, and local servers. Local is the best home for them: they keep working without a connection, and data stays on the device, which improves privacy and reliability. When a task does need the cloud, a small model still serves it cheaper and faster than a large one.

Specialization is stronger

Most production tasks are narrow. Examples include:

• classification of text or images
• extraction of data
• detection of objects
• recognition of speech commands
• tool use in agents

A small model that is trained on high-quality data for one task often performs better than a large general model. The small model focuses its capacity on the needed patterns.

When large models are still needed

Large models handle open-ended questions. They perform complex multi-step reasoning across many domains. They support creative tasks. The good production pattern is a small model for the narrow work and a large model behind it for the cases the small model cannot handle — the small one routes up when it is not confident.

Where small models are not suitable

Sometimes a small model is the wrong tool. Here are the main disadvantages:

Limited general knowledge

Small models hold less information than large models. They know fewer facts and cover fewer domains. They struggle more with questions outside their training focus.

Weaker complex reasoning

Small models perform worse on long multi-step problems. They handle open-ended or creative tasks less well. Large models still win when the task needs broad thinking.

Need for fine-tuning

A small base model often underperforms on a specific task. You must fine-tune it on high-quality domain data. This adds extra work and requires good training data.

Less robustness

Small models can fail more easily on unusual or new inputs. They may give weaker results when the data distribution changes. Large models are usually more stable in these cases.

Accuracy trade-offs with compression

Quantization to 8-bit or 4-bit reduces memory use. It can also reduce accuracy by a small amount. You must test the final quantized model carefully.

Not ideal for every workload

If the task needs strong zero-shot ability or very broad knowledge, a small model is not enough. You may still need a large model, or a mix of both.

The summary: small models win on cost, speed, memory, and specialized tasks. They lose on general knowledge, complex reasoning, and out-of-domain robustness. The memory math stays the quiet reason: GB = params × bytes per param. Choose the model size that matches the real requirements of your system.

Why this model

I picked a 4B model on purpose. I wanted to be able to keep the full-precision version in memory, because that version is my reference for every number below.

A 9B model at bf16 needs about 18 GB. My Mac has 24 GB, and macOS shares that between everything. When the model does not fit, macOS moves part of it to disk. Then every token pays for a disk read, and my speed numbers measure swapping rather than the model.

Qwen3.5-4B-Base at bf16 is 8.67 GB. It fits on the GPU with room left over, so I got a clean baseline. That is the whole reason for the choice.

How I ran it

Every number in this post comes from this exact setup. The results table has all the columns.

What is an importance matrix

Quantization works by replacing each weight with a small number. You have 16 bits for each weight in bf16. A 4-bit quant gives you about 4. You throw away the rest.

Most weights do not matter much. A model has channels that fire strongly on most text, and channels that almost never fire. If you spend the same precision on both, you waste it.

An importance matrix, or imatrix, is a file that lists how strongly each channel fired. You build it by running text through the model once and watching which channels light up. Then the quantizer spends its bits on the channels that matter, and spends less on the ones that never do anything.

It costs one extra pass over some calibration text at quantization time. For this model it took about three minutes. Everything else about the file stays the same.

What the importance matrix did

I built three pairs. The same quant, twice. The only difference is the importance matrix.

The model got closer to bf16. Every pair improved, and the gap is much bigger than the measurement error.

Perplexity did not move at all. The changes sit between 0.6 and 1.4 standard errors from zero. In plain terms, that is the same as nothing.

I am not picking the metric that flatters me. Both numbers come from the same run, on the same 25,500 tokens. Perplexity had every chance to show the gain.

Why the two disagree

Perplexity is one score. It looks mostly at the most likely token at each position. It does not care much about the rest.

KL divergence compares the whole probability distribution at every position. All 248,320 possible tokens, every time.

An importance matrix makes the model more accurate across that whole distribution, not just at the top. So KL divergence sees it and perplexity does not. When I sort by perplexity alone, I miss it completely.

Quality against size

Here are all eight quants on one chart. The dashed line is the Pareto frontier. That is the set of points where you cannot get a smaller file without also getting a worse one.

That is the practical result. If you want a 4-bit file, do not use the plain build. The imatrix version is the same size, or smaller, and it is closer to the original. I cannot see a reason to skip it, and I do not plan to again.

Q5_K_M plain is the only exception, and only because its imatrix version came out 256 bytes larger. At that point the two files are the same file.

The two 4-bit options are a trade-off

IQ4_XS is 205 MB smaller than Q4_K_M and 32% further from bf16. Neither one wins. IQ4_XS is the right choice if size is the problem. Q4_K_M is the right choice if you want the model to stay close to the original. That call is yours, not mine.

The worst cases hide behind the average

Q4_K_M has an average KL divergence of 0.0221. Its worst 1% of tokens sit at 0.163, and the single worst token is 3.46. That is a 157 times spread between typical and awful.

If a quant is going to embarrass you, it will be on one unlucky token, not on average. That is why I kept the percentiles.

Speed against size

Generation speed tracks file size almost exactly. That is not a coincidence. To write one token, the model reads every weight once. So the time per token is set by how many bytes it has to pull through memory.

The bf16 point is not a fair speed comparison, and the chart says so on it. llama.cpp 0.6.0 does not use an optimized bfloat16 matmul path on Metal, and my numbers match that: bf16 came out the slowest file of the nine. Read it as a floor, not as the real speed of the unquantized model.

Quantizing does not make the first token faster

Reading a prompt is arithmetic, not memory traffic. The GPU does the same number of calculations whatever the weights are stored as. So a smaller file does not shorten the wait. Look down the middle column: 941 to 1,018 ms across every quant, in no order that matches file size.

What quantizing does improve is the wait between tokens, because that is the part that reads memory. The 95th percentile gap between tokens falls from 68.5 ms at bf16 to 24.5 ms at IQ4_XS. If your app feels slow, that is the number that moves.

Three things that will bite you

The reference file is huge

Qwen3.5 has a 248,320-token vocabulary. To score one quant against bf16, you save the reference output first. That costs two bytes per vocabulary entry per scored token, so one 512-token chunk costs about 127 MB. Run the whole wikitext-2 test split and you get a file of roughly 90 GB.

I capped it at 100 chunks. That is 12.66 GB and 25,500 scored tokens, and it is what bartowski publishes, so my numbers line up with his. Check this before you start, not after.

Never subtract two perplexities

My first attempt computed the gap by taking the quant's perplexity and the bf16 perplexity and subtracting them. That is the wrong way.

llama.cpp scores both models on the identical chunks, so it can report the difference with a real error bar. That number is much tighter. Subtracting two separate scores made the perplexity differences look about ten times more important than they were. It was enough to turn "nothing changed" into "something changed". Use the Mean PPL(Q)-PPL(base) line in the log.

The draft layer gets no help

Qwen3.5 ships multi-token prediction layers. They sit at blk.32.nextn. They stay asleep unless you ask for speculative decoding, so they never fire during calibration. llama.cpp logs did not find weights for them and quantizes them with no imatrix data at all.

I kept speculative decoding off, so this cost me nothing. If you turn it on, those layers were never calibrated.

The commands

This is the whole pipeline. Four steps and a test. If you want to run it, this is what I ran.

1. Convert the model to bf16 GGUF.

python3 convert_hf_to_gguf.py ~/qwen_quants/models/hf \
  --outtype bf16 \
  --outfile ~/qwen_quants/gguf/qwen3.5-4b-base-bf16.gguf

2. Build the importance matrix. One pass over the calibration text.

llama-imatrix \
  -m gguf/qwen3.5-4b-base-bf16.gguf \
  -f data/calibration_datav6.txt \
  -o gguf/imatrix-v6.dat \
  -ngl 99 --chunks 200

3. Quantize. Same command with or without the imatrix flag.

# plain
llama-quantize gguf/qwen3.5-4b-base-bf16.gguf \
  gguf/qwen3.5-4b-base-Q4_K_M.gguf Q4_K_M

# with imatrix
llama-quantize --imatrix gguf/imatrix-v6.dat \
  gguf/qwen3.5-4b-base-bf16.gguf \
  gguf/qwen3.5-4b-base-Q4_K_M-imat-v6.gguf Q4_K_M

4. Score quality. Save the reference once, then compare every quant to it.

# save bf16 reference logits (writes 12.66 GB)
llama-perplexity -m gguf/qwen3.5-4b-base-bf16.gguf \
  -f data/wikitext-2-raw/wiki.test.raw \
  -ngl 99 -c 512 --chunks 100 \
  --kl-divergence-base results/ref.kld

# score a quant against that reference
llama-perplexity -m gguf/qwen3.5-4b-base-Q4_K_M-imat-v6.gguf \
  -f data/wikitext-2-raw/wiki.test.raw \
  -ngl 99 -c 512 --chunks 100 \
  --kl-divergence-base results/ref.kld --kl-divergence

5. Speed.

llama-bench -m gguf/qwen3.5-4b-base-Q4_K_M.gguf \
  -p 512,2048 -n 128 -ngl 99 -r 5 -o json

Rules I will not forget

I got some of these wrong first. They are the things worth keeping.

Never quantize a quantized model

This is the one that bites hardest. A quantized model has already thrown away most of its precision. If you run the quantizer again on that file, it reads the rounded values as if they were the originals and throws away more. You get a second round of rounding error on top of the first, and the model gets worse faster than the bit count suggests.

llama.cpp stops you, but only halfway. Point it at an already-quantized file and ask for a different type and it refuses: requantizing from type q4_K is disabled. Pass --allow-requantize and it will do exactly what you asked, and you get a second round of rounding on top of the first.

The other case is quieter. Ask for the type the file already has and it does nothing at all, hands you back a file the same size, and never mentions it. I tried it. No warning.

Always start from the bf16 or f16 file. That file is the only real copy of the weights.

Keep the unquantized file until the end

You need it twice. Once to quantize from, once as the reference for KLD and perplexity. If you delete it to save disk you cannot score anything, and you end up comparing one quant to another, which tells you almost nothing.

Count the reference file size before you start

A model with a 248,320-token vocabulary writes about 127 MB of reference data per 512-token chunk. Easy to miss, expensive to find halfway through.

Use the paired difference, never your own subtraction

llama.cpp scores the quant and the reference on the same chunks, so it can give you a difference with a real error bar. Subtracting two separate numbers threw that pairing away and made my perplexity results look ten times more important than they were.

Match the settings on both sides

If you save reference logits at one context length and score a quant at another, the comparison means nothing. Same chunks, same context, same batch size, both runs.

Change one thing at a time

My three quant pairs differ by exactly one thing: the importance matrix. Same weights, same calibration, same eval, same machine, same order. If you change the calibration set and the bit depth together, you learn nothing about either.

Report the worst case, not only the average

A mean hides the tokens where the quant actually broke. Q4_K_M has a mean KL of 0.0221 and a worst token of 3.46. Both numbers are true. Only one of them tells you when the model will embarrass you.

Name the hardware in the result

Quantization speed is a property of the machine as much as of the file. The same Q4_K_M runs at a different speed on a different chip, sometimes by a lot. A tokens-per-second number with no hardware attached is not a measurement.

The words I use

Plain meanings, so you do not have to look them up like I did.

The full table

All nine models, every column, is on the benchmarks page. That page also has the same numbers sorted by size, which is easier to read than this post.