One model, nine quants

Why this model

We picked a 4B model on purpose. We wanted to keep the full-precision version in memory, because it is the reference for every number below.

A 9B model at bf16 needs about 18 GB. Our Mac has 24 GB, and macOS shares it between all processes. When the model does not fit, macOS moves part of it to disk. Then every token pays for a disk read. Our speed numbers would measure swapping, not the model.

Qwen3.5-4B-Base at bf16 is 8.67 GB. It fits on the GPU with room left over, so we got a clean baseline. That is the whole reason for the choice.

How we ran it

Every number in this post comes from this exact setup. The full comparison below keeps the setup and result together.

What is an importance matrix

Quantization works by replacing each weight with a small number. bf16 uses 16 bits for each weight. A 4-bit quant keeps about 4. The rest is discarded.

Most weights do not matter much. A model has channels that fire strongly on most text, and channels that almost never fire. If you spend the same precision on both, you waste it.

An importance matrix, or imatrix, is a file that lists how strongly each channel fired. You build it by running text through the model once and watching which channels light up. Then the quantizer gives more bits to the channels that matter, and fewer to the ones that do not.

It costs one extra pass over some calibration text at quantization time. For this model it took about three minutes. Everything else about the file stays the same.

What the importance matrix did

We built three pairs. The same quant, twice. The only difference is the importance matrix.

Download this comparison as CSV ↓

The model got closer to bf16. Every pair improved, and the gap is much bigger than the measurement error.

Perplexity did not move at all. The changes sit between 0.6 and 1.4 standard errors from zero. In plain terms, that is the same as nothing.

We did not pick the metric that favours us. Both numbers come from the same run, on the same 25,500 tokens. Perplexity had every chance to show the gain.

Why the two disagree

Perplexity is one score. It looks mostly at the most likely token at each position. It does not care much about the rest.

KL divergence compares the whole probability distribution at every position. All 248,320 possible tokens, every time.

An importance matrix makes the model more accurate across that whole distribution, not just at the top. So KL divergence sees it and perplexity does not. When we sort by perplexity alone, we miss it completely.

Quality against size

Here are all eight quants on one chart. The dashed line is the Pareto frontier. That is the set of points where you cannot get a smaller file without also getting a worse one.

That is the practical result. If you want a 4-bit file, do not use the plain build. The imatrix version is the same size, or smaller, and it is closer to the original. We see no reason to skip it, and we will not skip it again.

Q5_K_M plain is the only exception. Its imatrix version came out 256 bytes larger. At that point the two files are the same file.

The two 4-bit options are a trade-off

IQ4_XS is 205 MB smaller than Q4_K_M and 32% further from bf16. Neither one wins. Choose IQ4_XS if size is the problem. Choose Q4_K_M if you want the model to stay close to the original. That call is yours.

The worst cases hide behind the average

Q4_K_M has an average KL divergence of 0.0221. Its worst 1% of tokens sit at 0.163, and the single worst token is 3.46. That is a 157 times spread between typical and awful.

A quant will fail on one unlucky token, not on the average. That is why we kept the percentiles.

Speed against size

Generation speed tracks file size almost exactly. That is not a coincidence. To write one token, the model reads every weight once. So the time per token is set by how many bytes it has to pull through memory.

The bf16 point is not a fair speed comparison. The chart says so. llama.cpp 0.6.0 does not use an optimized bfloat16 matmul path on Metal, and our numbers match that. bf16 was the slowest file of the nine. Read it as a floor, not as the real speed of the unquantized model.

Quantizing does not make the first token faster

Reading a prompt is arithmetic, not memory traffic. The GPU does the same number of calculations whatever the weights are stored as. So a smaller file does not shorten the wait. Look down the middle column: 941 to 1,018 ms across every quant, in no order that matches file size.

What quantizing does improve is the wait between tokens, because that is the part that reads memory. The 95th percentile gap between tokens falls from 68.5 ms at bf16 to 24.5 ms at IQ4_XS. If your app feels slow, that is the number that moves.

Three things to watch for

The reference file is huge

Qwen3.5 has a 248,320-token vocabulary. To score one quant against bf16, you save the reference output first. That costs two bytes per vocabulary entry per scored token, so one 512-token chunk costs about 127 MB. Run the whole wikitext-2 test split and you get a file of roughly 90 GB.

We capped it at 100 chunks. That is 12.66 GB and 25,500 scored tokens, and it is what bartowski publishes, so our numbers line up with his. Check this before you start, not after.

Never subtract two perplexities

Our first attempt computed the gap by taking the quant's perplexity and the bf16 perplexity and subtracting them. That is the wrong way.

llama.cpp scores both models on the identical chunks, so it can report the difference with a real error bar. That number is much tighter. Subtracting two separate scores made the perplexity differences look about ten times more important than they were. It was enough to turn "nothing changed" into "something changed". Use the Mean PPL(Q)-PPL(base) line in the log.

The draft layer gets no help

Qwen3.5 ships multi-token prediction layers. They sit at blk.32.nextn. They stay asleep unless you ask for speculative decoding, so they never fire during calibration. llama.cpp logs did not find weights for them and quantizes them with no imatrix data at all.

We kept speculative decoding off, so this adds no cost in our setup. If you turn it on, those layers were never calibrated.

The commands

This is the whole pipeline. Four steps and a test. If you want to run it, this is what we ran.

1. Convert the model to bf16 GGUF.

python3 convert_hf_to_gguf.py ~/qwen_quants/models/hf \
  --outtype bf16 \
  --outfile ~/qwen_quants/gguf/qwen3.5-4b-base-bf16.gguf

2. Build the importance matrix. One pass over the calibration text.

llama-imatrix \
  -m gguf/qwen3.5-4b-base-bf16.gguf \
  -f data/calibration_datav6.txt \
  -o gguf/imatrix-v6.dat \
  -ngl 99 --chunks 200

3. Quantize. Same command with or without the imatrix flag.

# plain
llama-quantize gguf/qwen3.5-4b-base-bf16.gguf \
  gguf/qwen3.5-4b-base-Q4_K_M.gguf Q4_K_M

# with imatrix
llama-quantize --imatrix gguf/imatrix-v6.dat \
  gguf/qwen3.5-4b-base-bf16.gguf \
  gguf/qwen3.5-4b-base-Q4_K_M-imat-v6.gguf Q4_K_M

4. Score quality. Save the reference once, then compare every quant to it.

# save bf16 reference logits (writes 12.66 GB)
llama-perplexity -m gguf/qwen3.5-4b-base-bf16.gguf \
  -f data/wikitext-2-raw/wiki.test.raw \
  -ngl 99 -c 512 --chunks 100 \
  --kl-divergence-base results/ref.kld

# score a quant against that reference
llama-perplexity -m gguf/qwen3.5-4b-base-Q4_K_M-imat-v6.gguf \
  -f data/wikitext-2-raw/wiki.test.raw \
  -ngl 99 -c 512 --chunks 100 \
  --kl-divergence-base results/ref.kld --kl-divergence

5. Speed.

llama-bench -m gguf/qwen3.5-4b-base-Q4_K_M.gguf \
  -p 512,2048 -n 128 -ngl 99 -r 5 -o json

Rules we will not forget

We got some of these wrong at first. These are the lessons worth keeping.

Never quantize a quantized model

This is the most important rule. A quantized model has already lost most of its precision. If you run the quantizer again on that file, it reads the rounded values as the originals and discards more. You get a second round of rounding error. The model gets worse faster than the bit count suggests.

llama.cpp stops you, but only halfway. Point it at an already-quantized file and ask for a different type and it refuses: requantizing from type q4_K is disabled. Pass --allow-requantize and it will do exactly what you asked, and you get a second round of rounding on top of the first.

The other case is quieter. Ask for the type the file already has and it does nothing at all, hands you back a file the same size, and never mentions it. We tried it. No warning.

Always start from the bf16 or f16 file. That file is the only real copy of the weights.

Keep the unquantized file until the end

You need it twice. Once to quantize from, once as the reference for KLD and perplexity. If you delete it to save disk you cannot score anything, and you end up comparing one quant to another, which tells you almost nothing.

Count the reference file size before you start

A model with a 248,320-token vocabulary writes about 127 MB of reference data per 512-token chunk. Easy to miss, expensive to find halfway through.

Use the paired difference, never your own subtraction

llama.cpp scores the quant and the reference on the same chunks, so it can give you a difference with a real error bar. Subtracting two separate numbers threw that pairing away and made our perplexity results look ten times more important than they were.

Match the settings on both sides

If you save reference logits at one context length and score a quant at another, the comparison means nothing. Same chunks, same context, same batch size, both runs.

Change one thing at a time

Our three quant pairs differ by exactly one thing: the importance matrix. Same weights, same calibration, same eval, same machine, same order. If you change the calibration set and the bit depth together, you learn nothing about either.

Report the worst case, not only the average

A mean hides the tokens where the quant actually broke. Q4_K_M has a mean KL of 0.0221 and a worst token of 3.46. Both numbers are true. Only one of them tells you when the model will fail.

Name the hardware in the result

Quantization speed is a property of the machine as much as of the file. The same Q4_K_M runs at a different speed on a different chip, sometimes by a lot. A tokens-per-second number with no hardware attached is not a measurement.

How to use this comparison

These measurements compare nine quantized versions of one 4B model against that same model in bf16. They are useful for choosing a file and runtime on a similar machine; they are not a ranking of models for your task. Build a small, representative evaluation set before deciding what quality is enough.