Qwen3.5-4B-Base quantization
I took one 4.33B base model, made eight quantizations of it, and scored every one against the same bf16 reference on the same held-out text.
Why this model
I wanted to be able to keep the full-precision model in memory. A 9B model at bf16 needs about 18 GB, and my Mac has 24 GB shared between everything. That leaves no room for a context buffer or anything else, and swapping to disk on every token is not a fair test.
Qwen3.5-4B-Base at bf16 is 8.67 GB. It fits on the GPU with room to spare, so I could run a real bf16 baseline instead of a fake one. That baseline is the reference for every perplexity and KL number on this page. A model that only fits on disk cannot give me that.
How I ran it
- Model
Qwen/Qwen3.5-4B-Base— 4.33B parameters in the text model. The Hugging Face repo lists 4.66B in total, but 334M of that is a vision encoder I did not quantize.- Reference
- bf16 GGUF, 8.67 GB, 16.02 bits per weight
- Hardware
- MacBook Pro, M5 Pro, 10-core, 24 GB, on AC power, Low Power Mode off
- Runtime
- llama.cpp 0.6.0 (Homebrew), Metal backend
- Test text
- wikitext-2-raw test split, 100 chunks at 512 context = 25,500 scored tokens
- Calibration
- bartowski calibration v6, 200 chunks at 512 context
- Speed test
llama-bench, 5 repeats, plus a streaming test at 128, 1k and 4k prompt tokens- Off
- Speculative decoding off everywhere
The results
| Quant | GB | bpw | PPL | ΔPPL | KLD | KLD p99 | Top-1% | tok/s | TTFT 1k | RSS MB |
|---|---|---|---|---|---|---|---|---|---|---|
| bf16 reference | 8.67 | 16.02 | 8.2897 | — | — | — | — | 14.7 | 991 | 8928 |
| Q8_0 | 4.61 | 8.53 | 8.2874 | +0.003 | 0.00020 | 0.001 | 99.20 | 26.0 | 941 | 5082 |
| Q6_K | 3.56 | 6.59 | 8.2873 | +0.003 | 0.00216 | 0.014 | 97.51 | 32.7 | 993 | 4111 |
| Q5_K_M +i | 3.16 | 5.85 | 8.3271 | +0.043 | 0.00426 | 0.028 | 96.56 | 35.8 | 1018 | 3738 |
| Q5_K_M | 3.16 | 5.85 | 8.3170 | +0.033 | 0.00618 | 0.043 | 96.07 | 35.7 | 1033 | 3739 |
| Q4_K_M +i | 2.78 | 5.15 | 8.4085 | +0.124 | 0.01552 | 0.110 | 93.73 | 39.8 | 994 | 3391 |
| Q4_K_M | 2.78 | 5.15 | 8.3986 | +0.114 | 0.02211 | 0.163 | 92.86 | 38.7 | 1027 | 3392 |
| IQ4_XS +i | 2.58 | 4.77 | 8.4426 | +0.158 | 0.02045 | 0.140 | 92.76 | 41.3 | 968 | 3204 |
| IQ4_XS | 2.59 | 4.80 | 8.4671 | +0.183 | 0.02473 | 0.182 | 92.16 | 42.5 | 974 | 3217 |
bf16 scores 8.2897 for perplexity. ΔPPL is the paired difference that
llama.cpp reports with its own error bar, not a subtraction of two separate
numbers. tok/s is llama-bench tg128 over 5 repeats. TTFT is the
median of 5 streaming requests at a 1024-token prompt. Memory is the
llama-server process, so it is a low figure.
+i means I built that file with an importance matrix.
What I take from this
- I use an importance matrix every time now. Three minutes to build, and every 4-bit and 5-bit file came out closer to bf16 with no size worth measuring.
- Perplexity alone would not have shown me this change.
- Quantizing gives you about 2.9× more tokens per second, from 14.7 to 42.5. It gives you almost nothing on time to first token.
- IQ4_XS with an importance matrix is the smallest file at 2.58 GB. Q4_K_M with one is 205 MB bigger but 32% closer to the original. Pick one based on what you need.
The charts for this set, the full method, and a plain-English list of every column name, are in the write-up.