Quantization Formats

Module FT19 · Course 3 — LLM Fine-Tuning Masterclass

75 minutes · Layer 4 (Export). Matrix · Trade-off · Workflows · Dynamic 2.0 · After-training

The format is determined by where the model will run. Pick the runtime, then the format. Default to four-bit.

Pillar 6 — Deploy · First module

Where it sits — Layer 4, the Export

You finished FT11. You have a fine-tuned checkpoint in FP16/BF16 — sixteen bits per parameter. That is far too large to deploy cheaply.

Layer 4 is downstream of training. You quantize after you fine-tune. It compresses what training produced — it does not change learned behavior (at sensible bitrates), only compactness and speed.

Not the same as QLoRA (FT08): QLoRA quantizes the base during training to fit VRAM (the adapter stays high-precision). That is quantize-to-train. FT19 is quantize-to-serve.

The format decision matrix

The format is determined by the inference engine. Pick the runtime, then the format.

Deployment target	Format
Local / CPU / mixed (Ollama, llama.cpp)	GGUF (Q4_K_M)
NVIDIA GPU prod serving (vLLM, TGI)	AWQ (AWQ-Marlin)
Max quality @ low bitrate (ExLlamaV2)	EXL2 (variable-rate)
Apple Silicon native	MLX (4-bit group)
VRAM-rich Hopper/Blackwell (high quality)	FP8
Blackwell-native 4-bit (2025 frontier)	MXFP4 / NVFP4

GPTQ is the mature 4-bit alternative — slightly worse than AWQ at 4-bit, ubiquitous.

GGUF — the universal local format

Consumed by llama.cpp and everything on it: Ollama, LM Studio, llamafile, KoboldCpp.

One self-contained .gguf — weights, tokenizer, config, chat template.

Runs on CPU, mixed CPU/GPU offload, Mac Metal.

The sweet spot

Q4_K_M

~4 bits/param · ~75% smaller than FP16 · minimal loss.

7B: 14 GB → ~4.4 GB

K-quant ladder: Q2_K → Q3_K_M → Q4_K_M → Q5_K_M → Q8_0

Maximum compatibility. No other format matches its portability. Reach for Q4_K_M first; only go lower if desperate.

The NVIDIA serving formats — AWQ & GPTQ

AWQ (preferred)

Activation-aware Weight Quantization. Calibration finds the ~1% of salient weights; those stay high-precision.

AWQ-Marlin = optimized vLLM kernel. Best speed/quality for server inference.

Lower perplexity than GPTQ at the same 4-bit size.

GPTQ (mature, ubiquitous)

One-shot post-training 4-bit. Slightly worse quality than AWQ at 4-bit. Showing its age.

Nearly every HF model has a GPTQ variant. Fine where it is what tooling supports.

GPTQ-Marlin kernel also exists for vLLM.

Decision: prefer AWQ-Marlin for new NVIDIA server deployments. GPTQ is defensible where it is what you have.

EXL2 · MLX · FP8 · the 2025 frontier

Format	Superpower	The catch
EXL2	Variable-rate per-layer quant → best perplexity at small sizes	ExLlamaV2-only (vLLM experimental)
MLX	Apple Silicon-native; exploits unified memory → best Mac perf	Mac only. 4-bit group quant default.
FP8	2x size/speed over FP16, negligible loss, Hopper/Blackwell HW accel	Needs VRAM headroom
MXFP4 / NVFP4	2025 Blackwell-native 4-bit (E2M1). Block: MXFP4=32, NVFP4=16 (finer → better accuracy)	Newest; Blackwell hardware

All four answer "what is the right format?" with "what is your hardware and what are you optimizing for?"

The quality/size trade-off

One curve. Q4 is the sweet spot.

Quant	Size vs FP16	Quality
Q2	~88% smaller	noticeable degradation — desperate only
Q3	~84% smaller	measurable degradation
Q4	~75% smaller	SWEET SPOT — minimal loss
Q5	~70% smaller	near-lossless
Q8	~58% smaller	effectively lossless

Start at Q4. Always. Only go lower with eyes open and a benchmark; only go higher with VRAM to spare.

The conversion workflows

One source checkpoint → three export artifacts.

# GGUF — llama.cpp
python convert_hf_to_gguf.py ./model --outfile model-f16.gguf
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
#  (or Unsloth: model.save_pretrained_gguf("out", quantization_method="q4_k_m"))

# AWQ — AutoAWQ
model = AutoAWQForCausalLM.from_pretrained("./model")
model.quantize(tokenizer, quant_config={"w_bit":4,"q_group_size":128,"version":"GEMM"})
model.save_quantized("./model-awq")

# MLX — mlx-lm
python -m mlx_lm.convert --hf-path ./model --quantize --q-bits 4 --mlx-path ./model-mlx

Quantization is cheap and repeatable — minutes, not training-hours. Convert the same checkpoint to every format you need. No retraining per target.

Unsloth Dynamic 2.0 — per-layer beats uniform

Uniform quant (standard Q4_K_M, AWQ 4-bit) gives every layer the same bitrate. But not all layers are equally sensitive.

Layer type	Uniform Q4	Dynamic 2.0
Attention projections (sensitive)	Q4 — quality lost	Q6 — kept high
Tolerant MLP block	Q4 — bits wasted	Q3 — compressed
Average	Q4	Q4 (same size, HIGHER quality)

Dynamic 2.0 beats uniform quants on quality at the same file size. Same insight as EXL2's variable-rate quantization, packaged for the standard conversion workflow.

Why AFTER training

Three reasons quantization is Layer 4, not part of Layer 3:

It compresses what training produced. A transform on the final weights, not a step in the optimization.
It does not change learned behavior (at Q4+). The steering — format, preferences, compliance — is preserved. It changes compactness/speed, not what the model learned.
It is cheap and repeatable. Minutes, not hours. Quantize one checkpoint to GGUF + AWQ + MLX without retraining.

The modularity of the stack is the point. Train once, export many. Keep Layer 4 downstream of Layer 3.

Anti-patterns

Q2 for size, surprised by quality loss. Q2 is desperate-only, not a default. Start at Q4; only go lower with eyes open and a benchmark.

Mixing formats carelessly. AWQ into Ollama, GPTQ where vLLM expects AWQ-Marlin. The format must match the runtime. Pick the target first (FT20), then convert.

Not benchmarking the quant against the original. Quantization is lossy in principle. Run the same eval on FP16 AND the quant. Ship only if the delta is within tolerance.

Treating quantization as part of training. Train (or QLoRA), merge, then quantize. Keep the layers separate.

What you can now do

State the format decision matrix — given a target, name the format and defend it.
Explain the quality/size trade-off and why Q4 is the sweet spot.
Run the three conversion workflows — GGUF, AWQ, MLX — from one checkpoint.
Explain Unsloth Dynamic 2.0 and why per-layer beats uniform.
Place quantization correctly: Layer 4, downstream of training — compressing what you built without changing what it learned.

Next: FT20 — Serving Stacks