Quantization Formats
The format decision matrix — GGUF for universal local, AWQ for NVIDIA serving, MLX for Mac, EXL2 for max quality at low bitrate, FP8 for VRAM-rich high-quality, MXFP4/NVFP4 for the 2025 Blackwell frontier. The quality/size trade-off (Q4 is the sweet spot, ~75% smaller than FP16 with minimal loss). The three conversion workflows. Unsloth Dynamic 2.0's per-layer sensitivity-aware quantization. And the load-bearing position: Layer 4, downstream of training — compress what you built, don't change what it learned.
The format is determined by the inference engine, not chosen in the abstract. GGUF for universal local (Ollama/llama.cpp/LM Studio — CPU, mixed offload, Mac Metal). AWQ (AWQ-Marlin) for NVIDIA GPU production serving in vLLM/TGI. MLX for Apple Silicon native. EXL2 for max quality at low bitrate. FP8 for VRAM-rich high-quality on Hopper/Blackwell. MXFP4/NVFP4 for the 2025 Blackwell-native 4-bit frontier. Pick the runtime first, then the format that runtime consumes.
Q4 (~4 bits/parameter) is the general sweet spot — roughly 75% size reduction vs FP16 with minimal perceptible quality loss. Q2/Q3 degrade noticeably; Q5/Q8 approach lossless. GGUF's Q4_K_M, AWQ's 4-bit, and MLX's 4-bit group quant are all instances of this same sweet spot. Start at Q4; only go lower with eyes open and a benchmark.
AWQ beats GPTQ at 4-bit because it is activation-aware — a calibration pass finds the ~1% of salient weights that matter most for output quality and keeps those high-precision. AWQ-Marlin is the default recommendation for new NVIDIA server deployments; GPTQ is mature, ubiquitous, and defensible where it is what your tooling supports.
Unsloth Dynamic 2.0 beats uniform quants on quality at the same file size by measuring each layer's sensitivity and allocating bitrates accordingly — sensitive layers stay high (5/6/8-bit), tolerant layers drop low (3-bit). It is the same insight as EXL2's variable-rate quantization, packaged for the standard conversion workflow. Strictly preferable when available.
Quantization is Layer 4, downstream of Layer 3 (training). It is a transform on the final trained weights, applied after training. At sensible bitrates (Q4+) it preserves the model's learned behavior — the steering — and changes only compactness and speed. It is cheap and repeatable: one source checkpoint exports to GGUF + AWQ + MLX with no retraining. (Distinct from QLoRA's FT08 training-time quant — 'quantize to train' vs 'quantize to serve'.)
MXFP4 and NVFP4 are both E2M1 4-bit (2025 Blackwell-native) but differ in block size: MXFP4 uses 32 elements per shared scaling block; NVFP4 uses 16. NVFP4's finer-grained blocks give better accuracy at the same 4-bit width. Both support training and inference at frontier scale.