# Diagrams — Module FT19: Quantization Formats

**Module**: FT19 — Quantization Formats
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — The Format Decision Matrix (format -> use case)

**Type**: Deployment-target decision map
**Purpose**: The single diagram that anchors the module. The format is determined by the inference engine / deployment target, not chosen in the abstract. Read the left column first, then the format, then the rationale.
**Reading the diagram**: Each row maps a deployment target to its format. The right column is the load-bearing "why."

```mermaid
flowchart LR
  subgraph Target["DEPLOYMENT TARGET"]
    T1["Local / CPU / mixed\n(Ollama, llama.cpp)"]
    T2["NVIDIA GPU prod serving\n(vLLM, TGI)"]
    T3["Max quality at low bitrate\n(ExLlamaV2)"]
    T4["Apple Silicon\n(macs)"]
    T5["VRAM-rich Hopper/Blackwell\n(high quality)"]
    T6["Blackwell-native 4-bit\n(2025 frontier)"]
  end
  subgraph Format["FORMAT"]
    F1["GGUF\n(Q4_K_M sweet spot)"]
    F2["AWQ\n(AWQ-Marlin)"]
    F3["EXL2\n(variable-rate)"]
    F4["MLX\n(4-bit group)"]
    F5["FP8"]
    F6["MXFP4 / NVFP4"]
  end

  T1 --> F1
  T2 --> F2
  T3 --> F3
  T4 --> F4
  T5 --> F5
  T6 --> F6

  style Target fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Format fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style T1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style T2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style T3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style T4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style T5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style T6 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style F1 fill:#08080c,stroke:#5eead4,color:#5eead4
  style F2 fill:#08080c,stroke:#5eead4,color:#5eead4
  style F3 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style F4 fill:#08080c,stroke:#5eead4,color:#5eead4
  style F5 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style F6 fill:#08080c,stroke:rgba(94,234,212,0.5),color:#e4e4e8
```

---

## Diagram 2 — The Quality/Size Trade-off Curve

**Type**: Trade-off curve (discretized)
**Purpose**: Make the non-linear shape of the trade-off concrete. Q4 is the sweet spot. Below Q4 the quality cost gets steep; above Q4 the size premium gets expensive for diminishing quality returns.
**Reading the diagram**: Left to right is more bits/param (higher quality, larger size). The Q4 column is the recommended default; Q2/Q3 are the danger zone; Q8 is near-lossless.

```mermaid
flowchart LR
  Q2["Q2\n~88% smaller\nNOTICEABLE\ndegradation"] --> Q3["Q3\n~84% smaller\nmeasurable\ndegradation"]
  Q3 --> Q4["Q4  ·  SWEET SPOT\n~75% smaller\nminimal loss"]
  Q4 --> Q5["Q5\n~70% smaller\nnear-lossless"]
  Q5 --> Q8["Q8\n~58% smaller\neffectively\nlossless"]
  Q8 --> FP["FP16\nbaseline\nreference"]

  Q2 -.->|"desperate only"| Danger["DANGER ZONE\nbenchmark before shipping"]
  Q4 -.->|"start here"| Sweet["DEFAULT\nreach for Q4_K_M first"]
  Q8 -.->|"VRAM-rich"| Safe["LOSSLESS\nuse when certainty matters"]

  style Q2 fill:#14141f,stroke:#f08080,color:#f08080
  style Q3 fill:#14141f,stroke:#f0a868,color:#f0a868
  style Q4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Q5 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Q8 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style FP fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Danger fill:#08080c,stroke:#f08080,stroke-dasharray: 4 2,color:#f08080
  style Sweet fill:#08080c,stroke:#5eead4,stroke-dasharray: 4 2,color:#5eead4
  style Safe fill:#08080c,stroke:rgba(94,234,212,0.5),stroke-dasharray: 4 2,color:#5eead4
```

---

## Diagram 3 — The Conversion Workflows

**Type**: Pipeline / conversion flow
**Purpose**: Show the three primary conversion paths from one trained checkpoint to three deployable formats. One source checkpoint -> three export artifacts. The modularity is the point.
**Reading the diagram**: Top is the single trained FP16 checkpoint (the output of FT11). Three branches convert it to the three formats via their respective tools. Each artifact targets a different runtime.

```mermaid
flowchart TD
  CKPT["YOUR FINE-TUNED CHECKPOINT\n(FT11 output · FP16/BF16)"]

  CKPT --> GGUF["GGUF path\nllama.cpp convert_hf_to_gguf.py\nOR Unsloth save_pretrained_gguf"]
  CKPT --> AWQ["AWQ path\nAutoAWQ.quantize\ncalibration -> 4-bit"]
  CKPT --> MLX["MLX path\nmlx_lm.convert --quantize --q-bits 4"]

  GGUF --> A1["model-Q4_K_M.gguf\n-> Ollama / llama.cpp / LM Studio"]
  AWQ --> A2["model-awq/\n-> vLLM / TGI (Marlin kernel)"]
  MLX --> A3["model-mlx/\n-> mlx-lm / LM Studio (Mac)"]

  Note["ONE source checkpoint\nTHREE export artifacts\nQuantize AFTER training (Layer 4)"]
  CKPT -.-> Note

  style CKPT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style GGUF fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style AWQ fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style MLX fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style A1 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style A2 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style A3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```

---

## Diagram 4 — Unsloth Dynamic 2.0 vs Uniform Quantization

**Type**: Comparison (uniform vs per-layer)
**Purpose**: Show why uniform quantization leaves quality on the table and how per-layer sensitivity-aware quantization (Unsloth Dynamic 2.0 / EXL2's approach) beats it at the same total size.
**Reading the diagram**: Two columns of the same model. Uniform gives every layer Q4. Dynamic measures each layer's sensitivity and varies the bitrate — sensitive layers stay high, tolerant layers drop low — hitting the same average with higher quality.

```mermaid
flowchart LR
  subgraph Uniform["UNIFORM Q4 (one bitrate for all)"]
    U1["Layer 1  ·  Q4"]
    U2["Layer 2  ·  Q4  (sensitive! quality lost)"]
    U3["Layer 3  ·  Q4"]
    U4["Layer 4  ·  Q4  (could be Q3, wasted bits)"]
    U5["Avg = Q4   ·   quality = baseline"]
  end
  subgraph Dynamic["DYNAMIC 2.0 (sensitivity-aware)"]
    D1["Layer 1  ·  Q4"]
    D2["Layer 2  ·  Q6  (sensitive -> keep high)"]
    D3["Layer 3  ·  Q3"]
    D4["Layer 4  ·  Q3  (tolerant -> compress)"]
    D5["Avg = Q4   ·   quality HIGHER at same size"]
  end

  Uniform -.->|"measure each layer's\nsensitivity, re-allocate"| Dynamic

  style Uniform fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Dynamic fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style U1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style U2 fill:#08080c,stroke:#f08080,color:#f08080
  style U3 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style U4 fill:#08080c,stroke:#f0a868,color:#f0a868
  style U5 fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style D2 fill:#08080c,stroke:#82e0aa,color:#82e0aa
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style D5 fill:#08080c,stroke:#5eead4,color:#5eead4
```

---

## Diagram 5 — Quantize-After-Training: Layer 4 in the Stack

**Type**: Stack position / pipeline
**Purpose**: Place quantization correctly. It is Layer 4 (Export), downstream of Layer 3 (the Steer / training). It compresses what training produced; it does not change learned behavior. Compare with QLoRA's training-time quant (a memory trick, a different concern).
**Reading the diagram**: Bottom to top is the stack. Layer 3 trains the behavior; Layer 4 exports it compressed. The dashed callout contrasts the deployment quant (this module) with QLoRA's training-time quant (FT08).

```mermaid
flowchart TD
  L5["5. THE BOUNDARY  ·  the harness  (Courses 1, 2A)"]
  L4["4. THE EXPORT  ·  QUANTIZE + SERVE\nTHIS MODULE (FT19) + FT20"]
  L3["3. THE STEER  ·  fine-tuning (SFT/DPO/GRPO)  (FT12-FT18)"]
  L2["2. THE ADAPTER  ·  LoRA / DoRA  (FT08-FT11)"]
  L1["1. THE BASE  ·  pretrained weights"]

  L1 --> L2 --> L3 --> L4 --> L5

  L3 -.->|"outputs a merged\nFP16/BF16 checkpoint"| CKPT["trained checkpoint"]
  CKPT -.->|"Layer 4 compresses it\n(GGUF/AWQ/MLX/etc.)\nbehavior PRESERVED"| L4

  Note["QLoRA (FT08) quantizes DURING training\n(memory trick to fit base in VRAM).\nFT19 quantizes AFTER training\n(deployment export).\nDifferent concerns, different layers."]
  L4 -.-> Note

  style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L2 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style L4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style L5 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style CKPT fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text, with `#82e0aa` (ok) and `#f08080` (danger) and `#f0a868` (warn) used semantically for quality/sensitivity and warnings.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with `subgraph`, `LR`/`TD` direction, dashed `-.->` for annotations) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live and inlined into reveal.js as needed.