# Diagrams — Module FT04: Dataset Formats

**Module**: FT04 — Dataset Formats
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — The Three Canonical Formats (side by side)

**Type**: Comparison
**Purpose**: The single mental model for "what shape does my data arrive in." Three input formats, one normalized internal representation.
**Reading the diagram**: Left to right. Three input shapes on the left all funnel through `.map(to_messages)` into the single `messages` column on the right, which is what every downstream tool consumes.

```mermaid
flowchart LR
  subgraph Inputs["THREE INPUT FORMATS"]
    direction TB
    F1["Raw messages\n[{role, content}]"]
    F2["Instruction / response\n(instruction + output)"]
    F3["ShareGPT\n[{from, value}]"]
  end
  Conv["to_messages()\nnormalize once"]
  Out["messages column\n[{role, content}, ...]\nTHE INTERNAL LINGUA FRANCA"]

  F1 --> Conv
  F2 --> Conv
  F3 --> Conv
  Conv --> Out

  style F1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style F2 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style F3 fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Conv fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray:4 2,color:#5eead4
  style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Diagram 2 — The apply_chat_template Flow

**Type**: Pipeline
**Purpose**: The single right way to turn messages into model-ready text. The anti-pattern (hand-concatenation) is shown as a forbidden branch.
**Reading the diagram**: Top-down. `messages` enters, the tokenizer's own Jinja template renders the bytes, the model sees the exact format it was instruction-tuned against. The hand-concat path is struck through because it produces wrong bytes.

```mermaid
flowchart TD
  Msgs["messages\n[{role, content}, ...]"]
  Tok["tokenizer\n(shipped WITH the model)"]
  Tpl["chat_template.jinja\n(publisher's exact format)"]
  Text["rendered string\nexact bytes the model expects"]
  IDs["input_ids\nsingle-token specials\n(no merging bugs)"]

  Hand["HAND-CONCATENATION\nf'<user>...' + q + ..."]
  HandBugs["token-boundary errors\nspecial tokens as text\nwrong role tokens"]
  Result["SILENT TRAINING FAILURE\nloss drops, model subtly wrong"]

  Msgs --> Tok
  Tok --> Tpl
  Tpl --> Text
  Text --> IDs
  IDs --> Model["MODEL"]

  Msgs -.->|"ANTI-PATTERN"| Hand
  Hand --> HandBugs
  HandBugs --> Result

  style Msgs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Tok fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Tpl fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Text fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style IDs fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Model fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Hand fill:#14141f,stroke:#f08080,stroke-dasharray:4 2,color:#f08080
  style HandBugs fill:#14141f,stroke:#f08080,color:#f08080
  style Result fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#f08080
```

---

## Diagram 3 — SFT Data vs Preference Data

**Type**: Comparison / structure
**Purpose**: Two training objectives, two data shapes. Mixing them is a bug.
**Reading the diagram**: Left = SFT shape (one correct response, cross-entropy on the assistant turn). Right = preference shape (chosen vs rejected sharing the same prompt, contrastive loss). The defining property of preference data is at the bottom.

```mermaid
flowchart LR
  subgraph SFT["SFT DATA — one correct response"]
    direction TB
    SP["messages:\nsystem / user / assistant"]
    SL["loss on the assistant turn only\n(cross-entropy)"]
    ST["-> SFTTrainer (FT12)"]
  end
  subgraph PREF["PREFERENCE DATA — chosen vs rejected"]
    direction TB
    PP["prompt: [{user...}]\nchosen:   [{assistant...good}]\nrejected: [{assistant...bad}]"]
    PL["contrastive loss\n(chosen > rejected)"]
    PT["-> DPOTrainer / KTO / IPO (FT13)"]
  end

  Rule["DEFINING PROPERTY:\nchosen and rejected share the same prompt\nand differ only in the final assistant turn"]

  SFT --> Rule
  PREF --> Rule

  style SP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style SL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0
  style ST fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4
  style PP fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style PL fill:#14141f,stroke:rgba(94,234,212,0.4),color:#9494a0
  style PT fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4
  style Rule fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Diagram 4 — The Inspection Loop

**Type**: Cyclic process
**Purpose**: The five-second check that catches every formatting bug. Do this once per dataset.
**Reading the diagram**: Clockwise from the top. After `apply_chat_template` and tokenize, decode the `input_ids` back to text and read it. If you see any of the four bug signatures, fix and re-inspect. Only when the decoded text is clean do you train.

```mermaid
flowchart TD
  A["messages"]
  B["apply_chat_template\ntokenize=False"]
  C["tokenizer(text)\n-> input_ids"]
  D["tokenizer.decode(input_ids[0])\nTHE INSPECTION"]
  E{"Decoded text clean?"}
  Bugs["BUG SIGNATURES TO SCAN FOR:\n- missing EOS at end\n- specials rendered as text\n- trailing spaces\n- user turns in the loss"]
  Fix["Fix the source format\n(re-run conversion)"]
  Train["TRAIN\nthe data is verified"]

  A --> B
  B --> C
  C --> D
  D --> E
  E -->|"NO — something looks wrong"| Bugs
  Bugs --> Fix
  Fix --> B
  E -->|"YES — bytes match the template"| Train

  style A fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style B fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style C fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style D fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style E fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Bugs fill:#14141f,stroke:#f08080,color:#f08080
  style Fix fill:#14141f,stroke:#f08080,color:#f08080
  style Train fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
```

---

## Diagram 5 — The Bug Taxonomy

**Type**: Matrix / classification
**Purpose**: The checklist of format bugs you will encounter, ranked by how silently they fail.
**Reading the diagram**: Left column = the bug. Middle = the symptom (what you see at inference). Right = the root cause (where to look). Every row is caught by Diagram 4's inspection loop; none is caught by loss curves.

```mermaid
flowchart LR
  subgraph Bugs["THE FORMAT BUG TAXONOMY"]
    direction TB
    B1["Missing EOS"]
    B2["Wrong role tokens"]
    B3["Specials as text"]
    B4["Trailing spaces"]
    B5["Loss on user turns"]
    B6["Mixed formats"]
  end
  subgraph Symptom["SYMPTOM (at inference)"]
    direction TB
    S1["Model never stops"]
    S2["Can't follow instructions"]
    S3["Trains on literal text"]
    S4["Stopping drifts"]
    S5["Model echoes the prompt"]
    S6["Half dataset malformed"]
  end
  subgraph Cause["ROOT CAUSE"]
    direction TB
    C1["No closing im_end / eot_id"]
    C2["ChatML template on Llama-3 (or vice versa)"]
    C3["Hand-concat instead of template"]
    C4["Whitespace before closing token"]
    C5["labels not -100-masked"]
    C6["Didn't normalize before map()"]
  end

  B1 --> S1 --> C1
  B2 --> S2 --> C2
  B3 --> S3 --> C3
  B4 --> S4 --> C4
  B5 --> S5 --> C5
  B6 --> S6 --> C6

  Note["ALL CAUGHT BY:\ntokenizer.decode(input_ids[0])\nNONE CAUGHT BY:\nloss curves"]

  Bugs --> Note

  style B1 fill:#14141f,stroke:#f08080,color:#f08080
  style B2 fill:#14141f,stroke:#f08080,color:#f08080
  style B3 fill:#14141f,stroke:#f08080,color:#f08080
  style B4 fill:#14141f,stroke:#f08080,color:#f08080
  style B5 fill:#14141f,stroke:#f08080,color:#f08080
  style B6 fill:#14141f,stroke:#f08080,color:#f08080
  style S1 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S2 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S3 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S4 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S5 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style S6 fill:#14141f,stroke:rgba(240,128,128,0.5),color:#e4e4e8
  style C1 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C2 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C3 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C4 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C5 fill:#14141f,stroke:#9494a0,color:#9494a0
  style C6 fill:#14141f,stroke:#9494a0,color:#9494a0
  style Note fill:#08080c,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `#f08080` danger for bugs/anti-patterns, `#82e0aa` success for verified/clean, `#9494a0` muted for secondary, `#e4e4e8` for primary text.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart`, `subgraph`) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.