# Diagrams — Module FT09: DoRA, rsLoRA, and Modern PEFT

**Module**: FT09 — DoRA, rsLoRA, and Modern PEFT
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — DoRA Magnitude/Direction Decomposition vs LoRA

**Type**: Side-by-side mechanism comparison
**Purpose**: The core FT09 idea on one picture. Vanilla LoRA couples magnitude and direction updates proportionally; DoRA separates them and adapts each independently.
**Reading the diagram**: Two parallel paths from the same frozen base weight. The LoRA path updates magnitude and direction together (locked ratio). The DoRA path splits them and updates magnitude (a tiny vector `m`) and direction (the LoRA adapter) separately.

```mermaid
flowchart TD
  W0["FROZEN BASE WEIGHT W₀\n(d × d)"]

  W0 --> LoRA["LoRA PATH\nΔW = (α/r) · B·A"]
  W0 --> DoRA["DoRA PATH\ndecompose W₀"]

  LoRA --> Coupled["UPDATE COUPLES\nmagnitude ⊥ direction\n(locked ratio)\ncannot rotate a direction\nwithout rescaling its magnitude"]
  Coupled --> LoRAOut["W = W₀ + ΔW\n~half the gap to full FT\nremains unreachable"]

  DoRA --> Split["split into\nmagnitude m (per column)\n+ direction (W₀ / ‖W₀‖)"]
  Split --> Mag["magnitude m\ntiny trainable vector\n(d scalars)"]
  Split --> Dir["direction\nunit-normalized\n+ LoRA adapter\n(r × d, d × r)"]
  Mag --> Recombine["recombine:\nm ⊙ direction + ΔW_dir"]
  Dir --> Recombine
  Recombine --> DoRAOut["W = m ⊙ (W₀/‖W₀‖) + ΔW_dir\ncloses ~half the gap to full FT\nzero inference overhead after merge"]

  style W0 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style LoRA fill:#14141f,stroke:rgba(240,168,104,0.6),color:#f0a868
  style Coupled fill:#08080c,stroke:rgba(240,128,128,0.5),color:#f08080
  style LoRAOut fill:#14141f,stroke:rgba(240,168,104,0.6),color:#f0a868
  style DoRA fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Split fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Mag fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Dir fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Recombine fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style DoRAOut fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
```

---

## Diagram 2 — The Modern PEFT Family Tree

**Type**: Taxonomy / family tree
**Purpose**: Place every modern PEFT method on the standard-vs-niche axis. DoRA, rsLoRA, PiSSA, GaLore are standard; VeRA, AdaLoRA, MiSS are niche or experimental.
**Reading the diagram**: LoRA (FT08) is the trunk. Three branches: the decomposition/scaling branch (DoRA, rsLoRA — the defaults), the initialization branch (PiSSA — alternative init), and the full-FT bridge (GaLore). The niche methods hang off as research leaves.

```mermaid
flowchart TD
  LoRA["LoRA (FT08)\nthe trunk\nfreeze base, train B·A"]

  LoRA --> Defaults["THE 2026 DEFAULTS"]
  LoRA --> Alt["ALTERNATIVE INIT"]
  LoRA --> Bridge["THE FULL-FT BRIDGE"]
  LoRA --> Niche["NICHE / RESEARCH"]

  Defaults --> DoRA["DoRA\nmagnitude/direction decomposition\ncloses ~half gap to full FT"]
  Defaults --> rsLoRA["rsLoRA\nα/√r scaling\nstabilizes high rank r ≥ 64"]

  Alt --> PiSSA["PiSSA\nSVD-based initialization\nfaster convergence on NLU"]

  Bridge --> GaLore["GaLore\ngradient low-rank projection\nfull-param FT at near-LoRA memory"]

  Niche --> VeRA["VeRA\nshared frozen random matrices\n~10× fewer params; storage-bottleneck only"]
  Niche --> AdaLoRA["AdaLoRA\nadaptive rank allocation\nlargely superseded by DoRA"]
  Niche --> MiSS["MiSS\nclaims SOTA\nnot widely reproduced"]

  DoRA -.stacks with.-> rsLoRA

  style LoRA fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Defaults fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Alt fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Bridge fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Niche fill:#14141f,stroke:rgba(240,168,104,0.5),color:#f0a868
  style DoRA fill:#08080c,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style rsLoRA fill:#08080c,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style PiSSA fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style GaLore fill:#08080c,stroke:rgba(94,234,212,0.5),color:#5eead4
  style VeRA fill:#08080c,stroke:rgba(240,168,104,0.5),color:#f0a868
  style AdaLoRA fill:#08080c,stroke:rgba(240,168,104,0.5),color:#f0a868
  style MiSS fill:#08080c,stroke:rgba(240,168,104,0.5),color:#f0a868
```

---

## Diagram 3 — Quality/Cost Positioning (DoRA Closes Half the Gap)

**Type**: Number-line / positioning
**Purpose**: Visualize the central DoRA claim — it closes roughly half the gap between vanilla LoRA and full fine-tuning, at only ~5–10% more VRAM than LoRA.
**Reading the diagram**: A quality axis from LoRA (left) to full FT (right). DoRA sits halfway. QDoRA brings the DoRA quality point down to QLoRA's memory cost. GaLore reaches full-FT quality at intermediate memory.

```mermaid
flowchart LR
  subgraph Axis["QUALITY  (← lower)            (higher →)"]
    direction LR
    L["LoRA\nbaseline quality\nbaseline VRAM"]
    D["DoRA + rsLoRA\ncloses ~HALF the gap\n+5–10% VRAM over LoRA"]
    G["GaLore\nfull-param quality\nnear-LoRA optimizer mem"]
    F["Full FT\nreference quality\n10–20× VRAM"]
    L --- D --- G --- F
  end

  subgraph Cost["MEMORY/COST (lower better)"]
    direction LR
    QL["QDoRA\nDoRA quality\nat QLoRA cost\nleading quality/cost"]
  end

  D -.quantize the base.-> QL

  style Axis fill:#08080c,stroke:rgba(255,255,255,0.08),color:#e4e4e8
  style L fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style D fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style G fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style F fill:#14141f,stroke:rgba(240,128,128,0.6),color:#f08080
  style Cost fill:#08080c,stroke:rgba(130,224,170,0.4),color:#e4e4e8
  style QL fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
```

---

## Diagram 4 — The Stack: DoRA + rsLoRA Combine

**Type**: Layered stack
**Purpose**: Show *why* DoRA and rsLoRA are complementary, not competing. They edit different parts of the update. Stacking them beats either alone.
**Reading the diagram**: The adapter update has three conceptual layers. The base weight (frozen). The direction adapter (LoRA matrices). The scaling factor applied to the adapter. DoRA adds the magnitude/direction split on top of the frozen base; rsLoRA swaps the scaling factor inside the direction adapter. Two independent knobs.

```mermaid
flowchart TD
  subgraph Stack["THE DoRA + rsLoRA STACK"]
    direction TB
    L4["SCALING FACTOR\nvanilla LoRA: α/r   →   rsLoRA: α/√r\n(only matters at r ≥ 64)\nFLAG: use_rslora=True"]
    L3["DIRECTION ADAPTER\nLoRA matrices B, A  (r × d, d × r)\ntrained by the optimizer"]
    L2["MAGNITUDE/DIRECTION SPLIT\nDoRA decomposes W₀ into m ⊙ (W₀/‖W₀‖)\nFLAG: use_dora=True"]
    L1["FROZEN BASE WEIGHT W₀\nnever updated"]
    L1 --> L2 --> L3 --> L4
  end

  L4 --> Out["effective update:\nm ⊙ (W₀/‖W₀‖) + (α/√r)·B·A\nDoRA decomposition + rsLoRA scaling\northogonal improvements — stack them"]

  style Stack fill:#08080c,stroke:rgba(255,255,255,0.08),color:#e4e4e8
  style L1 fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style L2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style L3 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style L4 fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Out fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
```

---

## Diagram 5 — The PEFT Decision: From LoRA to GaLore

**Type**: Decision flow
**Purpose**: Where each method belongs in the practical decision. The 2026 default is DoRA + rsLoRA; you escalate to QDoRA on memory pressure, or to GaLore only when DoRA is measured-insufficient.
**Reading the diagram**: Top-down. Start at the steering task. Default to DoRA + rsLoRA. If memory-constrained, QDoRA. If DoRA measured-insufficient on held-out eval, escalate to GaLore (full FT). Vanilla LoRA and vanilla full FT are the methods you no longer start with.

```mermaid
flowchart TD
  Start["Steering task on a base model\n(FT08 prereq: you know LoRA & QLoRA)"]

  Start --> Q1{"Held-out eval says\nDoRA-quality is sufficient?\n(measured, not vibes)"}

  Q1 -->|"Yes — the common case"| Default["DoRA + rsLoRA\nuse_dora=True, use_rslora=True\n2026 DEFAULT"]
  Q1 -->|"Yes, but GPU < 12 GB"| QDoRA["QDoRA\nDoRA on 4-bit quantized base\nleading quality/cost"]
  Q1 -->|"No — measured gap to full FT"| Q2{"Need true full-parameter\nupdate? (large domain shift)"}

  Q2 -->|"Yes"| GaLore["GaLore\nfull-param FT at near-LoRA memory\nthe bridge to full FT"]
  Q2 -->|"No — try harder adapter"| PiSSA["PiSSA initialization\n+ DoRA + rsLoRA\nfaster convergence, NLU-strong"]

  Default --> Ship["Ship merged adapter\nzero inference overhead"]
  QDoRA --> Ship
  GaLore --> Ship2["Ship full model\n(no adapter hot-swap)"]
  PiSSA --> Ship

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Q1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Q2 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Default fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style QDoRA fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style GaLore fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style PiSSA fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
  style Ship fill:#14141f,stroke:rgba(130,224,170,0.6),color:#82e0aa
  style Ship2 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#5eead4
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` (`--accent`) for the primary/DoRA path, `rgba(94,234,212,0.4)` for decision diamonds, `#f08080` (`--danger`) for the structural-deficit / high-cost nodes (full FT, LoRA coupling), `#f0a868` (`--warn`) for niche/research nodes, `#82e0aa` (`--ok`) for the success / default path, `#e4e4e8` / `#9494a0` for text.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart`, `subgraph`, dashed `-.label.->` links, `---` undirected links) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.