# Lab Specification — Deep-Dive FTDD-02: See the Three Steers (OLMo 3 base/instruct/think) **Course**: Course 3 — LLM Fine-Tuning Masterclass **Deep-Dive**: FTDD-02 — OLMo 2/3 + Tülu 3 (Ai2) **Duration**: 30–40 minutes (an inference-and-observation lab — no fine-tuning) **Environment**: A machine that can run a 7B model locally (CUDA GPU ≥12GB, or Apple Silicon M-series with 16GB+ unified memory, or a hosted notebook). No training. This is a reading-and-inference lab. --- ## Learning objectives By the end of this lab you will have: 1. **Loaded all three OLMo 3 variants** (base, instruct, think) at the 7B size — the same base, three steers. 2. **Run a fixed reasoning prompt** against each variant and captured the outputs — the raw material for comparison. 3. **Diffed the outputs** to isolate what each steering stage (none, SFT+DPO, +RLVR) contributed — the FT00 steering stack made visible. 4. **Located the Tülu 3 recipe** in `allenai/open-instruct` and identified the three pipeline stages (SFT, DPO, RLVR) in the repo. This lab is deliberately training-free. The point is to *see* what post-training does, not to do it. OLMo 3's three-variant release is the pedagogical gift that makes this possible — the comparison a single-variant release cannot provide. --- ## Phase 0 — Set up (5 min) Choose your inference path. Two options: **Option A — Local (recommended for the full experience).** Install a runner (Ollama or `transformers` + `accelerate`) and pull the three OLMo 3 7B variants: ```bash # Via Ollama (simplest) ollama pull allenai/OLMo-3-7B-base ollama pull allenai/OLMo-3-7B-instruct ollama pull allenai/OLMo-3-7B-think ``` > If the exact tags differ at run time, check `huggingface.co/allenai` for the current OLMo 3 7B release names. The three variants (base/instruct/think) are the point — match those. **Option B — Hosted.** If a 7B model is too heavy for your machine, use the Ai2 playground or a HuggingFace Space running OLMo 3. You will not get the CLI comparison, but you can still run the prompts and diff the outputs. --- ## Phase 1 — The reasoning prompt (5 min) Prepare a fixed reasoning prompt that has a verifiable answer. Use this one (or craft your own with a checkable result): ``` A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? Show your reasoning. ``` (The correct answer: the ball costs $0.05. The common wrong answer, from System-1 thinking, is $0.10.) Run this exact prompt against **all three variants**. Record each output verbatim in a file `ftdd02-three-steers.md`. --- ## Phase 2 — base vs instruct: what did SFT+DPO add? (10 min) Compare the **base** output to the **instruct** output. The base is Layer 1 only (no post-training); the instruct is base + SFT + DPO (Layer 3: format and preference). Observe and record: - **Format.** Did the base produce a well-formed answer, or raw continuation / completion behavior? Did instruct produce a cleaner, chat-shaped response? (SFT's effect — FT12.) - **Following instructions.** Did the base "show reasoning" as asked, or just emit a number? Did instruct comply with the "show your reasoning" instruction more reliably? - **Accuracy.** Did either get the right answer ($0.05)? The expected finding: the base model may complete the text but not follow the instruction cleanly (it is not instruction-tuned); the instruct model follows the format and instruction better (that is what SFT+DPO steered). The *knowledge* (arithmetic) is in both — the difference is behavioral steering, exactly as FT00 predicts. --- ## Phase 3 — instruct vs think: what did RLVR add? (10 min) Now compare the **instruct** output to the **think** output. Both share the same base and the same SFT+DPO. The think variant adds RLVR (Layer 3: reasoning steer, FT14). Observe and record: - **Reasoning depth.** Did the think variant produce more explicit step-by-step reasoning? - **Accuracy.** Did the think variant get the right answer ($0.05) where instruct got the common wrong answer ($0.10)? - **Reasoning style.** Did the think variant "think longer" (more tokens, more explicit checking)? The expected finding: the think variant is more likely to work through the problem explicitly and arrive at the correct answer, because RLVR rewarded correct reasoning on verifiable problems during training. The instruct variant is more susceptible to the System-1 trap ($0.10). This is the RLVR steer — making the model *use* its reasoning capability — made visible. If all three get the right answer, pick a harder reasoning problem (a multi-step word problem, a logic puzzle) and re-run. The steer is clearest on problems where System-1 fails. --- ## Phase 4 — Locate the recipe (5 min) Open the `allenai/open-instruct` repository on GitHub. Find the three Tülu 3 pipeline stages: 1. Locate the SFT stage configuration (data mixture, hyperparameters). 2. Locate the DPO stage configuration (preference dataset, DPO settings). 3. Locate the RLVR stage configuration (the verifiable-reward setup — which verifiers, which problem sets). Record the file paths. You do not need to run anything here — the point is to confirm that the recipe is executable and that each stage is a separable, inspectable component. This is the "ablation-ready" property: if you wanted to skip RLVR (Phase 3 of this lab, reversed), you know exactly which config to omit. --- ## Phase 5 — The reflection (5 min) Write 4–6 sentences answering: **For each pair you compared (base→instruct, instruct→think), did the difference reflect a change in behavior (steering) or a change in knowledge? What does this confirm about the FT00 thesis?** The expected finding: both differences were behavioral, not knowledge-based. SFT+DPO changed *how* the model responds (format, instruction-following); RLVR changed *whether* the model works through the problem carefully. Neither added new facts — the arithmetic and the reasoning capability were in the base. This is the FT00 thesis (fine-tuning steers behavior, not knowledge) demonstrated on three variants of the same model, the cleanest possible control. --- ## Deliverables Submit `ftdd02-three-steers.md` containing: - [ ] The three verbatim outputs (base, instruct, think) on the reasoning prompt. - [ ] The Phase 2 comparison (base vs instruct) — what SFT+DPO added. - [ ] The Phase 3 comparison (instruct vs think) — what RLVR added. - [ ] The three open-instruct file paths (SFT, DPO, RLVR stage configs). - [ ] The Phase 5 reflection (4–6 sentences on steering vs knowledge). --- ## Solution key The expected outputs vary by exact model version, but the *pattern* is stable: - **base** — produces text continuation; may not follow the "show reasoning" instruction; may emit the System-1 answer ($0.10) without checking. Not instruction-tuned. - **instruct** — follows the chat format and the instruction; produces a cleaner answer; still susceptible to the System-1 trap unless it happens to reason carefully. - **think** — produces explicit step-by-step reasoning; more likely to arrive at $0.05 (the correct answer) because RLVR rewarded careful, verifiable reasoning during training. The reflection should name both differences as behavioral (steering), not knowledge-based — confirming FT00. The arithmetic and reasoning capability were in the base model; SFT+DPO and RLVR each steered the model toward using them differently. This is why OLMo 3's three-variant release is the pedagogical gift: the same base, three steers, the steering thesis directly observable. If the differences are small, the most likely cause is that the prompt is too easy (all variants solve it). Move to a harder reasoning problem — the steer is most visible where System-1 fails. --- ## Stretch goals 1. **Run the lab at 32B.** If you have access to a larger GPU (or a hosted 32B), repeat the comparison at the 32B size. Observe how the base model is more capable at 32B (closer to correct even without post-training), which sharpens the question: as the base gets smarter, what does each steer still add? 2. **Ablate via open-instruct.** If you have GPU time, use `allenai/open-instruct` to run the Tülu 3 pipeline on OLMo 3 base with RLVR omitted — produce your own "instruct-without-RLVR" model and compare it to the official think variant. This is real ablation: you are reproducing a stage-skipped variant, the research workflow Tülu 3's openness enables. 3. **Compare OLMo 3 think to a closed reasoner.** Run the same reasoning prompt against a closed frontier reasoning model (if you have API access) and OLMo 3 think. Note the capability gap — this is the 'think ≠ GPT-class reasoning' anti-pattern, quantified. Use it to calibrate expectations for open reasoning models.