# Module FT03 — Base Model Selection **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT03 — Base Model Selection **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT00 (The Steering Stack), FT01 (VRAM Math), FT02 (The Open Spectrum) --- ## Learning Objectives After completing this module, you will be able to: 1. State the five-dimension base selection rubric — **task, hardware, license, openness, ecosystem** — and explain why their weights shift by use case rather than holding fixed. 2. Defend the choice of MiniCPM5-1B as the course's default teaching base, and name the signals that tell you to graduate to a larger base. 3. Identify the base-model properties that genuinely matter for fine-tuning (tokenizer/domain fit, context length, chat template quality, license, open-data availability, tooling support) versus those that people over-index on (raw benchmark scores, parameter count in isolation). 4. Place the major base families — OpenBMB MiniCPM, Qwen, Llama, DeepSeek, Mistral/Mixtral, SmolLM3/OLMo/Tülu — on a map of size × openness and describe each family's fine-tuning profile. 5. Decide, for a given goal, whether to start from a **base**, **instruct**, or **chat** checkpoint — and recognize the "wrong base" anti-patterns that waste compute or destroy provenance. --- # 3.1 — The Five-Dimension Rubric *Base selection is not a leaderboard lookup. It is a constrained optimization across five dimensions, and the weights are set by your use case.* ## The dimensions | # | Dimension | The question it answers | What "good" looks like | | --- | --- | --- | --- | | 1 | **Task** | Can the base, with a perfect prompt, already produce the target behavior? | The capability is in the pretraining distribution; fine-tuning only needs to steer it. | | 2 | **Hardware** | Can you afford to load, train, and serve it — per the FT01 VRAM math? | Fits your GPU budget for training *and* your deployment target (edge, server, phone). | | 3 | **License** | Does the license permit your deployment (commercial, government, HIPAA, air-gap)? | Apache-2.0 / MIT for frictionless commercial; check the Llama community license's 700M clause. | | 4 | **Openness** | Can you audit what the model saw — open-data vs open-weights-only (FT02)? | Open-data (MiniCPM, OLMo, Tülu, SmolLM3) for auditable/regulated work; open-weights acceptable otherwise. | | 5 | **Ecosystem** | Does the toolchain you intend to use support this base? | First-class transformers / TRL / Unsloth / Axolotl / vLLM support; a clean chat template. | Read them as a sequence, not a checklist. The order is deliberate: **task** sets the capability floor; **hardware** sets the size ceiling from FT01's VRAM math; **license** and **openness** are the gates that can veto an otherwise-perfect base; **ecosystem** is the friction coefficient that determines how many of your weekends you lose to `trust_remote_code` bugs. ### Why the weights shift The rubric is weighted, not ranked. Three examples make the point: - **A medical QA bot for a HIPAA-regulated hospital.** Openness dominates. An open-weights-only base is a non-starter because you cannot prove the training corpus is free of PHI leakage. License matters (commercial use). Ecosystem matters less — you'll pay the integration tax for auditability. Task still sets the floor, but the gate is dimension 4. - **A security exploit-code generator for authorized pentests.** License and ecosystem matter: you need a base whose license permits the use and whose toolchain supports the uncensoring pipeline (Llama-family ablation targets are the dominant choice here). Openness is secondary — you are not proving provenance to a regulator; you are removing refusals. Task dominates: the base must already know exploit syntax. - **A phone-based on-device assistant.** Hardware dominates absolutely. A 70B base is irrelevant no matter how good its benchmarks; the model must fit in phone RAM. Task still sets the floor, but the ceiling is physics. This is MiniCPM's home turf. The discipline: **write the weights down before you browse the leaderboard.** A team that skips this step ends up fine-tuning whatever topped Hacker News that week. ## The on-ramp principle There is a reason this course teaches on a 1B base and graduates upward: **iteration speed compounds.** A base that trains a LoRA in 8 minutes lets you run ten experiments before lunch. A base that trains a LoRA in 8 hours lets you run one. Most of what you learn about fine-tuning — data quality, format, hyperparameter sensitivity — you learn from iteration volume, not from any single run. Start small, validate the pipeline, then scale the base only when the task's reasoning demands it. --- # 3.2 — Why MiniCPM5-1B Is the Course Default *The teaching base is a pedagogical choice, not a production recommendation. MiniCPM5-1B wins on the dimensions that matter for learning: speed, openness, license, and ecosystem.* OpenBMB's MiniCPM5-1B — a dense ~1.08B-parameter causal LM, the first checkpoint of the MiniCPM5 series — is what every hands-on lab in this course will use unless stated otherwise. The Sam Witteveen walkthrough ("MiniCPM5 — Just How Good Can a 1B Model Be?", youtu.be/ox1mW2N9Z_Y) is the recommended companion watch; it shows the model punching above its weight on agentic and tool-use tasks, which is exactly the regime where a 1B model is a credible teaching target rather than a toy. The five-dimension scorecard: 1. **Task.** MiniCPM5-1B is capable enough to demonstrate every technique in this course — SFT, DPO, GRPO, abliteration. It will not beat a 70B on hard reasoning, but it does not need to. The point is to *feel* the steering, and a 1B model steers visibly. When you abliterate a 1B, the behavior change is obvious in minutes. 2. **Hardware.** A 1B model loads in ~2GB at FP16, trains a QLoRA on a free Colab T4, and runs on a phone. This is what makes the labs accessible. FT01's VRAM math is the constraint; MiniCPM5-1B clears it by a wide margin on consumer hardware. 3. **License.** Apache-2.0. Commercial use, modification, and redistribution without a questionnaire. This is frictionless for a course that publishes adapters and derivative work. 4. **Openness.** OpenBMB publishes the MiniCPM training pipeline and data recipes (UltraData tiered management); the MiniCPM line is the open-data hero alongside Ai2's OLMo/Tülu and HuggingFace's SmolLM3. This matters pedagogically because FT02's audit framing is not hypothetical — the base you train on is one you could, in principle, audit. 5. **Ecosystem.** MiniCPM5-1B ships as a standard `LlamaForCausalLM` architecture, first-class in transformers, with vLLM and Ollama serving recipes published by the maintainers. No exotic `trust_remote_code` archaeology. TRL and Unsloth work out of the box. ### When to graduate MiniCPM5-1B stops being the right base when the *task* dimension fails — when the reasoning demand exceeds what a 1B can steer toward. The signals: - You prompt the base with a perfect system prompt and it produces the wrong *kind* of answer, not just an unreliable one. That is a capability ceiling, not a steering problem (FT00's three-outcome test, outcome 3). - Your eval has a hard reasoning floor (multi-hop, long-chain tool use) and the 1B scores below your acceptable threshold *before* fine-tuning. Fine-tuning steers; it does not raise the ceiling. - You need a context window or tokenizer the 1B lacks for your domain (e.g., a non-Latin script where the 1B's tokenizer inflates sequence length). The graduation path: MiniCPM5-1B → MiniCPM5-3B/4B or Qwen2.5/3 at 3–7B →, for production reasoning, Qwen3 or DeepSeek-R1-distill lineage at 7–14B+. Each step roughly squares the VRAM and halves the iteration speed — which is why you validate the pipeline on the 1B first. --- # 3.3 — Properties That Matter (and Properties That Don't) *Most base-selection debates are spent on the wrong properties. Here is the load-bearing list and the over-indexed list.* ## Properties that matter for fine-tuning - **Tokenizer / domain fit.** A tokenizer that fragments your domain into many sub-tokens inflates sequence length, VRAM, and training time, and can hurt quality. A base whose tokenizer covers your script/code/notation efficiently is a real advantage. This is why Chinese-heavy work often prefers Qwen, and why code work prefers bases with code-aware tokenizers. - **Context length.** The native training context sets where RoPE extrapolation begins to degrade. MiniCPM5-1B supports a long context (the MiniCPM5 series targets on-device assistants with extended context); MiniCPM-V 4.6 reaches 262K. Match the context to your task's actual span — but remember FT01: context length multiplies VRAM linearly for attention. - **Chat template quality.** A clean, documented chat template (the Jinja applied by `apply_chat_template`) is what makes SFT data formatting tractable. Bases with messy or undocumented templates cost you debugging time and produce silent format mismatches between training and inference. This is an ecosystem-quality signal as much as a model-property one. - **License.** The gate. Apache-2.0 and MIT are frictionless. The Llama community license has a 700-million-monthly-active-user clause and redistribution restrictions that matter for some deployments. Know the license before you train — re-training on a different base because the license failed legal review is the most expensive redo in this list. - **Open-data availability.** The FT02 axis. Open-data bases (MiniCPM, OLMo, Tülu, SmolLM3) let you audit the training corpus; open-weights-only bases (Llama 3.x) do not. For regulated or reproducibility-sensitive work this is a hard requirement, not a preference. - **Ecosystem / tooling support.** First-class support in transformers, TRL, Unsloth, Axolotl, and vLLM/Ollama. A base that is a standard architecture with maintained configs saves you from `trust_remote_code` rot and integration tax. This is the dimension that determines whether your weekend project ships. ## Properties people over-index on - **Raw benchmark scores.** You are *steering* a base, not competing on a leaderboard. A base that scores 2 points higher on MMLU but has a worse tokenizer for your domain is the wrong choice. Benchmarks measure the base's general capability; they do not measure how well it steers toward *your* behavior. The FT00 thesis cuts both ways: if fine-tuning is steering, then the base's benchmark ceiling matters only insofar as your task needs that capability — and the steering itself is roughly dimension-orthogonal to the benchmark. - **Parameter count in isolation.** "Bigger is better" is the wrong frame. The VRAM math from FT01 decides what fits — parameters, optimizer state, activation memory, context. A 7B you can't afford to train is worse than a 1B you can iterate on ten times. Parameter count only matters once it has passed through the FT01 filter *and* the task-capability filter. --- # 3.4 — The Major Base Families and Their Fine-Tuning Profiles *Each family has a personality. Match the personality to the use case.* ## OpenBMB MiniCPM — the teaching/edge/open-data hero The MiniCPM family — MiniCPM5-1B, MiniCPM5-3B/4B, MiniCPM-V 4.6 (multimodal), MiniCPM-o 4.5 (omni) — is this course's default. Dense, efficient, Apache-2.0, open-data pipeline, strong on agentic and tool-use for the size. The V and o variants extend to vision and audio while staying edge-deployable. **Fine-tuning profile:** first-class ecosystem, fast iteration, ideal for learning and for on-device production where the 1B's capability is sufficient. ## Qwen (2.5, 3) — the production workhorse Alibaba's Qwen line is the dominant production base for teams that need strong reasoning and broad coverage without the Llama license terms. Qwen3 introduces a hybrid thinking mode (reasoning on/off) that matters for steering decisions: you may want to fine-tune the non-thinking path for speed, or the thinking path for reasoning. **Fine-tuning profile:** excellent tokenizer for multilingual and code, strong ecosystem support, the default when you graduate past 1B and need production-grade reasoning. ## Llama (3.x, 3.2, 3.3) — the dominant ablation target Meta's Llama is the most-ablated open-weights family in existence — the base for the vast majority of uncensored/Dolphin-style community work. **Fine-tuning profile:** open-weights-only (not open-data), Llama community license (check the 700M clause), but unmatched community tooling and ablation precedent. If you are doing alignment-control work (FT16–FT18), Llama is where the recipes live. The trade-off is the openness gap from FT02. ## DeepSeek (V3, R1, R1-distills) — the reasoning distillation lineage DeepSeek-R1 and its distills (R1-Distill-Qwen, R1-Distill-Llama) are the base when the task is reasoning distillation — teaching a smaller model to reason by training on R1's traces. **Fine-tuning profile:** the lineage matters more than the base size; if your task is "make this model reason like R1," you start from an R1-distill and steer. V3 is the dense/MoE production base; R1 is the reasoning teacher. ## Mistral / Mixtral (MoE) — Dolphin's home turf Mistral's dense models and Mixtral's Mixture-of-Experts are the historical home of the Dolphin (uncensored) lineage. **Fine-tuning profile:** MoE bases have a subtler fine-tuning story (experts can specialize or collapse under steering); the Dolphin recipes are battle-tested. Apache-2.0 for the early Mistral releases. A reasonable choice when you want the Dolphin lineage's provenance and the MoE efficiency for serving. ## SmolLM3, OLMo, Tülu — fully open for reproducibility HuggingFace SmolLM3, Ai2 OLMo, and Ai2 Tülu are the fully-open bases — weights, data, code, and (for OLMo/Tülu) the full training recipe. **Fine-tuning profile:** chosen when reproducibility and auditability are the overriding requirement (academic, regulated, or you need to *prove* provenance). Often smaller and less benchmark-competitive than Qwen/Llama, but the openness is the point. This is the FT02 extreme. --- # 3.5 — Base vs Instruct vs Chat: Where to Start *The checkpoint you start from is itself a steering decision.* Most bases ship in multiple checkpoints. The three you will encounter: - **Base.** The raw pretrained weights. No instruction tuning, no chat formatting. The model completes text. **Start here** when you want full control over the behavior and intend to do your own SFT from a clean slate. The FT00 lab uses the base-style interaction. - **Instruct.** Instruction-tuned — the model has had an SFT pass for instruction-following. **Start here** for most real fine-tuning: you inherit reliable instruction-following and steer *on top of* it. This is the default for production work. - **Chat.** A full chat-tuned checkpoint — instruction-tuned *and* preference-aligned (RLHF/DPO) for conversational behavior. **Start here** only when you specifically want to preserve the existing alignment and steer a narrow behavior on top of it. Starting from chat and then abliterating is the standard uncensoring path (FT17), because you keep the chat quality while removing the refusal. The rule of thumb: > **Default to instruct. Use base when you want to build the behavior from scratch. Use chat when you want to preserve existing alignment and steer narrowly.** The common error is fine-tuning an *instruct* model when you wanted a *base* — you then spend data and compute re-imposing behaviors the instruct pass already gave you, or fighting the instruct alignment. Know which checkpoint you loaded. --- # 3.6 — "Wrong Base" Anti-Patterns ### Fine-tuning an instruct model when you wanted base You intend to build instruction-following from scratch, but you loaded the instruct checkpoint. Now your SFT data is either redundant (the model already follows instructions) or fighting the existing alignment. **Symptom:** the model's behavior barely changes after SFT. **Fix:** check `config.json` and the model card — confirm whether you loaded `-base` or `-instruct`. ### Fine-tuning a merged community model with no provenance You grab a community merge ("SuperMega-L3-Uncensored-All-V8") with no documented provenance. This is the FT02 failure mode in base-selection clothing: you cannot audit what was merged in, you cannot reproduce it, and your fine-tune inherits an unknown capability and alignment surface. The result is unpredictable degradation and an un-debuggable model. **Fix:** prefer the upstream canonical checkpoint; if you must use a merge, require a documented merge recipe. This ties directly to FT02's audit principle. ### Choosing a base too small for the task's reasoning demands A 1B base is a teaching tool, not a universal answer. If your task requires multi-hop reasoning the 1B cannot do even with a perfect prompt, no amount of steering will get you there (FT00 outcome 3). **Symptom:** the base, prompted perfectly, produces the wrong *kind* of answer. **Fix:** graduate to a larger base from the same family, or switch families to one with stronger reasoning (Qwen3, DeepSeek-R1-distill). ### Ignoring tokenizer / domain fit You pick a base whose tokenizer fragments your domain (e.g., a Latin-centric tokenizer for a CJK-heavy task, or a text tokenizer for heavy code). Sequence lengths balloon, VRAM multiplies, and quality can suffer. **Symptom:** your training sequences are far longer than the source text implies; token efficiency is poor. **Fix:** check tokens-per-character for representative domain text *before* committing to the base. --- ## Anti-Patterns (summary) ### Leaderboard-driven selection Picking the base that tops the benchmark this week, ignoring license, openness, and tokenizer fit. Benchmarks measure general capability, not steerability toward your behavior. ### Skipping the VRAM filter Choosing a 70B because it's "best," then discovering you can't train it. The FT01 math is a gate, not an afterthought. ### Treating openness as a tie-breaker For regulated work, openness is a hard requirement, not a nice-to-have. Discovering this during legal review is too late. ### Mismatched checkpoint Loading instruct when you wanted base, or chat when you wanted instruct. Know which checkpoint you are steering from. --- ## Key Terms | Term | Definition | | --- | --- | | **Base selection rubric** | The five-dimension framework: task, hardware, license, openness, ecosystem — weighted by use case | | **Open-data model** | A base released with its training corpus (MiniCPM, OLMo, Tülu, SmolLM3) — auditable (FT02) | | **Open-weights-only** | A base released as weights without training data (Llama 3.x) — not auditable | | **Base checkpoint** | Raw pretrained weights; no instruction tuning; completes text | | **Instruct checkpoint** | Base + an SFT pass for instruction-following; the default starting point for most fine-tuning | | **Chat checkpoint** | Instruct + preference alignment (RLHF/DPO); start here to preserve existing alignment | | **Chat template** | The Jinja applied by `apply_chat_template` that formats turns for the model; an ecosystem-quality signal | | **Ablation target** | A base family commonly used for uncensoring/alignment-control work (Llama is the dominant one) | | **Graduation** | Moving from a small teaching base (MiniCPM5-1B) to a larger production base when the task's reasoning demands it | | **Tokenizer fit** | How efficiently a base's tokenizer covers your domain; poor fit inflates sequence length and VRAM | --- ## Lab Exercise See `07-lab-spec.md`. The "Pick Three" lab: for three use cases (a medical QA bot, a security exploit-code generator for authorized pentests, a phone-based assistant), select the base and defend each choice against the five-dimension rubric. No GPU needed — this is a judgment and architecture lab. --- ## References 1. **OpenBMB / MiniCPM5-1B** — the course's default teaching base. Dense ~1.08B-parameter causal LM, Apache-2.0, open-data pipeline. ModelScope/HuggingFace: `openbmb/MiniCPM5-1B`. 2. **Sam Witteveen (2025)** — *MiniCPM5 — Just How Good Can a 1B Model Be?* youtu.be/ox1mW2N9Z_Y. The companion walkthrough for this module's default base. 3. **Module FT00 — The Steering Stack** — the thesis (steering not teaching) and the three-outcome test that sets the task dimension's capability floor. 4. **Module FT01 — VRAM Math** — the hardware dimension; the filter that decides what fits. 5. **Module FT02 — The Open Spectrum** — the openness dimension; open-data vs open-weights-only and why it is a hard gate for regulated work. 6. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. arXiv:2007.07784. Why steering works at low rank — the reason parameter count is not the overriding property. 7. **NTIA (2024)** — *AI Open-Model Weights Report*. The policy backdrop for the openness dimension. 8. **Ai2 OLMo / Tülu** and **HuggingFace SmolLM3** — the fully-open reference bases for reproducibility-sensitive work.