"front"	"back"	"tag"
"What layer of the Steering Stack is quantization, and what is its defining property?"	"Layer 4 — the Export. Defining property: it is DOWNSTREAM of training. You quantize AFTER you fine-tune. It compresses what training produced (compactness/speed); it does not change learned behavior at sensible bitrates."	c3::ft19::recall
"What is the single most important rule for choosing a quantization format?"	"The format is determined by the INFERENCE ENGINE that will run the model, not chosen in the abstract. Pick the deployment target/runtime first, then the format that runtime consumes."	c3::ft19::recall
"Name the three formats you reach for most often, and the deployment target each serves."	"GGUF (universal local — Ollama/llama.cpp/LM Studio, runs on CPU/mixed/Mac Metal). AWQ (NVIDIA GPU production serving via vLLM/TGI, esp. AWQ-Marlin). MLX (Apple Silicon native via mlx-lm / LM Studio)."	c3::ft19::recall
"What is GGUF, and what makes it the universal local format?"	"The file format consumed by llama.cpp and everything built on it (Ollama, LM Studio, llamafile). Self-contained .gguf file (weights+tokenizer+config+chat template). Runs on CPU, mixed CPU/GPU offload, and Mac Metal. Maximum compatibility of any format."	c3::ft19::recall
"What is the GGUF sweet-spot quant and why?"	"Q4_K_M — roughly 4 bits/parameter, ~75% size reduction vs FP16 with minimal perceptible quality loss. The community default. A 7B model goes from ~14GB (FP16) to ~4.4GB. Reach for it first; only go to Q5/Q8 with VRAM to spare or Q3/Q2 if desperate."	c3::ft19::recall
"What does the '_M' mean in GGUF's Q4_K_M?"	"'Medium' — a balance of layers quantized at different precisions within the K-quant scheme. The K-quant ladder (Q4_K_S, Q4_K_M, Q5_K_M, etc.) lets you trade size for quality in small steps. When in doubt, take the _M."	c3::ft19::recall
"What is AWQ and where does it win?"	"Activation-aware Weight Quantization. Uses a calibration set to find the ~1% of 'salient' weights that matter most for output quality and keeps those high-precision while quantizing the rest. The 4-bit format for vLLM/TGI production serving on NVIDIA GPUs (AWQ-Marlin = optimized kernel). Best speed/quality for server inference."	c3::ft19::recall
"How does AWQ compare to GPTQ at 4-bit, in one sentence?"	"AWQ is slightly BETTER in quality (lower perplexity) than GPTQ at the same 4-bit size. GPTQ is more mature and ubiquitous but 'showing its age.' Prefer AWQ for new server deployments; GPTQ is fine where it is what tooling supports or what's already published."	c3::ft19::recall
"What is EXL2 and what is its defining advantage and limitation?"	"ExLlamaV2's format. VARIABLE-RATE per-layer quantization — measures each layer's sensitivity, assigns different bitrates so the total hits a target average. Best-in-class perplexity at small sizes (e.g., squeeze a 70B into the smallest footprint). Limitation: ExLlamaV2-only runtime (vLLM support experimental)."	c3::ft19::recall
"What is MLX and what is its recommended quant?"	"Apple's array framework, purpose-built for Apple Silicon's unified memory. Best Mac inference perf via mlx-lm / LM Studio. Recommended quant: 4-bit group quantization (mlx-community publishes as *-4bit) — ~75% size reduction, minimal loss, like GGUF's Q4_K_M."	c3::ft19::recall
"What is FP8, and when is it the right choice?"	"8-bit floating point. The high-quality serving option when VRAM allows — 2x size/speed win over FP16 with negligible quality loss. Has hardware acceleration on Hopper (H100) and Blackwell. Use when the model fits in FP8 and you want max quality short of staying FP16."	c3::ft19::recall
"What are MXFP4 and NVFP4, and how do they differ?"	"Both are 2025 Blackwell-native E2M1 4-bit floating-point formats (1 sign, 2 exponent, 1 mantissa). Differ in BLOCK SIZE for shared scaling: MXFP4 = 32 elements/block; NVFP4 = 16 elements/block. NVFP4's finer-grained blocks give better accuracy at the same 4-bit width. Both support training and inference."	c3::ft19::recall
"What is the general sweet spot of the quality/size trade-off, and roughly what size reduction does it give?"	"Q4 (~4 bits/param). ~75% size reduction vs FP16 with minimal perceptible quality loss (fraction of a point on MMLU, small perplexity uptick). Where you start and usually stop."	c3::ft19::recall
"What happens at Q2/Q3 vs Q8 on the quality/size curve?"	"Q2/Q3 (lower): save more memory but degradation becomes NOTICEABLE — more errors, worse reasoning, occasional incoherence. Desperate-only. Q8 (higher): ~58% smaller, effectively lossless (indistinguishable from FP16). Use when you have the memory and want certainty."	c3::ft19::recall
"What is Unsloth Dynamic 2.0 and why does it beat uniform quants?"	"Intelligent per-layer quantization. Measures each layer's sensitivity and assigns bitrates accordingly — sensitive layers stay at higher precision (5/6/8-bit), tolerant layers drop to 3-bit. Total hits the same average size as uniform Q4 but quality is HIGHER because bits went where they matter. Same insight as EXL2, packaged for the standard conversion workflow."	c3::ft19::recall
"Given a deployment target of a local developer laptop with CPU-only inference, which format and why?"	"GGUF (Q4_K_M). It is the only one of the three primary formats that runs well on CPU and mixed CPU/GPU offload — that portability is exactly its strength. AWQ needs CUDA kernels; MLX needs Apple Silicon. GGUF is the universal local format."	c3::ft19::application
"Given a deployment target of a Mac Studio M-series for an on-device assistant, which format and why?"	"MLX 4-bit. MLX is Apple Silicon-native and exploits unified memory (no CPU-to-GPU copy), giving the best inference perf on a Mac. GGUF-on-Metal works but MLX is faster on Apple hardware. mlx-lm or LM Studio loads it directly."	c3::ft19::application
"Given a deployment target of an H100 vLLM server with many concurrent users, which format and why?"	"AWQ 4-bit Marlin. AWQ-Marlin has first-class vLLM support with optimized kernels — best speed/quality for server inference. GGUF's portability is irrelevant on a fixed vLLM box; EXL2's vLLM support is experimental. AWQ-Marlin is the default for NVIDIA production serving."	c3::ft19::application
"You need to squeeze a 70B model into the smallest possible footprint without wrecking quality, and you control the runtime. Which format and why?"	"EXL2. Its variable-rate per-layer quantization gives best-in-class perplexity at small sizes — some layers 8-bit, some 4, some 3, tuned to each layer's sensitivity. The cost is ExLlamaV2-only runtime support, which you accept because you control the deployment."	c3::ft19::application
"You have an H100 with VRAM to spare and care about preserving max quality. Which format and why?"	"FP8. 2x size/speed win over FP16 with negligible quality loss, and H100 has hardware FP8 acceleration. It is the highest-quality compressed option short of staying in FP16. Q4 would be smaller but FP8 gives you the quality headroom the H100's VRAM affords."	c3::ft19::application
"A team quantized their model to Q2 to fit a small device and is surprised the output quality dropped. What went wrong and what should they do?"	"They treated Q2 as a default rather than a desperate measure. Q2 is in the 'noticeable degradation' zone of the quality/size curve — more errors, worse reasoning. Fix: step UP to Q4_K_M (the sweet spot) and re-measure against FP16; only go below Q4 with eyes open and a benchmark in hand."	c3::ft19::application
"Why must you benchmark the quantized model against the FP16 original before shipping?"	"Quantization is lossy in principle. The only way to know HOW lossy for your model and workload is to measure — run the same eval (GSM8K, MMLU, your domain benchmark, perplexity) on FP16 and the quant. If the delta is within tolerance, ship; if not, step up a bitrate or change format. Shipping an unmeasured quant is flying blind."	c3::ft19::analysis
"Why is quantization described as 'downstream of training' rather than part of it? What does it preserve?"	"Layer 4 compresses the artifact Layer 3 produced; it is a TRANSFORM on the final weights, not a step in the optimization. At sensible bitrates (Q4+) it perturbs weights slightly but does not move behavior across any decision boundary that matters — the model's steering (format, preferences, reasoning activation, compliance) is PRESERVED. It changes compactness/speed, not learned behavior."	c3::ft19::analysis
"How does QLoRA's quantization (FT08) differ from FT19's deployment quantization?"	"QLoRA quantizes the base DURING training as a memory trick to fit forward/backward passes in VRAM; the adapter itself stays high-precision. FT19 quantizes the merged checkpoint AFTER training as the deployment export step. Different concerns, different points in the pipeline. Even QLoRA's training quant is not the same as the export quant — you merge, then quantize fresh for serving."	c3::ft19::analysis
"Why does per-layer (variable-rate) quantization — EXL2, Unsloth Dynamic 2.0 — beat uniform quantization at the same average size?"	"Not all layers are equally sensitive. Some (often attention projections, certain MLP blocks) have weight distributions that degrade badly at 4-bit; others compress cleanly to 3-bit with no measurable effect. Uniform wastes bits on tolerant layers and starves sensitive ones. Per-layer allocation puts the bits where the quality cost of compression is highest, so the same total bitrate yields lower perplexity."	c3::ft19::analysis
"A team loaded an AWQ checkpoint into an Ollama instance and it failed. Diagnose the root cause."	"Format/runtime mismatch. AWQ is consumed by vLLM/TGI on NVIDIA GPUs; Ollama runs GGUF. The format must match the runtime — you decide the deployment target first, then convert to the format that target consumes. The fix is to convert the checkpoint to GGUF Q4_K_M for the Ollama target (and keep the AWQ copy for the vLLM server)."	c3::ft19::analysis
"Why is 'one source checkpoint, three export artifacts' the right mental model for the conversion step?"	"Because quantization is cheap and repeatable (minutes, not training-hours) and does not change learned behavior. You train ONCE in FP16/BF16, then convert the same merged checkpoint to GGUF (local), AWQ (NVIDIA server), MLX (Mac) as needed. No retraining per target. The modularity of Layer 4 downstream of Layer 3 is the point."	c3::ft19::analysis
"What is the single-sentence summary of FT19's decision logic?"	"Pick the format the deployment target's runtime consumes (GGUF for local, AWQ-Marlin for NVIDIA serving, MLX for Mac), default to Q4 (~75% smaller, minimal loss), prefer sensitivity-aware per-layer quants (Dynamic 2.0/EXL2) when available, and ALWAYS benchmark the quant against the FP16 original."	c3::ft19::analysis