# Teaching Script — Module FTDD-09: llama.cpp + vLLM

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FTDD-09 — llama.cpp + vLLM
**Duration**: ~45 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

This is deep-dive nine: llama.cpp and vLLM. You have spent the course building, aligning, and reasoning models. Now you face the question every fine-tuner eventually faces: how do you serve the model you just trained? The two answers every senior engineer must know are llama.cpp and vLLM. This module is about knowing which to reach for, when, and why.

[SLIDE 2 — The question every fine-tuner faces]

Here is the framing. You trained a model. How do you serve it? Two answers. llama.cpp is a single C/C++ binary with maximum hardware breadth — the air-gap default. vLLM is a Python GPU engine with maximum production throughput — the cloud default. And the most important thing to internalize up front: these are not competitors. They are complementary tools for different jobs. The deployment context decides which you reach for.

[SLIDE 3 — llama.cpp, the universal binary]

llama.cpp. It is effectively a single binary, written in C and C plus plus, with no Python runtime dependency. Its defining property is hardware breadth. The same binary runs on CPU, Apple Metal, NVIDIA CUDA, AMD ROCm, and Vulkan backends. If a machine exists, llama.cpp can probably run a model on it. When you serve with its built-in server, you get an OpenAI-compatible HTTP API on whatever port you choose, on whatever hardware you have.

It is also the engine underneath most local LLM tools. Ollama is llama.cpp with a wrapper. LM Studio is llama.cpp with a UI. So when people talk about running models locally, they are almost always talking about llama.cpp, even if they do not name it.

The long tail of deployment targets is llama.cpp's home. The developer laptop. The edge box. The air-gapped server with whatever hardware happens to be in it. The M-series Mac doing local inference.

[SLIDE 4 — vLLM, the production GPU engine]

vLLM. It is a Python-based inference engine built for one thing: maximum throughput on GPUs in a serving context. Where llama.cpp optimizes for breadth and portability, vLLM optimizes for requests per second on NVIDIA and increasingly AMD GPUs in a datacenter. Its defining properties are PagedAttention and continuous batching, which we will get to, plus multi-GPU tensor parallelism for models too large for one GPU.

It exposes an OpenAI-compatible API out of the box. Anything written against the OpenAI SDK works against vLLM with only a base-URL change. That is a big adoption advantage — you do not rewrite your application to switch from an OpenAI model to a self-hosted open model. For most cloud-deployed open-model services, vLLM is the default.

[SLIDE 5 — Two tools, two jobs, the decision]

The decision is about context, not preference. The mistake is treating them as competing options and picking a favorite. Reach for llama.cpp when the hardware is CPU, Mac, or heterogeneous; when you are air-gapped or in a sensitive domain; when you need a single self-contained binary; or when the deployment is local, edge, or desktop. Reach for vLLM when you have a GPU datacenter with concurrent users; when throughput is the constraint; when you want the OpenAI-compatible serving API; or when you are running a cloud API with real traffic.

Many teams use both. llama.cpp for the local or edge component, vLLM for the cloud service. That is not indecision — that is the correct read of two different deployment contexts.

[SLIDE 6 — Quant formats follow the server]

Now the hard constraint that catches people. The quant format you choose is downstream of the serving engine you choose. You cannot cross-serve. GGUF runs in llama.cpp. AWQ, GPTQ, and FP8 run in vLLM. The formats are tied to their engines' kernels.

GGUF is llama.cpp's native container. One file with weights, tokenizer, and metadata. Its quantization produces k-quants — Q4_K_M, Q5_K_M, Q8_0 — block-wise schemes that adapt precision to weight importance. Q4_K_M is the widely-recommended default, a good quality-to-size trade-off.

AWQ, GPTQ, and FP8 are vLLM's formats. They are GPU-native, optimized for CUDA and ROCm memory bandwidth and compute patterns. AWQ is activation-aware. GPTQ is a post-training method. FP8 is the native eight-bit float on newer NVIDIA hardware like Hopper and Blackwell.

The practical consequence. This creates two distinct export paths after fine-tuning. If you are deploying to llama.cpp, you export to GGUF. If you are deploying to vLLM, you export to AWQ or GPTQ, or serve in FP16 if you have the VRAM. A team that ships both local and cloud maintains two export artifacts from the same trained weights: a GGUF for the edge, an AWQ or FP16 for the cloud. Decide the server first, then quantize to its format.

[SLIDE 7 — Why vLLM wins production throughput]

Why does vLLM win production throughput? Two innovations. First, PagedAttention. During generation, a transformer stores the keys and values for every token it has processed — the KV cache. In a naive implementation, that cache is pre-allocated as a contiguous block per request, sized for the maximum sequence length. The problem: real requests have variable lengths, and most do not use their full allocation. The result is severe memory fragmentation and waste. PagedAttention, from the vLLM paper at SOSP twenty-twenty-three, manages the KV cache like an operating system manages virtual memory. The cache is broken into fixed-size blocks — pages — and each request's cache is a mapping allocated on demand. This eliminates fragmentation. Memory is only used for tokens actually generated. The effect is a dramatic increase in how many concurrent requests fit on a GPU.

Second, continuous batching. Traditional batching waits to fill a batch, processes it, then starts the next. If one request is much longer, the short ones finish early and their slots sit idle. Continuous batching admits new requests into the batch as soon as a slot frees — at the token level, not the batch level. The GPU stays saturated. Together with PagedAttention, there is always work and always memory for new requests.

[SLIDE 8 — The Red Hat benchmark]

Independent benchmarks reinforce this. Red Hat published a llama.cpp-versus-vLLM comparison as part of their OpenShift AI and InstructLab work. It characterizes the trade-off cleanly: llama.cpp offers broad deployability and lower-overhead single-stream performance, while vLLM delivers substantially higher aggregate throughput under concurrent multi-user load on GPUs. Neither dominates. Each wins in its regime. This matches the two-tools-two-jobs framing exactly.

[SLIDE 9 — llama.cpp, the air-gap default]

Now the air-gap story. llama.cpp makes no network calls and collects no telemetry. It is a single static binary you can drop onto a machine that has never touched the internet. This is not incidental. It is why llama.cpp is the default for air-gapped and sensitive-domain deployments. For HIPAA, government, and classified environments, a server that phones home is a non-starter. The single-binary posture satisfies the procurement and security requirements those environments impose.

vLLM's posture is also clean for self-hosting — it does not force telemetry on you. But it is a Python application with a dependency tree, which is a heavier assurance surface than a single static binary. For the strictest air-gap requirements, the single binary wins on auditability.

[SLIDE 10 — The serving tiers]

Above vLLM sits a tier of engines for the most demanding workloads. TensorRT-LLM is NVIDIA's low-latency, high-throughput engine built on TensorRT. It is the choice when you are on NVIDIA hardware, you need the absolute lowest latency, and you are willing to do the extra compilation and tuning work. SGLang is a newer engine with RadixAttention for prefix caching and structured-generation optimizations. It excels on workloads with shared prefixes — many requests sharing a long system prompt — or structured-output requirements like constrained JSON.

The tiering. llama.cpp for breadth and air-gap. vLLM for the production GPU default — start here. SGLang and TensorRT-LLM for the high end where you need to squeeze the last margins of latency and throughput. Most teams start with vLLM and only move up when they have a measured reason to.

[SLIDE 11 — Anti-patterns]

Four anti-patterns. First, choosing the quant before the server. GGUF will not run in vLLM. AWQ will not run in llama.cpp. Decide the server first, then export. Second, using llama.cpp for high-concurrency cloud serving. It is not built for the PagedAttention plus continuous-batching regime. If you have GPUs and real traffic, vLLM's advantage is large. Third, using vLLM for an air-gapped CPU-only box. vLLM is GPU-oriented. Fourth, reaching for TensorRT-LLM prematurely. Its complexity is justified only when vLLM is a measured bottleneck.

[SLIDE 12 — What you can now do]

You can now distinguish llama.cpp and vLLM by deployment context and choose correctly. You can predict which quant formats run in which engine. You can explain vLLM's throughput advantages. And you can state why llama.cpp is the air-gap default and when to reach for the high-end tier.

Next, deep-dive ten: distilabel. You have a serving stack. Now let's look at the framework for building the synthetic data that trains the models you serve.

---

*End of module FTDD-09. Duration: approximately forty-five minutes at one-hundred-forty words per minute.*