Module FTDD-09 — llama.cpp + vLLM

llama.cpp + vLLM

The serving deep dive. llama.cpp is the single-binary, air-gap-friendly, max-hardware-breadth server (GGUF). vLLM is the production GPU serving engine (PagedAttention, continuous batching, tensor parallelism, AWQ/GPTQ/FP8).

minutes

artifacts

sub-sections

Every fine-tuner eventually faces one question: how do I serve the model I just trained? The two answers every senior engineer must know are llama.cpp and vLLM. They are not competitors — they are complementary tools for different jobs. llama.cpp gives you maximum hardware breadth and air-gap deployability in a single binary. vLLM gives you maximum GPU throughput for production traffic. Pick wrong and you either cannot deploy or you cannot serve at scale.

Key Claims

Load-Bearing Claims

llama.cpp and vLLM are complementary, not competing. llama.cpp is a single C/C++ binary with no Python runtime, maximal hardware breadth (CPU, Metal, CUDA, ROCm, Vulkan), and air-gap friendliness. vLLM is a Python-based GPU serving engine built for production throughput. You choose based on deployment context, not preference.

The quant formats follow the server. llama.cpp uses GGUF (its own format, with k-quants like Q4_K_M). vLLM uses AWQ, GPTQ, and FP8 (GPU-native formats). You cannot serve a GGUF in vLLM or an AWQ in llama.cpp. The quant choice is downstream of the serving choice.

vLLM's throughput comes from PagedAttention and continuous batching. PagedAttention manages the KV cache like virtual memory, eliminating fragmentation waste. Continuous batching admits new requests mid-generation. Together they are why vLLM is the default for production GPU serving with an OpenAI-compatible API.

llama.cpp is the air-gap default. One static binary, no telemetry, no network calls, runs on whatever hardware exists. For HIPAA, government, and disconnected environments (Pillar 7), it is the standard. vLLM is the cloud/production default where GPUs and throughput matter.

After This Module

Distinguish llama.cpp and vLLM by deployment context: llama.cpp for hardware breadth and air-gap, vLLM for production GPU throughput. Choose correctly for a given environment.

Explain why quant formats follow the server: GGUF for llama.cpp, AWQ/GPTQ/FP8 for vLLM. Predict which quant you can serve in which engine.

Describe vLLM's throughput advantages (PagedAttention, continuous batching, tensor parallelism) and why they make it the production GPU default with an OpenAI-compatible API.

State the no-telemetry posture of each and why llama.cpp is the air-gap standard for sensitive domains (Pillar 7).

Artifacts

Teaching Document

~2,400 words; 4 sub-sections — two tools/two jobs, quant-format compatibility, why vLLM wins production, air-gap default + high-end tier

READ

Diagrams

4 diagrams — llama.cpp vs vLLM decision, quant-format compatibility map, PagedAttention + continuous batching, the serving tiers

READ

Slide Deck

reveal.js slide deck, dark theme, design-system teal

READ

Teaching Script