Serving Stacks

Module FT20 · Course 3 — LLM Fine-Tuning Masterclass

60 minutes · 4 sub-sections: The Decision Matrix · No-Telemetry & Air-Gap · Apple Silicon/MLX · Production Patterns

You arrive with a quantized model from FT19. This module answers: what do you run it with — and what are the rules when the deployment is sensitive?

Pillar 6 — Quantize & Deploy

Three variables, asked in order

Every serving decision collapses to three questions:

Variable	The question	Determines
1. Audience	One dev? A team? Hundreds of users?	Runtime (1 user) vs server (concurrent)
2. Hardware	CUDA? Apple Silicon? CPU? ROCm?	Which runtimes are even options
3. Telemetry posture	Open? Regulated? Air-gap?	What you may operationally run

Telemetry is the most underweighted variable — and the one Pillar 7 (FT21, FT22) builds directly on.

The decision matrix — a routing function

There is no "best" runtime. There is the runtime your variables point at.

PRODUCTION · NVIDIA · open subnet · vLLM

AIR-GAP / regulated · any HW · llama.cpp server

MAC FLEET · Apple Silicon · mlx-lm

DEV / SINGLE USER · any HW · Ollama (collapses at ~5 concurrent)

The rest — TGI, localai, SGLang, TensorRT-LLM — fills in around these four.

vLLM — the production standard

Three engineering choices made it standard on NVIDIA.

PagedAttention

KV cache in paged blocks (like OS virtual memory). Eliminates the fragmentation that wastes 60–80% of KV-cache memory — packs more sequences in the same VRAM.

Continuous batching

Insert new requests into running batches at every token step. GPUs stay saturated; tail latency bounded. 5–10× over static batching.

Tensor parallelism splits a too-big model across GPUs — a flag (--tensor-parallel-size N), not a recompile.

OpenAI-compatible API. Any OpenAI client works by changing one base_url.

vLLM's telemetry posture

vLLM is OpenTelemetry-native: it emits traces/metrics to a collector you configure. No vendor phone-home, no license check, no call-home server.

Point OTEL_EXPORTER_OTLP_ENDPOINT at your own collector on 127.0.0.1 — or leave unset and traces are no-ops.

This is why vLLM is acceptable inside sensitive subnets. The network egress is entirely yours to control.

The honest limitation: Apple Silicon / CPU support is limited. vLLM is built around CUDA. Macs want MLX.

The no-telemetry rule of thumb

llama.cpp and vLLM genuinely never need to phone home.
Ollama needs the network only for ollama pull; once models are local, block the network and bind OLLAMA_HOST=127.0.0.1.
For a true air-gap, pre-load the models, then sever the network.

Runtime	Network posture
llama.cpp	Single binary, no network code path for telemetry. Safe by construction.
vLLM	OTel-native — emits to YOUR collector. Safe by configuration.
Ollama	Needs net only for `pull`. Safe once configured + loopback bound.

The air-gap recipe

The same recipe works for any runtime:

Pre-load on a connected staging machine.
Transfer via approved media (USB, one-way diode, sneakernet).
SHA256-verify on the isolated side.
Serve with llama.cpp (preferred) or pre-loaded vLLM/Ollama, egress firewalled.
Patch on a schedule via the same media.

Air-gap means isolated, NOT invulnerable. Known CVEs still apply. An unpatched air-gapped Ollama carries the 2025 Oligo CVEs forever.

This recipe is the bridge to Pillar 7 (FT21, FT22).

Ollama — dev, not production

Best DX in the field. Privacy policy: "We don't see your prompts when you run locally." True — but three caveats:

Chat history in plaintext on disk (~/.ollama) — data-at-rest concern on fleet machines.
OLLAMA_HOST=0.0.0.0 exposes an unauthenticated API. Public-internet Ollama is routinely scraped.
Oligo found 6 CVEs (2025) — DoS, token exposure, auth bypass. Patched if you update.

The ceiling: Ollama collapses at ~5 concurrent users. Response times go from ~3s to over a minute. No PagedAttention, no real continuous batching.

The Ollama ceiling (measured)

Concurrent users	Ollama p95	vLLM p95
1	~1s	~0.6s
3	~4s	~0.8s
5 ← the ceiling	18s+	~1.2s
10+	60s+ / unusable	~2s

Same hardware, same model. The shape is the deployment decision.

Root cause: Ollama serializes requests + duplicates context per request. vLLM has PagedAttention + continuous batching. You cannot tune out the ceiling — you switch runtimes.

Apple Silicon & MLX

If your fleet is Macs, the runtime is MLX, not vLLM.

Why MLX

Apple's array framework, built for unified memory + Metal. CUDA runtimes can't exploit it.

WWDC25 session 298: Apple-endorsed local serving path.

4-bit recommendation

~75% size reduction, minimal quality loss. Pull pre-quantized from mlx-community on HF.

~143 tok/s on Qwen3-VL-4B on current M-series.

Federated local serving: MLX on EACH Mac, no central server. No central target to breach — privacy is structural.

Production deployment patterns

Pattern	The rule
Bind loopback	`127.0.0.1` by default. Expose to network only deliberately, with auth.
OpenAI-compatible API	Standardize on `/v1/chat/completions`. Swap `base_url` dev↔prod — no code change.
Concurrency + queueing	vLLM/TGI have it built in. Reverse proxy (nginx/envoy) enforces per-client limits.
Observability	OTel for vLLM, Prometheus for TGI. Metrics to YOUR collector. See the latency tail.

Unifying property: the model is one OpenAI endpoint behind your reverse proxy, metrics to your own stack. Same interface dev and prod.

Anti-patterns

Ollama for production multi-user. Collapses at ~5 concurrent users. Dev tool. The moment you have real concurrency, move to vLLM or TGI.

OLLAMA_HOST=0.0.0.0 without auth. Known, repeatedly-exploited attack surface. The Oligo CVEs make raw exposure actively dangerous. Loopback + reverse proxy.

Not benchmarking under realistic load. Single-request latency tells you nothing. vLLM's advantage and Ollama's collapse only show up under concurrency.

Ignoring telemetry posture. "Runs locally" ≠ "is private." Audit before shipping to a regulated domain. (FT21, FT22.)

What you can now do

Place each runtime on the decision matrix and justify a pick for a given target.
State the no-telemetry rule of thumb and the 5-step air-gap recipe.
Predict when Ollama collapses (~5 users) and route that workload to vLLM.
Configure a production deploy: loopback bind, OpenAI-compatible API, concurrency limits, observability to your own collector.

Next: FT21 — HIPAA and BAA Elimination