Serving Stacks
The serving runtime decision matrix and the no-telemetry/air-gap rules that gate sensitive deployments. vLLM for production, llama.cpp for air-gap, Ollama for dev (it collapses at ~5 concurrent users), MLX for Mac fleets — and the pre-load-then-sever recipe that makes any of them acceptable in a regulated subnet.
The serving choice is a routing function over three variables, not a ranking. Ask audience (one dev? team? hundreds?), then hardware (CUDA? Mac? CPU?), then telemetry posture (open? regulated? air-gap?). The runtime your variables point at: vLLM for NVIDIA production, llama.cpp for air-gap, Ollama for dev, MLX for Mac.
Ollama is a dev tool, not a production server — it collapses at ~5 concurrent users. Measured and reproducible: response times go from ~3s at one user to over a minute at five, because Ollama serializes requests and duplicates context per request (no PagedAttention, no real continuous batching). vLLM on the same hardware holds the workload at stable latency. The moment you have real concurrency, move to vLLM or TGI.
llama.cpp and vLLM genuinely never need to phone home. llama.cpp is a single binary with no network telemetry code path (safe by construction). vLLM is OpenTelemetry-native: it emits to a collector YOU configure — no vendor phone-home, no license check (safe by configuration). Ollama needs the network only for ollama pull; once local, block the network and bind OLLAMA_HOST=127.0.0.1. For a true air-gap, pre-load the models, then sever the network.
For Apple Silicon fleets, the runtime is MLX — and the architecture is federated local serving. MLX is built for unified memory (CUDA runtimes can't exploit it). 4-bit quantization (~75% reduction, minimal quality loss) from the mlx-community on HF. ~143 tok/s on Qwen3-VL-4B. Run MLX on each Mac, not one central server — there is no central target to breach, so the privacy property is structural.