{
  "module": "FT20 — Serving Stacks",
  "course": "3 — LLM Fine-Tuning Masterclass",
  "version": "1.0.0",
  "duration_minutes": 40,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What are the three variables in the serving-stack decision matrix, and in what order do you ask them?",
      "options": [
        "Hardware first, then telemetry, then audience — hardware is always the binding constraint.",
        "Audience first, then hardware, then telemetry posture. Telemetry is the most underweighted and the one Pillar 7 builds on.",
        "Telemetry first, then audience, then hardware — sensitive domains force the runtime choice.",
        "Budget, then hardware, then latency SLO — cost drives everything."
      ],
      "answer_index": 1,
      "rationale": "Ask audience (1 user? team? hundreds?) first because it determines runtime-vs-server. Then hardware (CUDA/Mac/CPU/ROCm) because it narrows the options. Then telemetry posture (open/regulated/air-gap) because it is the most underweighted variable and the one Pillar 7 (FT21-FT22) builds on directly."
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "State the no-telemetry rule of thumb from this module.",
      "options": [
        "All local runtimes are private by construction — if it runs locally, it cannot exfiltrate data.",
        "No serving runtime is safe for regulated domains; you must always deploy behind a VPN.",
        "llama.cpp and vLLM genuinely never need to phone home. Ollama needs the network only for `ollama pull`; once models are local, block the network and bind OLLAMA_HOST=127.0.0.1. For a true air-gap, pre-load the models, then sever the network.",
        "Ollama is the safest runtime because its privacy policy says it never sees your prompts."
      ],
      "answer_index": 2,
      "rationale": "The rule of thumb: llama.cpp (single binary, no network code path for telemetry) and vLLM (OTel-native, emits to YOUR collector, no vendor phone-home) never need to phone home. Ollama needs the network only for `ollama pull`; then block the network and bind loopback. For a true air-gap, pre-load then sever. Option A is the cardinal misconception — 'runs locally' is not equivalent to 'is private.'"
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is vLLM's telemetry posture, and why does it matter for sensitive deployments?",
      "options": [
        "vLLM phones home to verify the license on every request — unsuitable for air-gaps.",
        "vLLM uploads anonymous usage metrics to a vendor dashboard unless you opt out.",
        "vLLM is OpenTelemetry-native: it emits traces/metrics via standard OTel env vars to a collector YOU configure. No vendor phone-home, no license check. Point exporters at your own collector (127.0.0.1) or leave them unset and traces are no-ops.",
        "vLLM has no observability hooks at all; you must add them via sidecar proxies."
      ],
      "answer_index": 2,
      "rationale": "vLLM is OTel-native — it emits to whatever endpoint you set via OTEL_EXPORTER_OTLP_ENDPOINT (and siblings). There is no vendor call-home, no licensing server. Point them at your own collector or leave unset and traces drop. This is the property that makes vLLM acceptable behind a strict egress firewall — the egress story is entirely yours."
    },
    {
      "id": "Q04", "bloom": "application", "type": "multiple_choice",
      "prompt": "You are deploying a model for an internal 30-person engineering team with bursty concurrent use on an NVIDIA GPU box. Which runtime, and what goes in front of it?",
      "options": [
        "Ollama directly exposed on 0.0.0.0 — simplest, and 30 users is small.",
        "vLLM behind a reverse proxy (nginx/envoy) enforcing auth, rate limits, and a bounded queue. vLLM's continuous batching handles concurrency; the proxy handles per-client limits and timeouts.",
        "llama.cpp server with no proxy — single binary, minimal footprint.",
        "mlx-lm on a Mac mini shared by the team via SSH tunnels."
      ],
      "answer_index": 1,
      "rationale": "30 concurrent users is well past Ollama's ~5-user ceiling, so vLLM (PagedAttention + continuous batching) is the runtime. A reverse proxy in front adds the missing pieces: auth (vLLM has none by default), per-client rate limits, timeouts, and a bounded queue. llama.cpp is the air-gap choice, not the concurrency choice. mlx-lm is Mac-only."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "A healthcare startup wants patient-facing LLM chat on a fleet of 500 Mac workstations, where patient data cannot leave the device. What is the serving architecture?",
      "options": [
        "One central vLLM server on an NVIDIA box, accessed by all 500 Macs over the hospital VPN.",
        "Ollama on each Mac, with chat history synced to a central encrypted store.",
        "Federated local serving: mlx-lm on EACH workstation, running 4-bit quantized models from mlx-community. No central server holds anyone's prompts; the privacy property is structural.",
        "llama.cpp server on a single air-gapped Linux box, with Macs as thin clients."
      ],
      "answer_index": 2,
      "rationale": "For a Mac fleet where data cannot leave the device, the answer is federated local serving via mlx-lm on each workstation (the Mac-native runtime built for unified memory). No central server holds prompts — the privacy is structural, not a policy. A central vLLM box (option A) centralizes everyone's patient data into one compliance target. Ollama (option B) stores history in plaintext and collapses under concurrency."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your app currently calls OpenAI's API via the OpenAI Python client. You want to switch to a local vLLM server in dev and a vLLM cluster in prod. What code change is required?",
      "options": [
        "Rewrite the client to use vLLM's proprietary SDK and switch back for prod.",
        "Change the `base_url` from OpenAI's endpoint to the local vLLM endpoint (e.g., http://127.0.0.1:8000/v1) and, if needed, the model name. No other client code changes — vLLM speaks the same /v1/chat/completions dialect.",
        "Vendor-lock to vLLM by importing vllm directly into the app process.",
        "Add a translation layer that converts OpenAI requests to vLLM's internal format."
      ],
      "answer_index": 1,
      "rationale": "vLLM (and llama.cpp, Ollama, TGI, MLX, SGLang) all speak the OpenAI-compatible /v1/chat/completions dialect. The existing OpenAI client works unchanged — you only swap `base_url` (and possibly the model name). This is the single highest-leverage integration decision in the serving layer: dev and prod differ by one config value, no code change."
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "You need to deploy a model into a true air-gap (no network egress ever). What is the recipe?",
      "options": [
        "Install Ollama on the air-gapped machine and run `ollama pull` — it will work because Ollama is local.",
        "Pre-load the model on a connected staging machine, transfer via approved media (USB/diode) with SHA256 verification on the isolated side, serve with llama.cpp server (preferred) or pre-loaded vLLM/Ollama with egress firewalled, and patch on a schedule via the same media.",
        "Stand up a vLLM server and rely on its OTel collector to handle the isolation.",
        "Use mlx-lm because Macs are inherently air-gapped."
      ],
      "answer_index": 1,
      "rationale": "The air-gap recipe: (1) pre-load on a connected staging machine, (2) transfer via approved media with checksum verification, (3) serve with llama.cpp server (single binary, no network deps) preferred, (4) firewall egress on any runtime, (5) patch on a schedule via the same media. `ollama pull` (option A) requires network — impossible in a true air-gap. Air-gap means isolated, not invulnerable — known CVEs still apply, so patching cadence matters."
    },
    {
      "id": "Q08", "bloom": "application", "type": "multiple_choice",
      "prompt": "You want to use SGLang instead of vLLM for an agentic workload where every request begins with the same 4KB system prompt. What is SGLang's relevant advantage?",
      "options": [
        "SGLang has better Apple Silicon support than vLLM.",
        "RadixAttention caches the shared 4KB prefix across requests, so the long shared system prompt is computed once and reused — an edge on agentic workloads with shared prefixes.",
        "SGLang does not emit telemetry, making it air-gap-safe by default.",
        "SGLang serializes requests faster than vLLM batches them."
      ],
      "answer_index": 1,
      "rationale": "SGLang's RadixAttention does prefix caching across requests that share prefixes. On agentic workloads where many requests share a long system prompt, the prefix is computed once and reused, giving SGLang an edge over vLLM. On workloads with all-unique prompts, vLLM and SGLang are close. SGLang is CUDA-targeted (option A wrong), emits telemetry like vLLM (option C wrong), and option D inverts the concurrency story."
    },
    {
      "id": "Q09", "bloom": "application", "type": "multiple_choice",
      "prompt": "A teammate sets `OLLAMA_HOST=0.0.0.0` 'to make it work from another machine' on a cloud VM with a public IP. What is the correct assessment?",
      "options": [
        "Fine — Ollama's privacy policy says it doesn't see your data.",
        "This is a known attack surface. The default 127.0.0.1 bind is correct; 0.0.0.0 exposes an unauthenticated API to the network, and the 2025 Oligo CVEs (including an auth bypass) make raw exposure actively dangerous. Roll back to loopback and put an authenticated reverse proxy in front if remote access is genuinely needed.",
        "Acceptable as long as the VM is behind the corporate VPN.",
        "Safe because Ollama runs as a non-root user."
      ],
      "answer_index": 1,
      "rationale": "OLLAMA_HOST=0.0.0.0 without auth is a known, repeatedly-exploited attack surface (public-internet Ollama instances are routinely scraped). The Oligo 2025 disclosure (6 CVEs including CVE-2025-63389 auth bypass in v<=0.12.3) makes raw exposure actively dangerous. The fix is to revert to 127.0.0.1 and, if remote access is genuinely required, put an authenticated reverse proxy (nginx/envoy with auth) in front."
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "At roughly how many concurrent users does Ollama collapse, and what is the root cause?",
      "options": [
        "~5 concurrent users. Ollama serializes requests and duplicates context per request — no PagedAttention, no real continuous batching — so VRAM and latency explode together. vLLM on the same hardware holds the workload at stable latency.",
        "~50 concurrent users, due to Python's GIL in the Ollama server.",
        "~100 concurrent users, due to the GGUF format's overhead.",
        "Ollama does not collapse; it scales linearly because it wraps llama.cpp."
      ],
      "answer_index": 0,
      "rationale": "The ~5 concurrent user ceiling is measured and reproducible (Towards AI's test, corroborated by Red Hat's benchmark). Root cause: Ollama serializes requests and duplicates context per request — it lacks vLLM's PagedAttention (which would pack sequences via paged KV cache) and its continuous batching (which would insert requests at every token step). Response times go from ~3s at 1 user to over a minute at 5+. This is why Ollama is a dev tool, not a production server."
    },
    {
      "id": "Q11", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does vLLM's PagedAttention enable much higher batch sizes in the same VRAM than a naive contiguous-KV-cache implementation?",
      "options": [
        "PagedAttention compresses the KV cache with a learned codec.",
        "PagedAttention stores the KV cache in paged blocks (like OS virtual memory) rather than a contiguous tensor, eliminating the fragmentation that wastes 60-80% of KV-cache memory in naive implementations. The reclaimed memory holds more sequences.",
        "PagedAttention offloads the KV cache to CPU RAM automatically.",
        "PagedAttention skips storing KV for tokens the model considers unimportant."
      ],
      "answer_index": 1,
      "rationale": "A naive contiguous KV cache pre-allocates a max-length block per sequence; because sequences vary in length, 60-80% of that allocation is wasted to fragmentation. PagedAttention stores the KV cache in fixed-size pages (the OS-virtual-memory idea applied to attention), so sequences only use the pages they need and freed pages are immediately reusable. The reclaimed VRAM holds more concurrent sequences — hence higher batch sizes."
    },
    {
      "id": "Q12", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why is 'it runs locally' NOT equivalent to 'it is private' — what three concrete concerns does this module raise about Ollama specifically?",
      "options": [
        "Ollama uploads telemetry, stores prompts in the cloud, and requires a network connection at runtime.",
        "Ollama runs slowly, uses too much RAM, and crashes frequently.",
        "(1) Chat history is stored in plaintext on disk (~/.ollama) — a data-at-rest concern. (2) Misconfigured OLLAMA_HOST=0.0.0.0 exposes an unauthenticated API to the network. (3) Oligo found 6 CVEs in 2025 — patched if updated, but an unpatched air-gapped Ollama carries known vulnerabilities.",
        "Ollama's model registry tracks every download and correlates it to your IP."
      ],
      "answer_index": 2,
      "rationale": "Ollama's policy ('we don't see your prompts when you run locally') is true for prompt exfiltration but does not cover the three caveats: plaintext local chat history (data-at-rest), the OLLAMA_HOST exposure risk (network), and the 2025 Oligo CVEs (known vulnerabilities). 'Runs locally' is necessary but not sufficient for privacy — the telemetry posture is a first-class variable to audit, not an assumption."
    },
    {
      "id": "Q13", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why is llama.cpp server described as the 'air-gap champion,' while vLLM is described as 'acceptable in air-gaps with configuration'? What is the structural difference?",
      "options": [
        "llama.cpp is faster than vLLM on CPU, so it is preferred for air-gaps.",
        "llama.cpp is a single statically-linked binary with no network code path for telemetry (nothing to disable); vLLM is a Python runtime that emits OTel traces to whatever endpoint you configure (you must correctly point exporters at your own collector or leave them unset). Both work, but llama.cpp is safe by construction while vLLM is safe by configuration.",
        "llama.cpp does not support CUDA, so it cannot accidentally reach the internet.",
        "vLLM requires a GPU and air-gaps do not have GPUs."
      ],
      "answer_index": 1,
      "rationale": "The structural difference: llama.cpp has no network telemetry code path at all — it is a single binary that reads a local file and serves, provably by packet capture. vLLM is OTel-native: it emits traces to whatever endpoint you set. If you configure that endpoint correctly (your own collector on 127.0.0.1, or unset), vLLM is air-gap-safe; if you misconfigure it, you can exfiltrate data. Both are usable, but llama.cpp is safe-by-construction and vLLM is safe-by-configuration."
    },
    {
      "id": "Q14", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "An engineer says: 'We benchmarked Ollama on one developer's laptop and it was fast, so it's production-ready.' What is wrong with this reasoning, and what should they have done?",
      "options": [
        "Nothing — single-user laptop benchmarks are a valid proxy for production.",
        "They benchmarked on the wrong OS; Linux benchmarks would have shown the production ceiling.",
        "Single-request latency tells you nothing about behavior under concurrent load. Ollama's defining failure (the ~5-user collapse) only appears under realistic concurrency. They should benchmark under their actual concurrent user count, prompt-length distribution, and output-length distribution before declaring any runtime production-ready.",
        "They should have used a bigger GPU; the laptop was the problem."
      ],
      "answer_index": 2,
      "rationale": "This is the 'not benchmarking under realistic load' anti-pattern. Ollama's collapse at ~5 concurrent users is invisible at N=1 — single-request latency looks fine. vLLM's continuous-batching advantage only shows up under load. A valid production benchmark must reproduce the realistic concurrency, prompt lengths, and output lengths; otherwise it cannot surface the runtime's defining limit. This module exists in large part to prevent this single-request fallacy."
    },
    {
      "id": "Q15", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does the federated local serving pattern (mlx-lm on each Mac, no central server) provide a STRONGER privacy guarantee than a central vLLM box behind a VPN, for a Mac fleet handling sensitive conversations?",
      "options": [
        "Because MLX encrypts prompts at rest, while vLLM does not.",
        "Because Macs are physically harder to hack than Linux servers.",
        "Because there is no central server holding anyone's prompts — the privacy property is structural (there is nothing to breach, exfiltrate, or subpoena), not a policy layered on top. A central server, even behind a VPN, is a single compliance target aggregating everyone's data.",
        "Because MLX models are smaller than vLLM models, so less data is at risk."
      ],
      "answer_index": 2,
      "rationale": "The federated pattern's privacy guarantee is structural: with MLX on each workstation, each device holds only its own user's prompts. There is no central aggregator to breach, exfiltrate, or subpoena — the attack surface and the compliance target both disappear. A central vLLM box (even behind a VPN) is a single point that aggregates everyone's sensitive data into one target. Policy-based privacy (the VPN) can fail; structural privacy (no central store) cannot fail in the same way. This is why MLX is the answer for healthcare/defense Mac fleets."
    }
  ]
}