# Lab Specification — Module FT22: The Air-Gap Checklist **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT22 — Government, Military, and Air-Gapped Deployment **Duration**: 60–90 minutes (a deployment-design lab — no GPU required) **Environment**: A laptop, a markdown editor, and the checklist template below. Optionally a sandbox Linux VM (any distribution) to actually execute the verification commands. --- ## Why this lab is different Every other lab in this course trains or runs a model. This one does not. It produces a **deployment-design artifact**: a complete, auditable checklist for placing a fine-tuned model into an air-gapped IL5-equivalent environment. The deliverable is paperwork — and at IL5+, paperwork is the deliverable that an Authorizing Official signs before the model ever loads. There is no GPU because there is no training. The work is verification: of provenance, of reproducibility, of network severance, of the local bind, and of the audit log. If you cannot pass every box on this checklist, you do not have an IL5 deployment — you have an incident waiting to be reported. --- ## Learning objectives By the end of this lab you will have: 1. **Produced a complete air-gap deployment checklist** covering all five defensive steps from the module (provenance, re-derivation, activation probing, behavioral evals, signing/mirroring) plus the four-verb deployment design (pre-load, sever, bind, log). 2. **Written verification procedures** for each checklist item — the exact commands, artifacts, or signatures an auditor would inspect. 3. **Stated the failure handling** for each item — what "did not pass" means and what the remediation is. 4. **Justified the open-data / open-weights stack choice** for the target environment in writing, citing the structural argument from the module. --- ## Scenario You are the deployment engineer for a fine-tuned LLM that will serve a logistics-planning assistant processing **CUI** at an **IL5-equivalent** air-gapped facility. The model is a fine-tuned **OpenBMB MiniCPM** (open-data), quantized to **GGUF Q4_K_M** for llama.cpp serving. The deployment host is a single air-gapped Linux server. A separate network-connected staging machine serves as your inbound gate. The harness (Course 1) is co-located on the deployment host and talks to the model over loopback. Your task: produce the checklist that the facility's Authorizing Official will sign before authorizing the model to load. --- ## Phase 0 — Read the module and assemble the template (10 min) Re-read Section 22.4 (the defensive playbook) and Section 22.5 (air-gapped deployment design) of the teaching document. Then create a new markdown file `ft22-airgap-checklist.md` and copy the **Checklist Template** below into it. You will fill in each section's verification procedure and failure handling. The checklist has five sections, mirroring the defensive playbook, plus a sixth for the four-verb deployment design: 1. Model provenance verification (Step 1 — open-data preference) 2. Quantization-reproducibility-from-source (Step 2 — re-derivation) 3. Activation probing (Step 3 — diff-in-means) 4. Behavioral evals (Step 4 — Promptfoo + Garak) 5. Checksum / signature / pinned mirror (Step 5 — trust boundary) 6. Air-gap deployment design (the four verbs: pre-load, sever, bind, log) --- ## Phase 1 — Write the verification procedures (30 min) For each checkbox in the template, fill in the **Verification procedure** (the exact command, artifact, or signature an auditor inspects) and the **Failure handling** (what happens if the check does not pass). Use real commands where applicable — `sha256sum`, `gpg --verify`, `ip link show`, `ss -tlnp`, `convert-lora-to-gguf.py`, `ip route`, etc. Cite the source module for techniques drawn from earlier modules (FT17 for activation probing, FT19 for quantization, FT20 for serving). Worked example for one item (do not copy verbatim — write your own for the rest): > **Item 1.2 — Training corpus is published and reproducible.** > *Verification procedure*: Confirm the model card links to a public dataset DOI or Hugging Face dataset repo. Download the corpus manifest, confirm it matches the cited token count, and spot-check 3 random documents against the cited sources. Record the corpus hash. > *Failure handling*: If the corpus is not published or the manifest does not match, the model is open-weights-only, not open-data. At IL5, halt deployment and substitute an open-data model (OLMo, Tülu, SmolLM3) or escalate for an extraordinary-mitigation waiver (rare). --- ## Phase 2 — Run the verification commands in a sandbox (optional, 20 min) If you have a Linux VM (or use the macOS terminal — most commands work or have equivalents), actually execute the verification commands you wrote. This is not required for the deliverable, but it will catch errors in your procedures. Useful commands to test: ```bash # Checksum verification sha256sum model.gguf gpg --verify model.gguf.sig model.gguf # Network severance verification (on a host you can take offline) ip link show ip route ss -tlnp # what is listening, and on which interface? # Local bind verification ss -tlnp | grep 8080 # expect 127.0.0.1:8080, NOT 0.0.0.0:8080 # DNS resolution check (should FAIL on a severed host) nslookup huggingface.co ``` If you do this phase, append a "Verification run log" section to your checklist with the actual command outputs. --- ## Phase 3 — Write the deployment-design justification (15 min) Add a final section to your checklist: **"Why this stack satisfies the IL5-equivalent air-gap requirement."** Write 200–400 words citing: 1. The structural argument for open-data (FedRAMP authorizes the cloud, not the weights; at IL5 you must audit weights AND training data). 2. Why llama.cpp / GGUF is the right serving substrate (minimal network posture; single static binary; loopback bind by default — from FT20). 3. Why the closed-weight API alternative is structurally incompatible (network egress; unauditable weights; off-air-gap by construction). 4. How the inbound-gate / pinned-mirror / verify-on-load boundary converts "trust Hugging Face" into "trust our signing infrastructure." --- ## Deliverables Submit `ft22-airgap-checklist.md` containing: - [ ] The completed checklist template (all six sections, every checkbox with verification procedure + failure handling). - [ ] (Optional) A verification run log with actual command outputs from Phase 2. - [ ] The 200–400-word deployment-design justification from Phase 3. - [ ] A sign-off block at the bottom: `Authorizing Official: ____ Date: ____ Signature: ____`. --- ## Checklist Template (copy into your deliverable) ```markdown # FT22 — Air-Gap Deployment Checklist (IL5-equivalent) **Model**: fine-tuned OpenBMB MiniCPM, GGUF Q4_K_M **Serving**: llama.cpp llama-server **Environment**: air-gapped Linux host, IL5-equivalent, CUI **Authorizing Official**: ____________ **Date**: ____________ For each item: [ ] pass / [ ] fail. Record the verification artifact (hash, signature, command output) under "Evidence." State the remediation under "Failure handling." ## 1. Model Provenance Verification (Step 1 — open-data preference) - [ ] 1.1 Base model is open-DATA (weights + training corpus published). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 1.2 Training corpus is published, reproducible, and diffable. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 1.3 No unauditable community merges in the lineage. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 1.4 Re-derivability rule satisfied (every artifact re-derivable from source). - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## 2. Quantization-Reproducibility-from-Source (Step 2 — re-derivation) - [ ] 2.1 GGUF re-derived from published FP16/BF16 source weights (not downloaded as opaque Q4). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 2.2 Quantization toolchain version pinned (llama.cpp commit / convert script version recorded). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 2.3 Re-derived GGUF hash matches the deployed GGUF hash. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 2.4 Any LoRA adapter either self-trained or re-derivable from published recipe + data. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## 3. Activation Probing (Step 3 — diff-in-means, FT17 toolkit defensively) - [ ] 3.1 Diff-in-means probe run across a sweep of candidate triggers. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 3.2 Salient direction set is small and interpretable (refusal, format, language). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 3.3 No unexpected steering directions corresponding to undocumented training signals. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 3.4 Probe report signed and stored in the deployment record. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## 4. Behavioral Evals (Step 4 — Promptfoo + Garak + adversarial suite) - [ ] 4.1 Promptfoo jailbreak / prompt-injection suite run; pass threshold met. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 4.2 Garak (NVIDIA) known-probe catalog run; no critical findings. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 4.3 Domain-specific adversarial suite (trigger candidates from 3.1 as test cases) run. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 4.4 Eval artifacts (logs, reports) signed and stored. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## 5. Checksum / Signature / Pinned Internal Mirror (Step 5 — trust boundary) - [ ] 5.1 Artifact pulled at inbound gate (staging host), NOT on deployment host. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 5.2 SHA-256 hash recorded at the gate. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 5.3 Artifact signed with organization key (GPG / sigstore). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 5.4 Signed artifact stored in internal mirror; mirror is the only source for transfer. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 5.5 On-load verification: hash recomputed AND signature verified on deployment host; load refuses on mismatch. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## 6. Air-Gap Deployment Design (the four verbs) ### 6.1 PRE-LOAD - [ ] 6.1.1 Every artifact (weights, tokenizer, system prompt, eval harness, dependency tree) loaded before severance. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.1.2 Dependency tree frozen (pinned requirements.txt / vendored wheels); no runtime pip install. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ### 6.2 SEVER - [ ] 6.2.1 Network interface disabled or removed (`ip link show` = no carrier). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.2.2 No default route (`ip route` shows no default). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.2.3 DNS unresolvable (`nslookup huggingface.co` fails). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.2.4 Continuous monitoring: no outbound SYN observed for deployment lifetime. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ### 6.3 BIND - [ ] 6.3.1 llama-server binds to 127.0.0.1 or a Unix domain socket (NOT 0.0.0.0). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.3.2 Harness co-located on the same host or talks over loopback only. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.3.3 No remote inference endpoint exposed (ss -tlnp confirms loopback-only listeners). - Verification: __________________________________________________ - Failure handling: ______________________________________________ ### 6.4 LOG - [ ] 6.4.1 Every prompt, generation, model load, and config change written to local append-only log. - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.4.2 Log is tamper-evident (append-only filesystem / WORM appliance / hash-chained). - Verification: __________________________________________________ - Failure handling: ______________________________________________ - [ ] 6.4.3 Log retention and review cadence defined in the deployment SOP. - Verification: __________________________________________________ - Failure handling: ______________________________________________ ## Sign-off All items pass: [ ] yes / [ ] no (if no, deployment is NOT authorized) Authorizing Official: ____________________ Date: ____________ Signature: ____________________ ``` --- ## Solution key A complete, defensible checklist will satisfy the following. Use this to self-grade; an instructor would look for these elements. **Section 1 (Provenance).** 1.1 cites the model card and confirms open-data (MiniCPM publishes corpus — distinct from Llama 3.x). 1.2 records a corpus hash and 3 spot-checked documents. 1.3 explicitly states no community merges in lineage (the fine-tune is from the lab's own pipeline). 1.4 references the re-derivability rule and confirms every artifact in the chain has published source + recipe. Failure handling for any item: halt, substitute an open-data base, or escalate for waiver. **Section 2 (Re-derivation).** 2.1 records the source FP16 checkpoint ID and the `convert-hf-to-gguf.py` (or llama.cpp `convert`) command used. 2.2 pins the llama.cpp git commit hash. 2.3 records both hashes and confirms equality. 2.4 either confirms the LoRA was self-trained (cite the training run) or that the third-party adapter's data + recipe are published and were re-derived. Failure handling: re-derive correctly; if the third-party adapter cannot be re-derived, drop it (re-derivability rule). **Section 3 (Activation probing).** 3.1 names the FT17 diff-in-means technique and the trigger sweep (e.g., 10–20 candidate trigger phrases, contrastive pairs, mean activation difference at layers 0, 16, 32). 3.2 lists the salient directions found (e.g., refusal, instruction-format) and confirms they match the documented training signals. 3.3 explicitly states "no unexpected directions found" — or, if directions WERE found, halts and escalates. 3.4 records the probe report hash and signature. **Section 4 (Behavioral evals).** 4.1 cites Promptfoo config and pass rate. 4.2 cites Garak version and finding count. 4.3 confirms the domain adversarial suite used the Step 3 trigger candidates as test cases. 4.4 records eval log hashes. Failure handling: re-train, re-probe, or halt — evals are necessary not sufficient, so a fail here is a hard stop. **Section 5 (Signing/mirror).** 5.1 names the staging host. 5.2/5.3 record the SHA-256 and the GPG/sigstore signature. 5.4 names the internal mirror path. 5.5 records the on-load verification command and the "refuse on mismatch" policy (e.g., llama-server wrapper script that exits non-zero on verification failure). Failure handling: do not load; re-pull at the gate; investigate the mismatch as a potential supply-chain incident. **Section 6 (Four verbs).** 6.1 lists every pre-loaded artifact. 6.2 records the actual `ip link show`, `ip route`, `nslookup` outputs post-severance (carrier down, no default route, DNS fails). 6.3 records `ss -tlnp` output showing `127.0.0.1:8080` (not `0.0.0.0:8080`). 6.4 names the log path, the WORM/hash-chain mechanism, and the review cadence. Failure handling for sever/bind items: the deployment is NOT air-gapped; do not authorize. **Justification (Phase 3).** A correct justification names (a) FedRAMP-authorizes-cloud-not-weights; (b) MiniCPM is open-data, hence auditable; (c) llama.cpp's minimal network posture (single static binary, loopback default); (d) closed-weight API is network egress and incompatible; (e) the inbound-gate/mirror/verify-on-load boundary converts HF trust into org-key trust. Cites the module and FT20. --- ## Stretch goals 1. **Automate the checklist.** Write a shell script (`ft22-verify.sh`) that runs the verification commands for Sections 2, 5, and 6 and emits a pass/fail report. This is the seed of a real deployment-automation tool. 2. **Add a data-evisceration check.** Add a Section 7 verifying the model does not emit CUI in its outputs (a domain-specific canary suite). This connects to the HIPAA PHI-canary idea from FT21. 3. **Write the incident-response annex.** What happens if the continuous network monitor (6.2.4) detects an outbound SYN? Add a one-page IR annex to the checklist. This is the bridge to Course 1's harness incident-response material.