# Module B14 — MLSecOps: The ML Security Lifecycle **Course**: 2B — Securing & Attacking Harnesses and LLMs **Module**: B14 — MLSecOps: The ML Security Lifecycle **Duration**: 60 minutes **Level**: Senior Engineer and above **Prerequisites**: C3 Pillar 1 (Data Security for Fine-Tuning); C2B B3 (Memory Poisoning); C2B B4 / SDD-B07 (Tool/MCP Security & Agent SBOM); C2B B7 (Sandbox Execution & Output Handling); C2B B8 (Observability); C2B Capstone 2; C1 Module 5 (Secure Deployment); C1 Module 10 (Inference Monitoring) > *MLSecOps is not new content — it's the connective discipline that ties together C1's harness engineering, C2B's AI security, and C3's fine-tuning into a single security lifecycle. This module names the discipline and maps the existing content into it. You have already learned every stage. What you have not done is see them as one lifecycle, with a toolchain and an operating cadence that runs from data, through training, through deployment, through inference, through incident response. That is what this module provides.* --- ## Learning Objectives After completing this module, you will be able to: 1. Name the six stages of the MLSecOps lifecycle — data security, model supply chain, training security, deployment security, inference monitoring, incident response — and describe the security concern, the failure mode, and the control for each. 2. Map every stage of the MLSecOps lifecycle to existing course content: data security → C3 Pillar 1 + C2B B3; model supply chain → C2B B4 / SDD-B07 + B13 representation attacks; training security → C3 Pillars 2–5; deployment security → C2B B7 + C1 Module 5; inference monitoring → C2B B8 + C1 Module 10; incident response → C2B Capstone 2. 3. Articulate how MLSecOps extends DevSecOps for AI — the primitives that transfer (SBOM, CI/CD gates, signed artifacts, secrets management), the primitives that are new (model scanning, adversarial robustness, drift detection, prompt-injection monitoring), and why naively applying DevSecOps to ML produces gaps. 4. Evaluate the MLSecOps tool landscape — ModelScan (ProtectAI), the broader ProtectAI suite, HiddenLayer, Robust Intelligence — by the lifecycle stage each covers and the threat each addresses, and design an integrated toolchain rather than a collection of point products. 5. Build a model scanning pipeline that takes a `.safetensors` file, extracts its tensor metadata, and checks it against known backdoor patterns — the canonical deployment gate that prevents a poisoned or trojaned model from reaching production. --- ## Why this module exists This course has taught AI security in pieces. Course 1 taught harness engineering — the deterministic wrapper around a probabilistic model, with input validation, tool contracts, sandbox execution, output handling, and observability. Course 2B taught the attacks on that harness — prompt injection, memory poisoning, tool abuse, supply-chain compromise, inter-agent trust failures, the OWASP Agentic Top 10. Course 3 teaches fine-tuning, with the data-security, training-security, and evaluation disciplines that produce a model worth deploying. The pieces are complete. What is missing is the **lifecycle view** — the discipline that names the stages, defines the gates between them, specifies the toolchain, and runs the cadence. That discipline is **MLSecOps**: the application of security practices to the ML development and deployment lifecycle, analogous to how DevSecOps applied security practices to the software development lifecycle. The term appears in over thirty files across this course's corpus, always as a forward reference, never as a taught subject. This module teaches it. The load-bearing insight is the one in the lede: **MLSecOps is not new content.** Every stage maps to content you have already learned. What MLSecOps adds is the **connective tissue** — the naming, the ordering, the gates, and the operating cadence that turns the pieces into a lifecycle. A team that has learned C1, C2B, and C3 in isolation knows the controls; a team that has learned MLSecOps knows *when to apply each control, what gate it belongs in, and what tool implements it*. This is the module where the course's security curriculum becomes an operating discipline. Three sub-sections, twenty minutes each: - **B14.1 — The six stages of the MLSecOps lifecycle.** Data security, model supply chain, training security, deployment security, inference monitoring, incident response. For each: the security concern, the failure mode, the control, and the course module that teaches it. - **B14.2 — How MLSecOps extends DevSecOps.** The primitives that transfer from DevSecOps (SBOM, CI/CD gates, signed artifacts, secrets management). The primitives that are new (model scanning, adversarial robustness, drift detection, prompt-injection monitoring, representation-layer analysis). Why naively applying DevSecOps to ML produces gaps. - **B14.3 — The toolchain and the operating cadence.** The MLSecOps tools landscape — ModelScan, ProtectAI, HiddenLayer, Robust Intelligence — by lifecycle stage. Building an integrated toolchain. The deployment gate. The lab: a model scanning pipeline for `.safetensors`. --- # B14.1 — The Six Stages of the MLSecOps Lifecycle *One lifecycle, six stages, each mapping to content you have already learned. The value of MLSecOps is the naming and the ordering — the connective tissue.* ## Stage 1 — Data security **The security concern.** The model learns from data. If the data is poisoned, tampered with, or drawn from an untrusted source, the model learns the attacker's intent. Data security is the stage where you ensure the integrity, provenance, and representativeness of the training corpus. **The failure mode.** Data poisoning — the attacker injects crafted examples into the training data that produce a backdoor (the model behaves normally except when a specific trigger appears) or a bias (the model systematically favors an outcome the attacker chose). The clean-label attack is the hardest variant: every injected example is correctly labeled, so the poisoning is invisible to label-inspection defenses. The data-supply-chain attack is the structural variant: the dataset is sourced from a third party (Common Crawl, a Hugging Face dataset, a purchased corpus), and the third party is compromised or adversarial. **The control.** Data provenance — every example in the training corpus has a recorded source, and sources are classified by trust level. Chain-of-custody controls (SEI/Carnegie Mellon's framing) that track the data from collection through preprocessing to training. Dataset signing — the dataset is hashed and the hash is verified before training. Poisoning detection — statistical anomaly detection on the training distribution that flags clusters of injected examples. Data-access controls — the pipeline that writes to the training corpus is authenticated and auditable. **Maps to:** C3 Pillar 1 (Data Security for Fine-Tuning) and C2B B3 (Memory Poisoning). C3 Pillar 1 teaches the data-security practices for the fine-tuning corpus; B3 teaches the runtime analogue (memory poisoning) where the same concern applies to the agent's durable state. The MLSecOps data-security stage names both as one stage of a single lifecycle. ## Stage 2 — Model supply chain **The security concern.** Most teams do not train models from scratch. They start from a base or foundation model — often downloaded from a public hub (Hugging Face, a vendor's registry) — and fine-tune it. That base model is a supply-chain dependency, with the same risks as any third-party software dependency: it can be malicious, it can be trojaned, it can be a typosquat of a legitimate model, and the hash you verified at download time may not match the hash at load time if the registry was compromised. **The failure mode.** The trojaned base model — a model that behaves normally on standard inputs but produces attacker-chosen outputs when a trigger appears. The typosquatted model — a model whose name closely resembles a legitimate model's, published by an attacker, downloaded by a developer who mistyped. The evil-twin model — a model with a valid signature from a malicious publisher, because the signature verifies the publisher's identity, not the model's benignity (the same ASI08 supply-chain pattern from SDD-B01, now at the model layer). The representation-level backdoor — the model's internal representations encode a trigger that no input-layer inspection will find. **The control.** The model SBOM (Software Bill of Materials for the model) — an enumeration of the model's lineage: which base model, which dataset, which fine-tuning run, which evaluation results, signed by the producer and verified by the consumer. Model scanning — automated analysis of the model file for known backdoor patterns, anomalous weights, and trojan signatures (the subject of B14.3 and the lab). Provenance verification — the model is downloaded from a trusted registry, the hash is verified against a signed manifest, and the publisher's identity is authenticated. The discipline of treating a downloaded model the way you treat a downloaded dependency: with suspicion, verification, and an SBOM. **Maps to:** C2B B4 / SDD-B07 (Tool/MCP Security & Agent SBOM) and B13 representation attacks. SDD-B07 teaches the agent SBOM — the enumeration of an agent's tools and dependencies. The model SBOM is the same discipline applied to the model itself: the model is a dependency, and it gets the same provenance, scanning, and verification a tool gets. B13 (representation attacks, the forward-referenced module) will cover the representation-layer analysis that detects backdoors encoded in the model's internal geometry. ## Stage 3 — Training security **The security concern.** Training (and fine-tuning) is the stage where the model's behavior is determined. The concern is twofold: that the training process itself is secure (no unauthorized modification of the training run), and that the training produces a model robust to the threats it will face in deployment (prompt injection, adversarial inputs, distributional shift). **The failure mode.** Training-run tampering — an attacker with access to the training infrastructure modifies the training data, the hyperparameters, or the objective function, producing a model that serves the attacker's intent. Objective-function underspecification — the training optimizes for a metric that does not capture the security requirement (e.g., helpfulness without harmlessness), and the resulting model is exploitable. Insufficient adversarial training — the model is trained on benign inputs only and is brittle to adversarial perturbations at inference time. Fine-tuning as an attack surface — the fine-tuning interface (if exposed) lets an attacker submit crafted examples that backdoor the model, the same way data poisoning does but via an API. **The control.** Training-run integrity — the training infrastructure is access-controlled, the training data is hashed and verified before the run, the hyperparameters are version-controlled, and the run is reproducible from its recorded inputs. Adversarial training — the training corpus includes adversarial examples that teach the model to resist perturbations (the C3 discipline). Red-team evaluation before deployment — the trained model is evaluated against a red-team benchmark (the C2B and C1 discipline) before it is allowed to proceed to deployment. Fine-tuning interface hardening — if fine-tuning is exposed, the interface is rate-limited, authenticated, and the submitted examples are scanned for poisoning. **Maps to:** C3 Pillars 2–5. C3 Pillar 2 covers the training infrastructure, Pillar 3 the adversarial training practices, Pillar 4 the evaluation, and Pillar 5 the deployment readiness. The MLSecOps training-security stage names these as one stage and emphasizes the two concerns — training-run integrity and adversarial robustness — that span the pillars. ## Stage 4 — Deployment security **The security concern.** The model is trained and ready. Deployment is the stage where the model is packaged, served, and exposed to users (and adversaries). The concern is that the deployed system — the model plus its harness — is secure against the threats it will face in production. **The failure mode.** The un-harnessed model — a model served without an input-validation layer, a tool-contract layer, a sandbox, or an output-handling layer, directly exposed to user input. This is the failure mode the entire C1 harness-engineering curriculum exists to prevent. The misconfigured harness — the harness is present but its controls are wrong (the taint gate's threshold is too loose, the sandbox's allowlist is too broad, the output handler sanitizes for the wrong consumer). The secrets-in-the-model risk — the model has memorized training data (PII, credentials, proprietary content) and exposes it under the right prompts. **The control.** The five-layer harness from C1 — input validation, tool contracts, sandbox execution, output handling, observability — deployed as the deterministic wrapper around the probabilistic model. The C2B defenses — taint gates, instruction hierarchies, capability enumeration, signed manifests, circuit breakers — layered into the harness. Deployment as a gate, not an event: the model does not reach production until it has passed the harness checklist, the red-team evaluation, and the model scan. **Maps to:** C2B B7 (Sandbox Execution & Output Handling) and C1 Module 5 (Secure Deployment). B7 teaches the sandbox and output-handling layer of the harness; C1 Module 5 teaches the deployment discipline that assembles the harness around the model. The MLSecOps deployment-security stage names this as the gate between training and production. ## Stage 5 — Inference monitoring **The security concern.** The model is deployed. Inference monitoring is the stage where you observe the model's behavior in production and detect attacks, drift, and failures — because no model is safe at rest; safety is a property of the running system. **The failure mode.** The deployed attack — prompt injection, goal hijacking, tool abuse, the entire C2B threat surface, now observed live rather than prevented upfront. Drift — the model's input distribution shifts away from its training distribution, and its behavior degrades in ways the offline evaluation did not predict. The slow-compromise — an attacker probes the deployed system over weeks, mapping its capabilities and probing its defenses, building toward a chain (SDD-B01) that no single-turn monitor would catch. The silent failure — the model produces plausible-but-wrong outputs (hallucinations, cascading under ASI06) that no monitor flags because the outputs are structurally valid. **The control.** The observability substrate from C1 Module 10 and C2B B8 — every inference logged, every tool call recorded, every output scored for harm indicators. Anomaly detection on the inference stream — sudden changes in input patterns, output distributions, or tool-call volumes that indicate an attack in progress. Prompt-injection detection — the B2/B8 detectors that score inputs for injection signatures and flag suspicious sessions for review. Drift detection — statistical monitoring of the input distribution against the training baseline, with alerts when the drift exceeds a threshold. **Maps to:** C2B B8 (Observability & Attack Detection) and C1 Module 10 (Inference Monitoring). B8 teaches the observability substrate and the attack-detection layer; C1 Module 10 teaches the inference-monitoring practices. The MLSecOps inference-monitoring stage names this as the runtime stage of the lifecycle, the one that never ends as long as the model is in production. ## Stage 6 — Incident response **The security concern.** An attack succeeded, or a failure occurred, or a vulnerability was discovered. Incident response is the stage where you detect, contain, investigate, and remediate — and where you feed the lessons back into the earlier stages. **The failure mode.** The undetected incident — the attack succeeded and no monitor flagged it, so the incident continues until an external party reports it. The slow containment — the incident was detected but the team has no runbook, so containment takes days while the attacker exfiltrates. The repeated incident — the incident was contained but the root cause was not fed back into the lifecycle, so the same attack succeeds again next quarter. The un-attributed incident — the incident was contained but the investigation did not determine which lifecycle stage failed, so the fix is applied to the wrong stage. **The control.** Incident runbooks — for each class of incident (prompt-injection breach, data exfiltration, supply-chain compromise, model drift beyond thresholds), a documented containment procedure, a rollback path, and a communication plan. The red-team engagement as the incident-response rehearsal — the C2B Capstone 2 engagement is, among other things, a dry run of the incident-response procedure, with the red team playing the attacker and the blue team playing the responder. Root-cause mapping to the lifecycle — every incident is attributed to the stage where the control failed (was it a data-security failure? a model-supply-chain failure? a training-robustness failure? a deployment-configuration failure? a monitoring gap?), and the fix is applied to that stage. The feedback loop — incidents feed new evaluation cases, new monitoring rules, and new red-team procedures back into the earlier stages. **Maps to:** C2B Capstone 2. The Capstone engagement is the capstone because it requires the student to operate the entire lifecycle — not just run an attack, but detect it, contain it, investigate it, attribute it to a stage, and feed the lesson back. The MLSecOps incident-response stage names the Capstone as the lifecycle's culminating practice. --- # B14.2 — How MLSecOps Extends DevSecOps *The primitives that transfer, the primitives that are new, and why naively applying DevSecOps to ML produces gaps.* ## What transfers from DevSecOps MLSecOps is not a wholesale invention. The security-engineering primitives that DevSecOps developed over twenty years apply directly to the ML lifecycle, with the same names and the same shapes: - **SBOM (Software Bill of Materials).** The model SBOM is the ML analogue of the software SBOM. Both enumerate the dependencies of an artifact; both are signed by the producer and verified by the consumer; both are the substrate for supply-chain security. The agent SBOM from SDD-B07 generalizes: a software SBOM lists libraries; an agent SBOM lists tools and models; a model SBOM lists base models, datasets, and training runs. The primitive is the same. - **CI/CD gates.** A DevSecOps pipeline has gates: code does not merge without tests passing, artifacts do not deploy without scans passing. An MLSecOps pipeline has the same gates: a model does not train without data-validation passing, a model does not deploy without scanning and evaluation passing. The gate primitive transfers; the checks inside the gates are ML-specific. - **Signed artifacts.** DevSecOps signs container images and binaries; MLSecOps signs model files and datasets. The signing primitive — a producer's signature over a canonical serialization, verified by the consumer — is identical. The `.safetensors` format (the lab's subject) is a serialization that supports this. - **Secrets management.** DevSecOps manages secrets in a vault, never in code; MLSecOps does the same, with the added concern that the model itself may have memorized secrets from training data (a novel risk, but the secrets-management primitive transfers). - **Least privilege.** DevSecOps grants services the minimum permissions they need; MLSecOps does the same for training pipelines, model servers, and inference endpoints. These primitives transfer because the underlying problem is the same: an artifact is produced, transported, verified, deployed, and operated, and each step is a potential attack surface. DevSecOps solved this for software; MLSecOps applies the solution to models. ## What is new in MLSecOps Five concerns are genuinely new — they have no DevSecOps analogue because they arise from the properties of ML that software does not have: - **Model scanning.** A binary either contains a known vulnerability or it does not; a scanner checks. A model file contains millions of floating-point weights, and a backdoor may be encoded in the statistical structure of those weights — invisible to a binary scanner, detectable only by analysis that perturbs the inputs and observes the outputs, or by comparing the weight distribution against known-trojaned patterns. This is the subject of B14.3 and the lab. Tools: ModelScan (ProtectAI), the representation-analysis techniques B13 will cover. - **Adversarial robustness.** Software either has a bug or it does not; adversarial inputs exploit the bug deterministically. A model has a *robustness surface* — a continuous space of inputs, some of which the model handles correctly and some of which it does not, with no clean boundary. Adversarial robustness is a measurable property of the model, tested by perturbing inputs and measuring the change in output. This has no software analogue. - **Drift detection.** Software's behavior is fixed once deployed (modulo configuration); a model's behavior depends on its inputs, and its inputs drift over time as the world changes. Drift detection — statistical monitoring of the input distribution — is a runtime concern that DevSecOps never needed. - **Prompt-injection monitoring.** The specific attack surface of LLM-based systems — natural-language inputs that contain instructions the model follows — has no software analogue. The monitoring layer that scores inputs for injection signatures, tracks sessions for multi-turn drift, and flags suspicious patterns is ML-specific (and, more specifically, LLM-specific). - **Representation-layer analysis.** The model's internal representations — the activations of its intermediate layers — are a security surface that no software artifact has. A backdoor can be encoded in the geometry of the representation space; a memorized secret can be detected by probing the activations; a bias can be measured in the representation clusters. The analysis techniques (the subject of B13 representation attacks) are new. ## Why naively applying DevSecOps to ML produces gaps The failure mode is the team that says "we have a CI/CD pipeline, we sign our artifacts, we scan for vulnerabilities — we're doing MLSecOps." They are doing the part that transfers. They are not doing the part that is new. The gaps: - The model is deployed without a model scan, because the CI/CD pipeline's scanner is a binary vulnerability scanner that does not understand model files. - The model is deployed without adversarial-robustness evaluation, because robustness is not a property DevSecOps measures. - The model is deployed without drift detection, because the DevSecOps monitoring stack watches infrastructure metrics (CPU, memory, latency) not model-behavior metrics (input distribution, output distribution, confidence calibration). - The model is deployed without prompt-injection monitoring, because the DevSecOps logging stack records HTTP requests but does not score their content for injection signatures. - The model is deployed without representation-layer analysis, because the DevSecOps tooling has no concept of a model's internal geometry. The cure is not to abandon DevSecOps — the transferring primitives are necessary. The cure is to extend it: take the DevSecOps pipeline as the substrate, and add the ML-specific gates, scans, and monitors on top. The MLSecOps pipeline is the DevSecOps pipeline plus five new stages, one per new concern. The tools in B14.3 are the tools that implement those new stages. --- # B14.3 — The Toolchain and the Operating Cadence *The MLSecOps tools landscape, by lifecycle stage. Building an integrated toolchain rather than a collection of point products. The deployment gate. The lab.* ## The tools landscape Four vendors and tools recur in the MLSecOps landscape. Each covers a stage (or stages) of the lifecycle, and each addresses a specific threat. The discipline is to evaluate them by the stage they cover and the threat they address, and to assemble an integrated toolchain rather than buying one product and expecting it to cover the lifecycle. **ModelScan (ProtectAI).** A model-file scanner — the ML analogue of a binary vulnerability scanner. ModelScan opens a model file (`.safetensors`, `.pickle`, `.h5`, `.pt`), inspects its metadata and weights, and flags known-bad patterns: pickled Python objects that execute arbitrary code on load (the classic `.pickle` deserialization attack), suspicious tensor names, anomalous weight distributions, and signatures of known trojans. ModelScan belongs in the **model-supply-chain stage** as a deployment gate: a model does not proceed to deployment until ModelScan has passed. The lab implements a minimal ModelScan-style scanner for `.safetensors`. **ProtectAI suite (beyond ModelScan).** ProtectAI's broader portfolio covers the training and deployment stages: ray-based adversarial robustness testing (perturb inputs, measure output change), prompt-injection detection for LLMs (score inputs for injection signatures — the B2/B8 discipline, now as a tool), and model-monitoring for production. The ProtectAI suite is the most complete single-vendor MLSecOps toolchain, but like any single-vendor stack, it has gaps at the boundaries (data-security, incident-response) that other tools fill. **HiddenLayer.** A focused ML-security vendor whose products cover the **inference-monitoring stage** and the **training-security stage**. HiddenLayer's runtime product monitors the deployed model for adversarial inputs, drift, and extraction attempts (an attacker querying the model repeatedly to reconstruct it). Their training product detects poisoning in the training pipeline. HiddenLayer is the strongest choice when the inference-monitoring stage is the priority — for example, a high-throughput production model serving adversarial users. **Robust Intelligence (now part of Cisco).** A continuous-validation platform that covers the **training-security and deployment stages** with adversarial-robustness testing, data-poisoning detection, and model-behavior monitoring. Robust Intelligence's distinguishing property is the *continuous* framing: the platform runs adversarial tests against the model on a cadence, not just at deployment, catching drift-induced vulnerabilities that emerge after deployment. This maps directly to the MLSecOps inference-monitoring stage's concern that safety is a property of the running system. ## Building an integrated toolchain No single tool covers the lifecycle. An integrated MLSecOps toolchain assembles tools by stage: | Stage | Tools | Gate | | --- | --- | --- | | Data security | Dataset signing, provenance tracking, poisoning-detection (statistical), chain-of-custody | Training does not start until data passes | | Model supply chain | ModelScan (file scanning), model SBOM, registry verification, hash matching | Model does not deploy until scan passes | | Training security | Adversarial training (C3), red-team evaluation, training-run integrity | Model does not exit training until eval passes | | Deployment security | The C1 five-layer harness, C2B defenses, deployment-as-a-gate | Model does not serve traffic until harness checklist passes | | Inference monitoring | HiddenLayer runtime, ProtectAI monitor, drift detection, injection detection | Alerts fire on attack/drift; no auto-rollback to a worse state | | Incident response | Runbooks, red-team rehearsal (Capstone 2), root-cause mapping | Every incident produces a feedback item for an earlier stage | The integration principle: each tool produces evidence (a scan result, an evaluation score, a monitoring alert), and the evidence is the input to the next stage's gate. The model SBOM carries the evidence forward — the SBOM records that the data passed its checks, the scan passed, the evaluation passed, and the harness checklist passed, so the deployment gate can verify the full chain of evidence rather than re-checking each stage. ## The deployment gate The single most important MLSecOps practice is the **deployment gate**: the model does not reach production until a defined set of checks has passed, and the evidence is recorded in the model SBOM. The gate is not a single check; it is the composition of the stage gates: 1. Data security: the training corpus hash matches the signed manifest; poisoning detection found no anomalies above threshold. 2. Model supply chain: ModelScan passed on the model file; the base model's SBOM is verified; the publisher's signature is valid. 3. Training security: the red-team evaluation scored above the deployment threshold; adversarial-robustness testing found no critical perturbations. 4. Deployment security: the harness checklist (the C1 five layers, the C2B defenses) is complete and passing. A model that fails any of these gates does not deploy. A model that passes all four deploys with its SBOM as the certificate. This is the MLSecOps analogue of a DevSecOps deployment gate — and the reason the gate exists is the same: production is a privilege earned by evidence, not a default. ## The lab — a model scanning pipeline for `.safetensors` The lab builds the canonical model-supply-chain gate: a scanner that takes a `.safetensors` file, extracts its tensor metadata (names, shapes, dtypes), and checks it against known backdoor patterns. The `.safetensors` format is the modern, safe model-serialization format (it does not execute arbitrary code on load, unlike `.pickle`), so the scan focuses on the *content* of the tensors rather than the *deserialization* risk. The lab implements three checks: (1) a tensor-name check that flags names matching known-trojan conventions; (2) a weight-distribution check that flags tensors whose statistics (mean, variance, extreme values) match known-backdoor signatures; and (3) a metadata check that flags missing or inconsistent provenance fields. The scanner produces a report — PASS (deploy) or FAIL (do not deploy) with the reasons — that is the gate's evidence. This is the empirical anchor for the module's central practical claim: the model-supply-chain stage is enforceable as a deployment gate, the gate is a scanner, and the scanner is buildable with the same discipline as any other security tool. The model is a dependency; dependencies get scanned; the scan is the gate. --- ## Anti-Patterns ### "We have a CI/CD pipeline, so we're doing MLSecOps" The pipeline covers the transferring primitives (build, sign, deploy) but not the new ones (model scan, robustness eval, drift detection, injection monitoring, representation analysis). Cure: extend the pipeline with the ML-specific gates — one per new concern. The DevSecOps pipeline is the substrate, not the complete solution. ### Deploying a model without a model scan A binary scanner does not understand model files; a model file with a trojan passes the binary scan. Cure: a model-specific scanner (ModelScan or equivalent) as a deployment gate. The model does not deploy until the scan passes. ### Treating the model as a black box in production A DevSecOps monitoring stack watches infrastructure metrics (CPU, latency) not model-behavior metrics (input distribution, output distribution, confidence). A model under attack looks healthy to the infrastructure monitor. Cure: add the model-behavior monitors — drift detection, injection detection, output-harm scoring — to the production monitoring stack. ### No incident runbooks for ML-specific incidents The team has runbooks for infrastructure incidents (outage, degradation) but not for ML incidents (prompt-injection breach, supply-chain compromise, drift beyond thresholds). When the incident occurs, containment is slow. Cure: a runbook per ML-incident class, with a containment procedure, a rollback path, and a communication plan. The Capstone 2 engagement is the rehearsal. ### No feedback loop from incidents to stages The incident was contained but the root cause was not mapped to a lifecycle stage, so the fix is applied to the wrong stage (or no stage). The same attack succeeds again. Cure: every incident is attributed to the stage where the control failed, and the fix is applied to that stage's gate. Incidents feed the lifecycle. ### Single-vendor toolchain lock-in One vendor's product covers three stages well and one stage poorly, but the team uses it for all four because it is already purchased. The poorly-covered stage is a gap. Cure: evaluate tools by stage and threat, assemble an integrated toolchain, and accept that no single vendor covers the lifecycle. Integrate via shared evidence (the SBOM) rather than via a single vendor's stack. --- ## Key Terms | Term | Definition | | --- | --- | | **MLSecOps** | The application of security practices to the ML development and deployment lifecycle; the connective discipline that names the stages, defines the gates, and runs the cadence | | **The six stages** | Data security, model supply chain, training security, deployment security, inference monitoring, incident response — the lifecycle this module names | | **Model SBOM** | The Software Bill of Materials for a model — an enumeration of its lineage (base model, dataset, training run, evaluation results), signed by the producer and verified by the consumer | | **Model scan** | Automated analysis of a model file for known backdoor patterns, anomalous weights, and trojan signatures; the ML analogue of a binary vulnerability scan; the deployment gate | | **`.safetensors`** | The modern, safe model-serialization format (does not execute arbitrary code on load, unlike `.pickle`); the format the lab scans | | **Trojaned model** | A model that behaves normally on standard inputs but produces attacker-chosen outputs when a specific trigger appears; a model-supply-chain threat | | **Clean-label attack** | A data-poisoning variant where every injected example is correctly labeled, making the poisoning invisible to label-inspection defenses | | **Drift detection** | Statistical monitoring of the deployed model's input distribution against its training baseline, with alerts when the drift exceeds a threshold | | **Deployment gate** | The composed check (data + supply-chain + training + harness) that a model must pass before reaching production, with evidence recorded in the SBOM | | **ModelScan** | ProtectAI's open-source model-file scanner; the canonical model-supply-chain gate tool | | **HiddenLayer** | An ML-security vendor focused on runtime inference monitoring and training-pipeline poisoning detection | | **Robust Intelligence** | A continuous-validation platform (now Cisco) for adversarial-robustness testing, poisoning detection, and model-behavior monitoring | --- ## Lab Exercise See `07-lab-spec.md`. You build a model scanning pipeline that takes a `.safetensors` file, extracts its tensor metadata (names, shapes, dtypes), and runs three checks — tensor-name pattern matching, weight-distribution anomaly detection, and metadata provenance verification — against known backdoor signatures. The scanner produces a deployment-gate report (PASS/FAIL with reasons). The lab is the canonical model-supply-chain gate in code. --- ## References 1. **OpenSSF** — *A Practical Guide for Building Robust AI/ML Pipeline Security (MLSecOps Whitepaper, 2025).* `openssf.org/wp-content/uploads/2025/08/OpenSSF_MLSecOps_Whitepaper.pdf`. The industry-consolidated MLSecOps best-practices reference. 2. **C3 Pillar 1 — Data Security for Fine-Tuning.** The data-security stage's primary content. 3. **C2B B3 — Memory Poisoning Defense.** The runtime analogue of data poisoning; the data-security stage's second source. 4. **C2B B4 / SDD-B07 — Tool/MCP Security & Agent SBOM.** The agent SBOM that the model SBOM generalizes from; the model-supply-chain stage's framework. 5. **B13 — Representation Attacks (forward-referenced).** The representation-layer analysis that detects backdoors encoded in the model's internal geometry; the model-supply-chain stage's advanced content. 6. **C3 Pillars 2–5 — Training infrastructure, adversarial training, evaluation, deployment readiness.** The training-security stage's content. 7. **C2B B7 — Sandbox Execution & Output Handling.** The deployment-security stage's sandbox and output-handling layer. 8. **C1 Module 5 — Secure Deployment.** The deployment discipline that assembles the harness around the model. 9. **C2B B8 — Observability & Attack Detection.** The inference-monitoring stage's observability substrate. 10. **C1 Module 10 — Inference Monitoring.** The inference-monitoring stage's monitoring practices. 11. **C2B Capstone 2 — Run a Full Agent Red-Team Engagement.** The incident-response stage's rehearsal; the lifecycle's culminating practice. 12. **SEI / Carnegie Mellon** — *Data Poisoning in AI Models: The Case for Chain of Custody Controls.* The chain-of-custody framing the data-security stage inherits. 13. **ProtectAI ModelScan** — `github.com/protectai/model-scan`. The canonical model-file scanner; the lab's reference implementation.