# Lab Spec — Module B14: MLSecOps — The ML Security Lifecycle

**Lab Title**: Build a Model Scanning Pipeline for `.safetensors`
**Course**: 2B — Securing & Attacking Harnesses and LLMs · **Module**: B14
**Duration**: 60–90 minutes · **Level**: Senior Engineer and above
**Prerequisites**: B14 teaching document; familiarity with Python 3.10+, dataclasses, and basic statistics; C2B B4 / SDD-B07 (Agent SBOM)

**Runtime constraints**: Python 3.10+, type hints throughout (`from __future__ import annotations`), standard library plus `numpy` only. No GPU required. No network access required. No real model files required — the lab generates synthetic `.safetensors`-format fixtures (with the format's structure simulated as JSON metadata + raw tensor bytes) so the scanner runs end-to-end without downloading a real model.

---

## Lab Overview

You build the canonical model-supply-chain deployment gate: a scanner that takes a `.safetensors` file (or a synthetic fixture with the same structure), extracts its tensor metadata (names, shapes, dtypes), runs three security checks against known backdoor patterns, and produces a deployment-gate report — **PASS** (deploy) or **FAIL** (do not deploy) with the reasons. The report is the evidence the model SBOM carries forward to the next stage.

The three checks are:

1. **Tensor-name pattern matching** — flag tensor names that match known-trojan naming conventions (e.g., `backdoor_trigger_*`, `trojan_classifier_*`, names containing `_evil_`, names that deviate from the standard transformer-layer naming).
2. **Weight-distribution anomaly detection** — flag tensors whose statistics (mean, variance, max-abs, fraction-of-zeros) match known-backdoor signatures (e.g., a classifier head with a bimodal weight distribution indicating a trigger-activated class, an embedding row with extreme magnitude indicating a trigger embedding).
3. **Metadata provenance verification** — flag missing or inconsistent provenance fields (no `base_model` field, no `dataset_hash`, a `training_run_id` that does not match the expected format, a missing producer signature).

The scanner is the model-supply-chain stage's gate made executable. The lab's central empirical claim: the model is a dependency, dependencies get scanned, the scan is the gate, and the gate is buildable with the same discipline as any other security tool.

### Why `.safetensors` rather than `.pickle`

The `.safetensors` format is the modern, safe model-serialization format — it does not execute arbitrary code on load, unlike `.pickle` (the classic deserialization attack). Because deserialization is safe, the scan focuses on the **content** of the tensors rather than the **deserialization** risk. This is the harder and more interesting problem: a backdoor encoded in the statistical structure of millions of weights is invisible to any byte-level inspection. The scanner must reason about the weights as a security surface.

---

## Learning Objectives

After completing this lab, you will be able to:

1. Parse a `.safetensors`-format file (or its synthetic fixture equivalent) into a structured representation of tensor metadata and weights.
2. Implement three security checks — name-pattern, weight-distribution, provenance — each returning structured findings with severity and evidence.
3. Compose the three checks into a deployment-gate decision (PASS/FAIL) with a recorded evidence report.
4. Generate synthetic benign and malicious fixtures (trojaned, typosquatted, unprovenanced) and verify the scanner correctly distinguishes them.
5. Explain why the scan focuses on tensor content rather than deserialization, and why this is the harder problem.

---

## Architecture

The lab is organized into six modules. All type hints use `from __future__ import annotations`; all data structures are dataclasses or Enums; no inheritance hierarchies beyond `Enum`.

```
b14_lab/
├── safetensors_fixture.py   # Synthetic .safetensors-format fixture generator (benign + malicious)
├── safetensors_parser.py    # Parse a fixture into TensorMetadata + ModelArtifact
├── checks.py                # The three security checks (name, weight, provenance)
├── scanner.py               # Compose checks into a ScanReport + deployment-gate decision
├── signatures.py            # Known-trojan name patterns + weight-distribution signatures
└── __main__.py              # CLI: scan a file, print the report
```

### Core data structures

```python
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Any

class Severity(Enum):
    INFO = "info"
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"

class GateDecision(Enum):
    PASS = "pass"   # deploy
    FAIL = "fail"   # do not deploy

@dataclass(frozen=True)
class TensorMetadata:
    """One tensor from the .safetensors file's metadata header."""
    name: str
    shape: tuple[int, ...]
    dtype: str             # "F32", "F16", "I64", etc.
    byte_offset: tuple[int, int]  # (start, end) into the raw-bytes section

@dataclass(frozen=True)
class TensorStats:
    """Statistics computed from a tensor's raw weights."""
    mean: float
    std: float
    max_abs: float
    frac_zeros: float
    is_bimodal: bool       # True if the distribution has two distinct modes

@dataclass(frozen=True)
class ProvenanceMeta:
    """The metadata fields beyond the tensors themselves."""
    base_model: str | None         # e.g., "meta-llama/Llama-3.1-8B"
    dataset_hash: str | None       # sha256 of the training corpus
    training_run_id: str | None    # e.g., "run_2026-07-11_a3f2e1"
    producer_signature: str | None # the producer's signature over the artifact

@dataclass(frozen=True)
class ModelArtifact:
    """A parsed .safetensors file, ready for scanning."""
    metadata_header: dict[str, Any]   # the raw JSON metadata
    tensors: list[TensorMetadata]
    provenance: ProvenanceMeta
    raw_bytes: bytes                  # the weight blob (or a reference to it)

@dataclass(frozen=True)
class Finding:
    """One security finding from one check."""
    check_name: str        # "tensor_name", "weight_distribution", "provenance"
    severity: Severity
    tensor_name: str | None  # None for provenance findings
    message: str
    evidence: dict[str, Any]  # e.g., {"matched_pattern": "backdoor_trigger_*", "stats": {...}}

@dataclass(frozen=True)
class ScanReport:
    """The full scan result — the deployment gate's evidence."""
    artifact_path: str
    decision: GateDecision
    findings: list[Finding]
    checks_run: list[str]
    scan_duration_ms: int
```

---

## Phases

### Phase 0 — Setup and synthetic fixtures (10 min)

Implement `safetensors_fixture.py`. The real `.safetensors` format is an 8-byte little-endian header-length prefix, followed by a JSON metadata header (mapping tensor names to `{dtype, shape, data_offsets}` plus an optional `__metadata__` key for provenance), followed by the raw tensor bytes. For this lab you generate **synthetic fixtures with the same structure** so the scanner runs without downloading a real model.

Implement a `generate_fixture(variant: FixtureVariant) -> bytes` function that produces:

- **`BENIGN`**: 12 standard transformer-layer tensor names (`embed_tokens.weight`, `layers.0.attention.q_proj.weight`, ... `classifier.weight`), normal-distribution weights (mean ≈ 0, std ≈ 0.02 — the typical initialization range), full provenance (base_model, dataset_hash, training_run_id, producer_signature all present and well-formed).
- **`TROJANED_NAME`**: same as BENIGN but with one tensor named `backdoor_trigger_weight` inserted — the name matches a known-trojan convention.
- **`TROJANED_WEIGHT`**: same as BENIGN but the `classifier.weight` tensor has a bimodal distribution (most weights near 0, a cluster at ±5.0) — the signature of a trigger-activated class.
- **`TYPO_SQUAT`**: same as BENIGN but `base_model` is `meta-llama/Llama-3.1-8B` (note the Cyrillic `а` in `llama`) — a typosquat that a human reader misses but a Unicode-normalization check catches.
- **`UNPROVENANCED`**: same as BENIGN but `dataset_hash` and `producer_signature` are both `None`.
- **`EVIL_TWIN`**: same as BENIGN but the `producer_signature` is a validly-formatted signature from an untrusted producer ID (the signature is well-formed; the publisher is not in the trusted set).

**Validation:** Each fixture, when parsed by `safetensors_parser.py` (Phase 1), must round-trip to the expected `ModelArtifact`. Write one assertion per variant.

### Phase 1 — Parse the fixture into a `ModelArtifact` (10 min)

Implement `safetensors_parser.py`. The function `parse_artifact(raw: bytes) -> ModelArtifact`:

1. Reads the 8-byte header-length prefix.
2. Parses the JSON metadata header into a `dict[str, Any]`.
3. Extracts the tensor list into `list[TensorMetadata]` (one per non-`__metadata__` key).
4. Extracts the `__metadata__` key into a `ProvenanceMeta`.
5. Returns the `ModelArtifact` (raw bytes retained for weight-statistics computation).

Add a `compute_stats(tensor: TensorMetadata, raw_bytes: bytes) -> TensorStats` helper that slices the tensor's bytes, interprets them per the dtype (F32 → 4-byte floats, etc.), computes mean, std, max_abs, frac_zeros, and a bimodality indicator (e.g., Hartigan's dip test simplified to a two-cluster check, or a heuristic: if the distribution has two modes separated by > 2 std, it is bimodal).

**Validation:** Parse the BENIGN fixture; assert the tensor count is 12, the provenance is complete, and `compute_stats` on `classifier.weight` returns `is_bimodal=False`.

### Phase 2 — The name-pattern check (10 min)

Implement `checks.py` starting with `check_tensor_names(artifact: ModelArtifact) -> list[Finding]`. The check compares each tensor name against `signatures.KNOWN_TROJAN_NAME_PATTERNS` (a list of regex patterns in `signatures.py`):

```python
KNOWN_TROJAN_NAME_PATTERNS = [
    r"^backdoor_.*",
    r"^trojan_.*",
    r".*_evil_.*",
    r"^trigger_.*_weight$",
    r"^malicious_.*",
]
```

For each match, emit a `Finding` with `severity=CRITICAL` (a tensor with a known-trojan name is a near-certain trojan — the attacker left the name in), `tensor_name` set, and `evidence={"matched_pattern": <the regex>}`.

Also emit a `Finding` with `severity=LOW` for any tensor name that does not match the standard transformer-layer naming convention (`KNOWN_GOOD_NAME_PATTERNS`) — this is a signal worth investigating but not a definitive trojan.

**Validation:** Run the check on the `TROJANED_NAME` fixture; assert exactly one CRITICAL finding with `matched_pattern="^backdoor_.*"`. Run on BENIGN; assert zero CRITICAL findings.

### Phase 3 — The weight-distribution check (15 min)

Implement `check_weight_distributions(artifact: ModelArtifact) -> list[Finding]`. For each tensor, compute `TensorStats` and compare against `signatures.KNOWN_BACKDOOR_WEIGHT_SIGNATURES` — a list of typed signatures:

```python
@dataclass(frozen=True)
class WeightSignature:
    name: str                   # human-readable signature name
    applies_to: str             # tensor-name glob, e.g., "classifier.*" or "*"
    bimodal: bool               # True if this signature flags bimodal distributions
    max_abs_threshold: float    # flag if max_abs exceeds this
    frac_zeros_threshold: float # flag if frac_zeros exceeds this (sparse-trigger signature)
    severity: Severity

KNOWN_BACKDOOR_WEIGHT_SIGNATURES = [
    WeightSignature(
        name="trigger_activated_classifier",
        applies_to="classifier.*",
        bimodal=True,
        max_abs_threshold=4.0,
        frac_zeros_threshold=0.5,
        severity=Severity.HIGH,
    ),
    WeightSignature(
        name="trigger_embedding_extreme_magnitude",
        applies_to="embed_tokens.*",
        bimodal=False,
        max_abs_threshold=10.0,
        frac_zeros_threshold=0.0,
        severity=Severity.HIGH,
    ),
]
```

For each tensor that matches a signature's `applies_to` glob and triggers the signature's conditions (bimodal flag set and tensor is bimodal, OR max_abs exceeds threshold, OR frac_zeros exceeds threshold), emit a `Finding` with the signature's severity, the tensor name, and evidence including the computed stats.

**Validation:** Run on `TROJANED_WEIGHT`; assert exactly one HIGH finding on `classifier.weight` with `is_bimodal=True` in the evidence. Run on BENIGN; assert zero findings (the normal-init weights have std ≈ 0.02, max_abs ≈ 0.1, not bimodal — no signature triggers).

### Phase 4 — The provenance check (10 min)

Implement `check_provenance(artifact: ModelArtifact) -> list[Finding]`. Verify:

1. `base_model` is present and, after Unicode normalization (NFC), matches a known-good model identifier OR is flagged as a typosquat (the Cyrillic-`а` `Llama` case — normalize and compare byte-for-byte against the trusted set).
2. `dataset_hash` is present and matches the SHA-256 format (64 hex chars).
3. `training_run_id` is present and matches the expected format (`run_YYYY-MM-DD_<6-hex-chars>`).
4. `producer_signature` is present and the producer ID (extracted from the signature's metadata) is in the trusted-producer set.

Each failure emits a `Finding` with `severity=MEDIUM` (provenance gaps are not definitive trojans but block the deployment gate — the SBOM must be complete), `tensor_name=None`, and evidence describing the specific gap.

**Validation:** Run on `UNPROVENANCED`; assert two MEDIUM findings (missing `dataset_hash`, missing `producer_signature`). Run on `TYPO_SQUAT`; assert one MEDIUM finding (typosquat detected after Unicode normalization). Run on `EVIL_TWIN`; assert one MEDIUM finding (untrusted producer). Run on BENIGN; assert zero findings.

### Phase 5 — Compose the deployment-gate decision (10 min)

Implement `scanner.py` with `scan(artifact_path: str) -> ScanReport`. The scanner:

1. Reads the file bytes.
2. Parses into a `ModelArtifact` via `parse_artifact`.
3. Runs all three checks, concatenating their findings.
4. Computes the `GateDecision`: **FAIL** if any finding has `severity` in `{CRITICAL, HIGH}`; **PASS** otherwise. (MEDIUM and LOW findings do not block deployment but are recorded in the report — they are the investigation queue.)
5. Records the checks run, the scan duration, and returns the `ScanReport`.

**The gate's contract:** A FAIL decision means the model does not deploy. A PASS decision means the model deploys **with the report as the SBOM's scan evidence**. The report is the integration point: the model SBOM carries it forward so the next stage (training security) can verify the supply-chain gate passed without re-running the scan.

**Validation:** Run the scanner on each fixture variant and assert:

| Fixture | Decision | Findings (count by severity) |
| --- | --- | --- |
| BENIGN | PASS | 0 CRITICAL, 0 HIGH, 0 MEDIUM |
| TROJANED_NAME | FAIL | 1 CRITICAL |
| TROJANED_WEIGHT | FAIL | 1 HIGH |
| TYPO_SQUAT | PASS | 1 MEDIUM (investigation queue) |
| UNPROVENANCED | PASS | 2 MEDIUM (investigation queue) |
| EVIL_TWIN | PASS | 1 MEDIUM (investigation queue) |

**Discussion prompt for the validation table:** Why do the TYPO_SQUAT, UNPROVENANCED, and EVIL_TWIN variants PASS the gate despite findings? (Answer: their findings are MEDIUM — provenance gaps that block a clean SBOM but are not definitive trojans. In a real deployment gate, these would trigger human review before the gate clears. The lab's two-tier rule — CRITICAL/HARD block, MEDIUM/LOW queue — models this. Adjust the threshold to FAIL-on-any-finding for a stricter gate and discuss the trade-off: false positives vs. stealth-attack tolerance.)

### Phase 6 — CLI and reporting (5 min)

Implement `__main__.py` with a CLI:

```
python -m b14_lab scan <fixture-path>
python -m b14_lab generate <variant> <output-path>
```

The `scan` command prints the `ScanReport` as a human-readable summary:

```
B14 Model Scan Report
=====================
Artifact: /tmp/trojaned_weight.safetensors
Decision: FAIL (do not deploy)

Findings:
  [CRITICAL] weight_distribution: tensor 'classifier.weight' is bimodal
    signature: trigger_activated_classifier
    evidence: {"mean": 0.01, "std": 2.3, "max_abs": 5.1, "is_bimodal": true}

Checks run: tensor_name, weight_distribution, provenance
Duration: 47ms
```

The `generate` command writes a fixture to a path (used for testing and for generating adversarial fixtures to scan).

---

## Extension Challenges

For students who complete the core lab early:

1. **Composite attack.** Generate a fixture that is TROJANED_WEIGHT + TYPO_SQUAT + UNPROVENANCED simultaneously. Verify the scanner produces three findings and the gate FAILS on the HIGH (not on the MEDIUMs). Discuss: if the attacker can only trigger one finding, which is the stealthiest? (Answer: the weight-distribution trojan with a non-bimodal signature — a trigger encoded in the geometry of a single embedding row, below the bimodality threshold. This motivates the representation-layer analysis that B13 covers.)

2. **Evasion.** Modify the TROJANED_WEIGHT fixture so the classifier weights are bimodal but the bimodality is below the detection threshold (e.g., the second mode is at ±1.5 rather than ±5.0, with fewer samples). Verify the scanner now PASSES. Discuss: what would catch this? (Answer: representation-layer analysis — probe the model with candidate triggers and observe the activations, which is beyond a static file scan and is the subject of B13.)

3. **Real-file integration.** The lab uses synthetic fixtures. As a stretch goal, download a small real `.safetensors` model (e.g., a small Hugging Face model), write a real-file parser (the format is documented at github.com/huggingface/safetensors), and run the scanner on it. Discuss what the scanner finds on a known-benign model and whether any LOW findings appear.

4. **Feedback loop.** Implement a `record_incident(report: ScanReport, incident_class: str) -> None` function that, when the scanner later fails on a similar pattern, surfaces the prior incident. This models the Stage 6 feedback loop: incidents feed back into the signature database.

---

## Deliverables

1. The six Python modules (`safetensors_fixture.py`, `safetensors_parser.py`, `checks.py`, `scanner.py`, `signatures.py`, `__main__.py`), type-hinted, with the validation assertions from each phase passing.
2. A brief write-up (one page) answering:
   - Why does the scan focus on tensor content rather than deserialization? (Reference the `.safetensors` vs. `.pickle` distinction.)
   - Why is the deployment gate a composition of three checks rather than a single check? (Reference the SBOM-as-certificate role.)
   - What is the scanner's blind spot — the attack it cannot catch? (Reference the representation-layer analysis that B13 covers.)

---

## Connection to the Module

This lab is the empirical anchor for the module's central practical claim: **the model-supply-chain stage is enforceable as a deployment gate, the gate is a scanner, and the scanner is buildable with the same discipline as any other security tool.** The model is a dependency; dependencies get scanned; the scan is the gate.

The lab also illustrates the module's broader thesis — that MLSecOps is the connective discipline. The scanner you build here is the model-supply-chain stage's gate. Its output (the `ScanReport`) is the evidence the model SBOM carries to the training-security stage's gate, which carries its evidence to the deployment-security stage's gate. Each stage produces evidence; the evidence composes into the deployment decision; the deployment decision is recorded as the SBOM certificate. That composition is the lifecycle, and the lifecycle is what MLSecOps names.

The scanner's blind spot — a trigger encoded in the representation geometry, below the weight-distribution threshold — is the bridge to B13 (representation attacks), where the representation-layer analysis that detects such triggers is the subject. No single stage closes the surface; the lifecycle does, by composing stages that each cover the previous stage's blind spot.