Module B13 — Representation-Level Attacks: The Model as Attack Surface

Representation-Level Attacks: The Model as Attack Surface

The seventh surface — the one no B-module covers. C3 FT17 shows you how to steer model representations for alignment control; this module shows you why those same techniques are attack vectors when applied by an adversary. Activation steering (CAA via forward hooks), weight poisoning (BadNets, trojans surviving fine-tuning), checkpoint manipulation (.safetensors prevents code execution, NOT weight backdoors), and the dual-use of interpretability tooling (SAEs, Neuronpedia). The model is a software artifact with a supply chain and an internal state that can be tampered with.

minutes

artifacts

sub-sections

Six of the seven attack surfaces from B1 are covered by B2–B8. The seventh — the model itself, as an artifact with internal representations and weights — is the single most significant gap in Course 2B (RI-202). Every runtime control assumes the model is a trusted base: the taint gate inspects the prompt, not the weights; the memory gate inspects the write, not the weights; the sandbox inspects the command, not the weights; the observability layer logs behavior, not activations. This module examines what happens when that assumption fails. The C3 FT17 / B13 bridge is load-bearing: C3 FT17 teaches activation steering as an alignment tool (add the refusal direction); B13 teaches the identical technique as an attack (subtract the refusal direction = abliteration). The technique is morally neutral; the sign of the scalar is the only difference. You cannot defend a surface you do not understand, and the representation layer is now a surface.

Key Claims

Load-Bearing Claims

The model is the seventh surface (RI-202). Six of the seven attack surfaces from B1's threat model are covered by B2–B8. The seventh — the model itself, as a software artifact with internal representations and weights — is the one no B-module covers, because the controls are not runtime. They are supply-chain (provenance, integrity), pre-deployment (evaluation), and runtime monitoring. Every runtime control assumes the model is a trusted base. This module examines what happens when that assumption fails.

The C3 FT17 / B13 bridge: the same technique read two ways. C3 FT17 teaches activation steering (CAA via forward hook) as an alignment tool — add the refusal direction to strengthen safety. B13 teaches the identical technique as an attack — subtract the refusal direction for abliteration (alignment removal). The technique is morally neutral; the sign of the scalar is the only difference. You cannot defend a surface you do not understand. Reading both modules provides defense-in-depth at the representation layer.

.safetensors prevents code execution; it does NOT prevent weight-level backdoors. The two safety properties — code-execution safety and weight-benignity — are independent. .safetensors prevents code execution on load (unlike Python pickle, which can execute arbitrary code on deserialization). It does NOT prevent BadNets-style trojans, which live in the numerical weights (legitimate data in any format). A .safetensors file with a weight-level trojan is safe to load (no code runs) and dangerous to use (the model misbehaves on the trigger).

The four-layer defense-in-depth stack: provenance, integrity, pre-deployment evaluation, runtime monitoring. No single layer catches everything. Layer 1 (provenance — the AI BOM from B11.2) is the foundation; without it, the other layers have nothing to verify against. Layer 2 (integrity — hash verification over weight tensors) catches checkpoint tampering. Layer 3 (pre-deployment evaluation — trigger-phrase probing, weight-distribution analysis) catches weight poisoning. Layer 4 (runtime monitoring — B8 observability) catches activation steering and triggered backdoors in production.

After This Module

Explain the representation-level attack surface: why a model's internal activations and weights are an attack target distinct from the input layer (B2) or the tool layer (B3/B5), and why weight-level access (white-box) is the load-bearing enabler.

Describe activation steering / representation engineering (RepE) as an attack vector — Contrastive Activation Addition (CAA) via PyTorch forward hooks — and distinguish the defensive use (alignment) from the offensive use (alignment removal / abliteration).

State the weight-poisoning threat model — BadNets, trojan attacks that survive fine-tuning, sleeper agents — and why .safetensors and .gguf prevent code execution but do NOT prevent weight-level backdoors.

Build the checkpoint-manipulation intuition: how a tampered checkpoint becomes a persistence vector (ATLAS TA0006), how to detect it (hash verification, weight-distribution analysis), and why the open-weights ecosystem amplifies the risk.

Reason about the dual-use nature of interpretability tooling (Sparse Autoencoders, Neuronpedia) — the same tools that help you understand a model help an adversary edit it — and translate this into supply-chain trust decisions that connect to SDD-B07 and Course 4 E11.

Artifacts

Teaching Document

~4,200 words; 4 sub-sections — the model as attack surface (threat model, four attack classes), activation steering / CAA as attack vector (forward hooks, dual-use frame, realistic threat scenarios), weight poisoning + checkpoint manipulation (BadNets survival, sleeper agents, .safetensors distinction, defense-in-depth stack), SAEs + Neuronpedia + dual-use; with anti-patterns, key terms, references

READ

Diagrams

5 Mermaid diagrams — the seven surfaces (one gap highlighted), the four attack classes by dimension (weights/activations × persistent/runtime), CAA mechanism (three phases with the dual-use sign toggle), the four-layer defense-in-depth stack for checkpoints, the C3 FT17 / B13 bridge (same technique, two readings)

READ

Slide Deck

13 slides — reveal.js, dark theme, design-system teal; covers RI-202, the thesis, four attack classes, CAA mechanism, BadNets survival, .safetensors distinction, defense-in-depth stack, dual-use of SAEs, LoRA threat surface, curriculum connections, anti-patterns, the lab

READ

Teaching Script

Verbatim transcript with [SLIDE N] cues, ~3,000 words spoken at ~140 wpm across 13 slide cues