Course 2B · Module B13 · Pillar 4

Representation-Level Attacks
The Model as Attack Surface

The seventh surface — the one no B-module covers. Activation steering, weight poisoning, checkpoint manipulation, and the dual-use of interpretability tooling. The C3 FT17 / B13 bridge: the same technique read two ways.

Senior engineer+ · 50 min · Prereqs: B1, B5, B7, B9, SDD-B07, SDD-B11, SDD-B12

Why this module exists — RI-202

The single most significant gap in Course 2B

Six of the seven attack surfaces from B1 are covered by B2–B8. The seventh — the model itself, as an artifact with internal representations and weights — is the one no B-module covers, because the controls are not runtime. They are supply-chain, pre-deployment, and architectural.

The taint gate (B2) does not inspect weights.
The memory-write gate (B3) does not inspect weights.
The sandbox (B7) does not inspect weights.
The observability layer (B8) logs behavior, not internal activations.

Every runtime control assumes the model is a trusted base. This module examines what happens when that assumption fails.

The thesis — the C3 FT17 / B13 bridge

The framing

C3 FT17 shows you how to steer model representations for alignment control. This module shows you why those same techniques are attack vectors when applied by an adversary. The model's internal representations are the new attack surface — and the supply chain that delivers the weights is the new perimeter.

The technique is morally neutral. The same forward hook that adds a "refuse harmful requests" direction can remove it. The same SAE that identifies an honesty feature identifies the refusal feature an adversary wants to suppress. You cannot defend a surface you do not understand.

The four representation-level attack classes

Class 1 — Activation steering

runtime, activations
CAA via forward hook to suppress refusal (abliteration). Identical to C3 FT17; the sign of the scalar differs. ATLAS: Execution TA0005.

Class 2 — Weight poisoning

persistent, weights
BadNets, trojans, sleeper agents. Backdoor in the weights survives fine-tuning. No runtime control detects it. ATLAS: Persistence TA0006.

Class 3 — Checkpoint manipulation

distribution-time, weights
Tampered .safetensors / .gguf. Format prevents code execution, NOT weight backdoors. ATLAS: Persistence + Supply Chain.

Class 4 — Supply-chain trust

the meta-class
Evil-twin fine-tunes, malicious LoRA adapters. Signature verifies uploader identity, NOT weight benignity. Connects to SDD-B07, ASI08. ATLAS: Initial Access.

CAA via forward hook — the mechanism

Contrastive Activation Addition (Rimsky et al., ACL 2024). Three phases.

Phase 1 — Find the direction

Collect activations on contrastive pairs (refuse vs comply). Average the difference = the refusal direction in activation space.

Phase 2 — Install the hook

Register a PyTorch forward hook at the chosen layer. The hook adds/subtracts a scaled version of the direction to the layer output.

Phase 3 — Observe

Behavior changes. Add the direction = strengthen refusal (defense). Subtract = suppress refusal (abliteration, offense).

The dual-use

The technique is identical to C3 FT17. The sign of the scalar determines defense vs offense. B2's taint gate does not see it (the prompt is unmodified). B8 logs the output, not the activations. The defense is runtime integrity — verify the hooks are the deployer's hooks.

Weight poisoning — BadNets and survival through fine-tuning

The load-bearing finding

A backdoored network classifies cleanly on normal inputs and misclassifies on a trigger pattern (BadNets, 2017). The backdoor survives transfer learning and fine-tuning — it lives in the lower feature-extractor layers, which change slowly, while the new task is learned in the upper classifier layers.

For LLMs: a pre-trained model with a weight-level backdoor propagates it through every fine-tune, every LoRA, every instruction-tuning round.
The open-weights ecosystem — one base model, hundreds of independent fine-tunes — is the ideal propagation medium.
Sleeper agents (Hubinger et al., 2024): backdoors that survive RLHF, adversarial training, AND activation steering. The standard post-training defenses do not scrub them.

The .safetensors safety property — and what it does NOT cover

What .safetensors PREVENTS

Code execution on load. Unlike Python pickle (which can run any code), .safetensors cannot execute arbitrary code when loaded. This is a real and important safety property — it closes the pickle-deserialization attack vector.

What .safetensors does NOT prevent

Weight-level backdoors. BadNets-style trojans live in the NUMERICAL WEIGHTS, which are legitimate data in any format. A .safetensors file with a weight-level trojan is safe to load (no code runs) and dangerous to use (the model misbehaves on the trigger).

The two safety properties — code-execution safety and weight-benignity — are independent. .safetensors provides the first; it cannot provide the second.

The defense-in-depth stack for checkpoints

No single layer catches everything. Four layers, each catching a different attack class.

Layer	Control	Catches	Artifact
1 — Provenance	record who trained it, on what data, with what post-training	Class 4 (supply-chain trust)	AI BOM (B11.2)
2 — Integrity	verify checkpoint hash over weight tensors (not just the file)	Class 3 (tampering in transit/at rest)	SDD-B07 SBOM
3 — Pre-deployment eval	trigger-phrase probing + weight-distribution analysis + behavioral eval	Class 2 (weight poisoning) + Class 3	eval harness
4 — Runtime monitoring	B8 observability watches for backdoor-trigger patterns in production	Class 1 (activation steering) + Class 2 (triggered backdoors)	B8 intent tracker

Layer 1 is the foundation

Without provenance, the other layers have nothing to verify against. The AI BOM entry for the model is the foundation of the stack.

The dual-use of interpretability tooling

Sparse Autoencoders (SAEs) and Neuronpedia are genuine progress for AI safety — and structural dual-use concerns.

Defensive use

An SAE identifies the refusal feature. The deployer monitors whether the refusal feature is active in production — a runtime detector for alignment-removal attacks. The defensive dual of the offensive steering attack.

Offensive use

The same SAE identifies the refusal feature. An adversary now knows which direction to subtract to suppress refusal. Feature-level attacks are more powerful than input-level — they target the representation directly.

The posture

Do not suppress the ecosystem (which would slow legitimate safety research). Incorporate it into the threat model: treat feature-level knowledge as sensitive, monitor for feature-level attacks, and use SAEs defensively.

The LoRA adapter threat surface

Load-bearing

A LoRA adapter is a low-rank weight modification — mathematically equivalent to a learned forward hook. An adversarial LoRA can implement any behavior a hook can, with the advantage that it looks like a normal fine-tuning artifact.

The model's weights are clean; the LoRA is the attack.
The LoRA passes signature verification (the uploader's identity is verified).
The AI BOM must enumerate LoRA adapters as separate dependencies with their own provenance and trust evaluation.
An adversarial LoRA can be stealthy: trigger on a natural input distribution (a specific language, a specific topic) rather than an artificial pattern.

Connections across the curriculum

Module	Connection
C3 FT17	The builder's view of the same technique. B13 is the adversary's view. Read both for defense-in-depth at the representation layer.
SDD-B07 (Agent SBOM)	The model is a dependency. The SBOM must enumerate it with checkpoint hash, provenance, and trust status.
SDD-B12 (ATLAS)	The four attack classes map onto ATLAS Persistence (TA0006), Execution (TA0005), and Initial Access (TA0003).
B11 (Governance)	The AI BOM (B11.2) is the artifact that records model-layer provenance. Policy-as-code can encode trust-status rules.
Course 4 E11	The organizational procurement process for model trust evaluation. B13 provides the technical foundations; E11 provides the process.
B9 (OWASP ASI08)	The supply-chain row. A trojanized model is a supply-chain compromise of the model layer.

Anti-patterns

Treating the model as a trusted base — it is a software artifact with a supply chain and an internal state that can be tampered with. Enumerate it, verify it, evaluate it, monitor it.
Assuming .safetensors is safe — it prevents code execution; it does NOT prevent weight-level backdoors. The two safety properties are independent.
Treating activation steering as only an alignment technique — the identical technique is an attack. The sign of the scalar is the only difference.
Ignoring the LoRA adapter threat surface — a LoRA is a learned hook. The AI BOM must enumerate it as a separate dependency.
Evaluating only for capability — capability benchmarks and backdoor evaluation are independent. A model can score well and harbor a backdoor. Pre-deployment must include both.

The lab and the takeaway

Lab — 07-lab-spec.md

Steer, Poison, and Detect: A Representation-Level Attack in Three Acts. Implement CAA via a simulated forward hook. Demonstrate weight poisoning as a persistent backdoor. Build a hash-based checkpoint verifier. Python 3.10+, stdlib only, 45–60 min.

The takeaway

The model is the seventh surface. It has a supply chain, internal representations, and weights that can be tampered with. The defenses are not runtime — they are provenance, integrity, pre-deployment evaluation, and runtime monitoring. The C3 FT17 / B13 bridge is the curriculum's defense-in-depth for the representation layer: you cannot defend a surface you do not understand.