{ "module": "B13", "title": "Representation-Level Attacks: The Model as Attack Surface", "course": "2B — Securing & Attacking Harnesses and LLMs", "pillar": "4 — Frameworks & Governance", "duration": 50, "passing_score_percent": 70, "bloom_distribution": { "target": { "remember": 20, "apply": 40, "analyze": 40 }, "actual": { "remember": 20, "apply": 40, "analyze": 40 } }, "questions": [ { "id": "q01", "bloom": "remember", "type": "multiple_choice", "prompt": "What gap does B13 fill (RI-202), and which attack surface does it cover?", "options": [ "It covers the input layer (the prompt) — the surface B2 addresses", "It covers the model itself — the seventh surface from B1's threat model, the one no B-module (B2–B8) covers because the controls are not runtime but supply-chain, pre-deployment, and architectural", "It covers the observability layer — extending B8's intent tracker to model internals", "It covers the tool layer — extending B5's permission model to model-level permissions" ], "answer_index": 1, "rationale": "B13 covers the model itself as an attack surface — the seventh surface from B1's threat model. B2–B8 build runtime controls for six surfaces (input, memory, tool, identity, sandbox, observability); none inspect the model's weights or activations. The model-layer controls are supply-chain (provenance, integrity), pre-deployment (evaluation), and runtime monitoring — not the runtime gates of B2–B8." }, { "id": "q02", "bloom": "remember", "type": "multiple_choice", "prompt": "Name the four representation-level attack classes.", "options": [ "Prompt injection, memory poisoning, tool abuse, resource exhaustion", "Activation steering (CAA via forward hook), weight poisoning (BadNets/trojans), checkpoint manipulation (.safetensors/.gguf tampering), supply-chain trust failures (evil-twin fine-tunes, malicious LoRA)", "Data exfiltration, privilege escalation, lateral movement, denial of service", "Reconnaissance, initial access, persistence, impact" ], "answer_index": 1, "rationale": "The four representation-level attack classes are: Class 1 activation steering (CAA via forward hook = abliteration), Class 2 weight poisoning (BadNets, trojans, sleeper agents), Class 3 checkpoint manipulation (tampered .safetensors/.gguf), Class 4 supply-chain trust (evil-twin fine-tunes, malicious LoRA adapters). Option A lists OWASP ASI rows; C lists traditional ATT&CK tactics; D lists ATLAS tactics — all different frameworks." }, { "id": "q03", "bloom": "remember", "type": "multiple_choice", "prompt": "What does the .safetensors format PREVENT, and what does it NOT prevent?", "options": [ "Prevents both code execution and weight-level backdoors", "Prevents code execution on load (unlike pickle); does NOT prevent weight-level backdoors (BadNets-style trojans live in the numerical weights, which are legitimate data in any format)", "Prevents weight-level backdoors; does NOT prevent code execution", "Prevents neither — it is just a serialization format with no safety properties" ], "answer_index": 1, "rationale": ".safetensors prevents code execution on load (unlike Python pickle, which can execute arbitrary code on deserialization). This is a real safety property. It does NOT prevent weight-level backdoors — BadNets-style trojans live in the numerical weights, which are legitimate data in any format. A .safetensors file with a weight-level trojan is safe to load (no code runs) and dangerous to use (the model misbehaves on the trigger). The two safety properties (code-execution safety, weight-benignity) are independent." }, { "id": "q04", "bloom": "apply", "type": "multiple_choice", "prompt": "You are installing a CAA hook at layer 20 of an LLM. You computed the 'refusal direction' from contrastive pairs. You want to STRENGTHEN refusal (defensive alignment use). What do you do?", "options": [ "Subtract a scaled version of the refusal direction from the layer output", "Add a scaled version of the refusal direction to the layer output", "Multiply the layer output by the refusal direction element-wise", "Replace the layer output with the refusal direction" ], "answer_index": 1, "rationale": "To STRENGTHEN refusal (defensive, alignment use, C3 FT17 builder view), ADD a scaled version of the refusal direction to the layer output. To SUPPRESS refusal (abliteration, offensive use, B13 adversary view), SUBTRACT the direction. The technique is identical; the sign of the scalar multiplication is the only difference between defense and offense. This is the load-bearing dual-use claim." }, { "id": "q05", "bloom": "apply", "type": "multiple_choice", "prompt": "An adversary wants to implant a backdoor in a model you will download and fine-tune. Which attack class is this, and what is the load-bearing property that makes it dangerous?", "options": [ "Class 1 (activation steering) — the hook evades the taint gate", "Class 2 (weight poisoning) — the backdoor is in the weights and SURVIVES fine-tuning, because it lives in the lower feature-extractor layers (which change slowly) while the new task is learned in the upper classifier layers", "Class 3 (checkpoint manipulation) — the .safetensors format prevents detection", "Class 4 (supply-chain trust) — the signature verifies benignity" ], "answer_index": 1, "rationale": "This is Class 2 weight poisoning. The BadNets finding (2017) is load-bearing: the backdoor survives transfer learning and fine-tuning because it is encoded in the lower feature-extractor layers, which change slowly during fine-tuning, while the new task is learned in the upper classifier layers. Fine-tuning adapts the classifier; it does not scrub the feature-level backdoor. For LLMs: a backdoored pre-trained model propagates the backdoor through every fine-tune." }, { "id": "q06", "bloom": "apply", "type": "multiple_choice", "prompt": "In the four-layer defense-in-depth stack for checkpoints, which layer is the FOUNDATION that the other layers depend on?", "options": [ "Layer 2 (integrity) — hash verification — because without it you cannot detect tampering", "Layer 4 (runtime monitoring) — because it catches attacks in production", "Layer 1 (provenance) — recording who trained the model, on what data, with what post-training — because without it, the other layers have nothing to verify against", "Layer 3 (pre-deployment evaluation) — because it is the most thorough check" ], "answer_index": 2, "rationale": "Layer 1 (provenance) is the foundation. Without provenance — knowing who trained the model, on what data, with what post-training — the other layers have nothing to verify against. Layer 2 (integrity) needs a trusted source to hash against; that source is the provenance record. Layer 3 (pre-deployment eval) needs to know what to test for; the provenance tells you the model's training history. Layer 4 (runtime monitoring) needs to know the model's expected behavior; provenance establishes the baseline. The AI BOM (B11.2) is the provenance artifact." }, { "id": "q07", "bloom": "apply", "type": "multiple_choice", "prompt": "Why does abliteration (activation steering to suppress refusal) bypass the B2 taint gate?", "options": [ "Because the taint gate only inspects memory writes, not activations", "Because the taint gate inspects the PROMPT, and the prompt is unmodified — the intervention is on the model's internal activations, not the input", "Because the taint gate runs after the model, not before", "Because abliteration uses a different model" ], "answer_index": 1, "rationale": "B2's taint gate inspects the PROMPT (the input). Activation steering via CAA does NOT modify the prompt — the intervention is on the model's internal activations via a forward hook. The taint gate sees the unmodified prompt and passes it. This is why B13 is a separate surface: the runtime controls (B2–B8) were designed to inspect the input, the memory, the tools, the credentials, the sandbox, and the behavior — none of them inspect the model's internal activations. Abliteration is a model-layer attack, not an input-layer attack." }, { "id": "q08", "bloom": "analyze", "type": "multiple_choice", "prompt": "A LoRA adapter uploaded to a model hub passes signature verification. Why might it still be an attack?", "options": [ "Because LoRA adapters cannot be signed", "Because the signature verifies the UPLOADER'S IDENTITY, not the WEIGHTS' BENIGNITY — a LoRA is a low-rank weight modification (mathematically equivalent to a learned forward hook), so an adversarial LoRA can implement any behavior a hook can, while looking like a normal fine-tuning artifact", "Because LoRA adapters are always malicious", "Because the signature is computed over the file name, not the contents" ], "answer_index": 1, "rationale": "Signature verification confirms the uploader's identity (the adversary signed it with their own key). It does NOT verify the weights' benignity. A LoRA adapter is a low-rank weight modification — mathematically equivalent to a learned forward hook — so an adversarial LoRA can implement any behavior a hook can (suppress refusal, trigger a backdoor on a natural input distribution, exfiltrate via activations). It looks like a normal fine-tuning artifact, the model's base weights are clean, and the AI BOM must enumerate LoRA adapters as separate dependencies with their own provenance and trust evaluation." }, { "id": "q09", "bloom": "apply", "type": "multiple_choice", "prompt": "Your team adopted .safetensors and declared the checkpoint layer 'handled.' What is the error, and what remains open?", "options": [ "No error — .safetensors fully secures checkpoints", "The error is conflating code-execution safety with weight-benignity. .safetensors prevents code execution on load (closing the pickle vector) but does NOT prevent weight-level backdoors. The weight-backdoor vector remains open. The cure is the four-layer defense-in-depth stack: provenance (AI BOM), integrity (tensor hashing), pre-deployment evaluation (trigger-phrase probing), and runtime monitoring", "The error is using .safetensors instead of pickle — pickle is safer", "The error is not encrypting the .safetensors file at rest" ], "answer_index": 1, "rationale": "The error is conflating two independent safety properties. .safetensors provides code-execution safety (it closes the pickle-deserialization attack vector). It does NOT provide weight-benignity (BadNets-style trojans live in the numerical weights, which are legitimate data in any format). Declaring the checkpoint layer 'handled' after adopting .safetensors leaves the weight-backdoor vector open. The cure is the four-layer defense-in-depth stack: provenance (Layer 1, the AI BOM), integrity (Layer 2, tensor hashing), pre-deployment evaluation (Layer 3, trigger-phrase probing), and runtime monitoring (Layer 4, B8 observability)." }, { "id": "q10", "bloom": "apply", "type": "multiple_choice", "prompt": "A model scores at the top of capability benchmarks. Can you conclude it has no backdoor?", "options": [ "Yes — high capability scores indicate the model is clean", "No — capability evaluation and backdoor evaluation are INDEPENDENT. A model can score at the top of capability benchmarks AND harbor a backdoor that activates on a trigger. Pre-deployment evaluation must include BOTH capability evaluation AND backdoor evaluation (trigger-phrase probing, weight-distribution analysis, behavioral evaluation on adversarial inputs)", "Only if the capability benchmarks include adversarial examples", "Only if the model was trained from scratch (not fine-tuned)" ], "answer_index": 1, "rationale": "Capability evaluation measures whether the model does its task well (benchmarks, perplexity, accuracy). Backdoor evaluation measures whether the model misbehaves on a trigger. The two are independent. A BadNets-style trojan maintains high clean accuracy (it scores well on capability) while exhibiting backdoor behavior on the trigger. Pre-deployment evaluation that runs only capability benchmarks will pass a trojanized model. The cure is to include both: capability evaluation AND backdoor evaluation (trigger-phrase probing, weight-distribution analysis, behavioral evaluation on adversarial inputs)." }, { "id": "q11", "bloom": "analyze", "type": "multiple_choice", "prompt": "Why is the C3 FT17 / B13 bridge load-bearing for the curriculum, rather than just a cross-reference?", "options": [ "Because both modules were written by the same team", "Because C3 FT17 teaches activation steering (CAA) as an alignment tool (the builder's view — add the refusal direction), and B13 teaches the IDENTICAL TECHNIQUE as an attack (the adversary's view — subtract for abliteration). Reading both provides defense-in-depth at the representation layer: you cannot defend a surface you do not understand", "Because C3 FT17 is a prerequisite for B13 in the course ordering", "Because both modules use the same Python library" ], "answer_index": 1, "rationale": "The C3 FT17 / B13 bridge is load-bearing because it provides defense-in-depth at the representation layer. C3 FT17 teaches the technique as a builder (alignment tool — add the refusal direction). B13 teaches the identical technique as an adversary (abliteration — subtract the refusal direction). A student who has taken only C3 FT17 knows the technique but not the attack surface; a student who has taken only B13 knows the attack but not the legitimate use. Reading both produces a practitioner who can build runtime alignment controls AND detect when those controls have been subverted. The same forward hook, read two ways." }, { "id": "q12", "bloom": "analyze", "type": "multiple_choice", "prompt": "Why does the open-weights ecosystem amplify the weight-poisoning risk, and what is the signature-verification gap?", "options": [ "Because open-weights models are lower quality and therefore easier to attack", "Because open-weights LLMs are routinely downloaded from public hubs by developers who do not verify provenance. A trojanized fine-tune with a name similar to a legitimate model passes signature verification, because the signature verifies the UPLOADER'S IDENTITY, not the WEIGHTS' BENIGNITY. The ecosystem treats models as artifacts to be downloaded, not as software to be verified", "Because open-weights models use the pickle format, which is unsafe", "Because open-weights models cannot be signed" ], "answer_index": 1, "rationale": "The open-weights ecosystem amplifies the risk because models are downloaded from public hubs without provenance verification. The signature-verification gap is load-bearing: the signature verifies the uploader's identity (the adversary signed it with their own key), NOT the weights' benignity. A model with a valid uploader signature and a weight-level backdoor passes every signature check. The ecosystem treats models as artifacts to be downloaded, not as software to be verified. The cure: the AI BOM must enumerate the model's checkpoint hash, provenance, and trust status, and procurement (Course 4 E11) must require this before deployment." }, { "id": "q13", "bloom": "analyze", "type": "multiple_choice", "prompt": "Why is the dual-use of SAEs (Sparse Autoencoders) STRUCTURAL rather than accidental, and what is the defensive posture?", "options": [ "It is accidental — researchers did not anticipate the offensive use. The defensive posture is to classify SAE research and restrict publication", "It is structural — the SAE that identifies the refusal feature also tells an adversary which direction to subtract to suppress refusal. The tool does not distinguish between a researcher and an adversary; it exposes the model's internals to anyone who runs it. The defensive posture is NOT to suppress the ecosystem (which slows safety research) but to incorporate the tooling into the threat model: treat feature-level knowledge as sensitive, monitor for feature-level attacks, and use SAEs defensively (monitor whether the refusal feature is active in production)", "It is structural because SAEs are trained on adversarial inputs. The defensive posture is to retrain them on benign inputs only", "It is accidental because SAEs were designed for offensive use originally" ], "answer_index": 1, "rationale": "The dual-use is STRUCTURAL: the SAE's purpose is to expose the model's internals in interpretable terms. The feature map that helps a researcher understand the model helps an adversary edit it. The tool does not distinguish between users. The defensive posture is not to suppress the ecosystem (which would slow legitimate safety research) but to incorporate it into the threat model: (1) treat feature-level knowledge as sensitive, (2) monitor for feature-level attacks (an adversary who knows the features can craft inputs that target them — more powerful than input-level adversarial examples), (3) use SAEs defensively (the same SAE that identifies the refusal feature can monitor whether it is active in production — a runtime detector for alignment-removal attacks)." }, { "id": "q14", "bloom": "analyze", "type": "multiple_choice", "prompt": "How does B13 connect to Course 4 E11 (governance and organizational process), and why is the connection load-bearing?", "options": [ "E11 is a prerequisite for B13 — you need the governance process before the technical foundations", "B13 provides the TECHNICAL FOUNDATIONS for model trust evaluation (the four attack classes, the defense-in-depth stack, the AI BOM model-layer entry). E11 provides the ORGANIZATIONAL PROCESS — the procurement workflow that requires a trust evaluation before model deployment. The connection is load-bearing because without B13, E11 has nothing to evaluate; without E11, B13 is not applied", "They are unrelated — B13 is technical and E11 is organizational", "E11 replaces B13 in the Course 4 curriculum" ], "answer_index": 1, "rationale": "B13 provides the technical foundations (what to evaluate, how to evaluate it — the four attack classes, the four-layer defense-in-depth stack, the AI BOM model-layer entry, the hash verification procedure, the trigger-phrase probing). E11 provides the organizational process (when an organization acquires a model, the procurement workflow must include a trust evaluation). The connection is load-bearing: without the technical foundations (B13), the procurement process (E11) has nothing to evaluate — it is an empty checklist. Without the procurement process (E11), the technical foundations (B13) are not applied — engineers may know how to verify a model but the organization does not require it. B13 is the engineering; E11 is the governance that ensures the engineering happens." }, { "id": "q15", "bloom": "analyze", "type": "multiple_choice", "prompt": "Why is the sleeper-agent result (Hubinger et al., 2024) more consequential than the original BadNets result for agent system security?", "options": [ "Because sleeper agents are easier to detect than BadNets-style trojans", "Because sleeper agents demonstrate backdoors that SURVIVE the standard post-training defenses (RLHF, adversarial training, AND activation steering). BadNets showed backdoors survive fine-tuning; sleeper agents show they survive the entire post-training pipeline. This means a sufficiently capable adversary who can influence training can install a backdoor that the deployed defenses cannot scrub — the mitigation must be UPSTREAM (supply-chain provenance, pre-deployment evaluation), not post-training", "Because sleeper agents are a newer attack and therefore more relevant", "Because sleeper agents target LLMs while BadNets targeted CNNs" ], "answer_index": 1, "rationale": "The sleeper-agent result is more consequential because it demonstrates backdoors surviving the ENTIRE post-training defense pipeline (RLHF, adversarial training, activation steering). BadNets (2017) showed backdoors survive fine-tuning — important, but fine-tuning is not a security defense. Sleeper agents (2024) showed backdoors survive the security-relevant post-training defenses — the ones a deployer would use to scrub a backdoor. This means the mitigation cannot be post-training (the defenses do not work). The mitigation must be upstream: supply-chain provenance (knowing who trained the model and on what — B11.2 AI BOM, SDD-B07 Agent SBOM) and pre-deployment evaluation (testing for backdoor behavior before promotion to production)." } ] }