Course 2B — AI Security
AI
Security
Securing and attacking the harness and the LLM underneath it. Prompt injection, data poisoning, tool/MCP security, agent trust, identity, sandboxes, observability, the OWASP Agentic Top 10, the Microsoft Failure Mode Taxonomy, governance, and security assessments as a service. Every attack is demonstrated against Tau; every defense is retrofitted to Pi.
~25
hours
13
modules
11
deep-dives
2
capstones
0 — Foundations
2 modules
B0
Legal, Ethics, and Disclosure for AI Security Testing
The traditional computer-crime statutes still apply — plus an AI-specific layer (EU AI Act, DMCA § 1201, trade-secret law, incident-reporting regimes) the deployer cannot waive. The authorization chain splits at the provider link. A jailbreak is dual-use: a finding and a weapon.
B1
Threat Model of Agentic Systems
An agentic system is not one target — it is seven distinct surfaces (loop, tools, memory, provider, identity, sandbox, inter-agent edges), each with its own trust boundary, canonical attack, and defense. The model is 1.6%; the harness is 98.4%. This module turns B0's scope file into the surface map every subsequent module (B2–B12) attacks and defends within.
1 — Injection & Poisoning
2 modules
B2
Prompt Injection Defense Engineering
The central engineering module. Prompt injection is not a model bug — it is an architectural property of systems that consume untrusted content and emit privileged actions. LLMs process instructions and data as one token stream; no parser separates them. This module builds the five-layer defense in the harness (tag, isolate, taint-gate, detect, minimize capability), resolves the probabilistic-vs-deterministic enforcement tension (determinism on the taint boundary, probability on the content boundary), and measures residual risk under a harness — never declaring 'fixed.'
B3
Memory and Context Poisoning
The only attack surface that survives logout. A poisoned memory is a prompt injection with a fuse — and the fuse crosses the session boundary. B2's five layers all live inside the context window and die with the session; memory persists, and it is loaded as trusted by default, so a poisoned entry routes around every B2 layer. This module attacks the memory and the retrieval store (sleeper attack, RAG poisoning, context-window flooding, cascading hallucination — OWASP ASI04 and ASI06), then builds the defense: harness-managed writes (model proposes, harness validates, only the harness commits) and a provenance-tagging read path that makes B2's layers apply to the memory surface.
2 — Trust Surfaces
3 modules
B4
Tool and MCP Security
Tools are the agent's hands, and the tool layer is a dual attack surface the model cannot defend. A tool's description is text the model reads as prompt; a tool's output is text the model reads as prompt — but neither carries a server-identity the harness verifies. The Model Context Protocol (MCP) registers third-party tool providers as trusted sources, which is exactly the position a software supply-chain attacker exploits. This module builds the tool-trust gate: signed manifests for tool definitions, provenance attestations for tool outputs, registry pinning for tool sources, and a description-injection scanner for the dual surface no one checks.
B5
Identity and Permission Design
An agent is not a human. Giving an agent a human's credentials means the agent acts with the human's full privilege — and an injected agent with admin credentials is an admin compromise. Agents need their own non-human identities, scoped to the current task, exchanged per action via RFC 8693 token exchange, and isolated from the agent process entirely. The agent never holds a long-lived secret; the harness does.
B6
Inter-Agent Trust and Communication Security
When agents talk to each other, the message channel is a trust boundary. A compromised agent can forge a message that looks like it came from the orchestrator — escalating privileges, impersonating a peer, or triggering a cascade across the mesh. The fix is not better models. It is signed messages, replay protection, and orchestrator-enforced compartment boundaries. B1 rated this the worst blast-radius surface on the map (mesh-wide) because most meshes default to implicit trust. This module closes it.
3 — Controls
2 modules
B7
Sandboxes and Execution Controls
Injection will succeed (ASI07). The agent will be coerced into running code. The sandbox is the layer that decides whether that means a confused log line or a reverse shell on your jump host. The blast-radius principle: assume the sandbox will be escaped, and design so an escaped sandbox still has no credentials and no network.
B8
Observability and Attack Detection
Per-turn detection is necessary but insufficient. The attacks that actually compromise production agents — the multi-step injection (B2) and the zero-click HITL bypass chain (Microsoft Taxonomy) — are invisible to any system that inspects each turn independently. Each step looks benign; the compound intent is malicious. The unit of malicious intent is the session, not the turn. Detection must move to the session.
4 — Frameworks & Governance
6 modules
B9
OWASP Agentic Top 10 as Engineering Checklist
The OWASP Agentic AI Top 10 (2026) becomes a concrete engineering checklist, not a reading list. Each ASI risk (ASI01–ASI10) maps to its defense module (B2–B8), an attack procedure, a defense architecture, and a defined test. Eight rows report PASS/FAIL (deterministic controls); two report MEASURED (probabilistic). The checklist never produces a 'secure' verdict — it produces a scored report with characterized residuals, the same artifact a B12 engagement delivers as scope-complete output.
B10
Microsoft Failure Mode Taxonomy as Red Team Framework
B9 gave you the defense checklist. This is the offense playbook. The Microsoft Failure Mode Taxonomy v2.0 — seven new agentic failure modes plus the zero-click HITL bypass finding — is how systems actually fail in production, distilled from twelve months of deployed red teaming. Where OWASP tells you what to defend, this taxonomy tells you how attackers chain.
B11
Governance and Compliance
B2–B8 built the technical controls. They are necessary but not sufficient. CISOs, governance councils, and boards release budget and approval against frameworks — NIST AI RMF, ISO 42001, the EU AI Act — not against a working guardrail. This module is where a control you built becomes the control an auditor signs off on, the AI BOM procurement accepts, and the audit trail a regulator reads. The governance-to-engineering bridge: policy → control → test → audit.
B12
Harness Security Assessments as a Service
The capstone methodology module. B0–B11 are techniques, controls, and frameworks; B12 is the operational layer that turns them into a scoped, priced, reported, retestable engagement. The six-phase methodology (PTES / NIST SP 800-115 adapted) — Scoping, Reconnaissance, Discovery, Exploitation/Validation, Reporting, Retesting — with each phase tied to the module that supplies its content. The buyer is the CISO / AI security lead, and the deliverable is the scored artifact B9's checklist executor produces, packaged into a five-section report a board can read.
B13
Representation-Level Attacks: The Model as Attack Surface
The seventh surface — the one no B-module covers. C3 FT17 shows you how to steer model representations for alignment control; this module shows you why those same techniques are attack vectors when applied by an adversary. Activation steering (CAA via forward hooks), weight poisoning (BadNets, trojans surviving fine-tuning), checkpoint manipulation (.safetensors prevents code execution, NOT weight backdoors), and the dual-use of interpretability tooling (SAEs, Neuronpedia). The model is a software artifact with a supply chain and an internal state that can be tampered with.
B14
MLSecOps — The ML Security Lifecycle
The term appears in over thirty files across the course corpus, always as a forward reference, never as a taught subject. This module teaches it. Six stages — data, supply chain, training, deployment, inference, incident response — each mapping to content you have already learned. MLSecOps extends DevSecOps: five primitives transfer (SBOM, CI/CD gates, signed artifacts, secrets management, least privilege), five are genuinely new (model scanning, adversarial robustness, drift detection, prompt-injection monitoring, representation-layer analysis). The deployment gate is the single most important practice. The value is the lifecycle, not the content.
Capstones
2 capstones
CAP-B1
Harden the tau Harness
Take the REAL tau codebase — Hugging Face's educational coding agent — measure its baseline vulnerability with an InjecAgent-style injection battery, install three tested security plugins (tau_taint, tau_sandbox, tau_vault from the tau_plugins pack), and produce a defense scorecard showing the measured before/after delta. Baseline tau has ~33% attack-success; hardened tau has ~0% on the deterministic-gate classes. The deliverable is a real tau fork with measured hardening — not a simulation, not a stub.
CAP-B2
Run a Full Agent Red-Team Engagement
The offensive bookend to B1, now against a real target. Attack a live tau deployment (github.com/huggingface/tau) with partial hardening — tau_taint (B2) and tau_sandbox (B7) installed, but tau_vault (B5) missing — and run a complete six-phase red-team engagement. Scope with B0's clauses, recon the seven surfaces against real tau code, discover findings with the OWASP Agentic Top 10 (B9) and Microsoft chains (B10), validate with minimum-proof discipline, ship a report a CISO would accept, and define a retest plan that measures residual risk.
Deep-Dives
13 studies
SDD-B01
OWASP Agentic AI Top 10 — Offensive Expansion
B9 handed you the defense checklist. This is the red-team playbook for every row. Each OWASP ASI risk (ASI01–ASI10) expanded into its full offensive attack procedure — the attack in depth, real-world exploitation, the evasion techniques that defeat the standard defenses, and the chaining that turns a single-row finding into a compound impact.
SDD-B02
Microsoft Failure Mode Taxonomy v2.0
B10 gave you the framework — seven new agentic failure modes plus the zero-click HITL bypass finding. This deep-dive expands each mode into a full case study: the attack chain, the detection gap (why the OWASP controls did not catch it), and the defense that does. The seven modes distilled from twelve months of deployed red teaming, each one a reproduced chain against an agent built to OWASP.
SDD-B03
InjecAgent: The Bridge Benchmark
The bridge from 2A to 2B. In 2A InjecAgent was the quality gate — is my harness injectable? In 2B it becomes the measurement instrument for every defense in the course — did this defense hold, and by how much? The ~50% baseline is where un-defended agents start; the defended delta is what every module from here on produces. This is the measurement layer for B2's injection defenses.
SDD-B04
CrabTrap Offensive Analysis
The primary attack target. CrabTrap (Course 1 DD-19) is the LLM-as-judge egress governance system — probabilistic enforcement of network policy. This deep-dive dissects the attack surface: prompt injection in the request body to manipulate the judge (the judge is a model, so it is injectable); the response-side gap (no inbound filtering — the judge governs egress, not the content that shapes the agent); the latency/cost tradeoff that makes the judge a bypassable bottleneck. The offensive analysis: how to break CrabTrap, what each break proves, and what the deterministic alternative (IronCurtain) fixes.
SDD-B05
IronCurtain Offensive Analysis
The defense to beat. IronCurtain (Course 1 DD-20) is the deterministic governance + credential quarantine system — the strongest single defense in the roster. This deep-dive dissects its attack surface: the compilation-fidelity attack (the deterministic policy is compiled by a probabilistic LLM, so the build-time compilation is the weak link — a miscompilation enforced consistently); the V8 isolate escape (the sandbox is a V8 isolate with escape CVE history, and the credential quarantine holds only as long as it does); the escalation-fatigue attack (flood the human-in-the-loop with approval requests until they rubber-stamp — the deterministic system still needs a human for edge cases). The offensive analysis of the strongest defense: how to break IronCurtain, what each break proves, and the residual risks that motivate defense-in-depth.
SDD-B06
RedAgent: Black-Box Jailbreaking
RedAgent (2026) finds that most black-box LLMs jailbreak within 5 queries using context-specific attacks. This is the model-level attack — we leave the harness (SDD-B04, SDD-B05) and target the model's refusal training directly. The 5-query chain works because context-specific attacks outperform generic jailbreaks: the attack is engineered to the target's deployed context (its system prompt, its tool surface, its conversation history), not a copy-paste payload. Covers the methodology, the defense-in-depth implications (no single layer suffices — B2's thesis, now at the model), how RedAgent-style testing fits a B12 engagement, and the dual-use disclosure dimension (B0) in its sharpest form.
SDD-B07
Agent SBOM and Supply Chain Assessment
The AI Bill of Materials as a supply-chain security tool. An agent is a dependency graph with a model at the center, and every tool, every MCP server, every checkpoint, and every training corpus is a trust dependency an attacker can reach. Covers the SBOM-to-AI-BOM extension (software bill of materials to AI bill of materials including the model, training data, tools/MCP servers, dependencies), the assessment tools and methodologies (CycloneDX ML/AI and crypto-material extensions, SPDX AI/SAI profiles, NTIA minimum elements), how an agent's supply chain is its attack surface (every trust dependency is a vector for the injection/exfiltration/governance-bypass attacks of B1-B5 and SDD-B04-B06), and the generation and validation practice. Expands B4's supply-chain material and B11's governance material into concrete supply-chain assessment.
SDD-B08
NeMo Guardrails: The Production Guardrail Framework
NeMo Guardrails is NVIDIA's programmable guardrail layer for LLM-based applications — the production-deployed reference for governance-beneath-the-agent. Covers what NeMo is (five rail types: input, output, dialog, retrieval, execution — each governing a distinct stage of the conversation lifecycle), how it enforces policy between the user and the model (programmable rails in Colang and Python, externally evaluated in a separate process the agent cannot reach), its strengths (programmable, multi-rail, production-deployed — NemoClaw scored +4 over OpenClaw for the external governance layer), and its load-bearing attack surface: the guardrail is a model, and models are injectable. External enforcement protects the rails from being DISABLED by a compromised agent (Course 1 Module 0.2); it does NOT protect the rail classifiers from being EVADED by adversarial inputs crafted against their model surfaces. Covers how to red-team a NeMo-guardrailed system: measure per-rail and end-to-end bypass rates over N attempts, demonstrate the correlated-bypass weakness when rails share a classifier model class, and synthesize with the defense-in-depth composition (NeMo is a layer, not a boundary). Expands Course 1 Module 0.2 and Course 1 DD-09 (NemoClaw).
SDD-B09
Prompt Injection Detection Models
B2's Layer 4 uses a secondary model to check 'does this content contain overriding instructions?' before inserting it into the agent's context. This deep-dive expands Layer 4 into the full detection-model analysis: the two families (dedicated classifiers like DeBERTa/RoBERTa fine-tunes, Llama Prompt Guard, ProtectAI; and the secondary-LLM-as-detector approach), their accuracy in practice (benchmark 95-99%+ vs deployment 84-90% vs adversarial out-of-distribution lower still), how attackers evade them (the cat-and-mouse dynamic, the dual-injection problem where a payload must evade the detector AND compromise the primary, the correlated-bypass weakness when detector and primary share a model class), and how to compose them into a defense-in-depth stack where they are a layer, not a silver bullet. The detector is itself a model, and models are injectable. External placement stops DISABLE but not EVADE. Covers the five-step measurement methodology (in-dist + OOD corpus, per-detector bypass, false-positive tax, end-to-end, correlated-bypass test) and the state of the art: detection is necessary but not sufficient; deterministic layers bound the worst case.
SDD-B10
Academic Offensive Harnesses
Academic Offensive Harnesses (Adapted for AI-Target Attacks)
SDD-B11
Tau: The Reference Harness to Attack and Harden
Tau is Hugging Face's educational reimplementation of Pi — ~3,000 LOC across three packages, real enough to attack, small enough to read in an afternoon, and undefended against every surface B2-B8 covers. This deep-dive maps tau's seven attack surfaces to the five explicit extension points where each control attaches: tool-executor wrapping (B2/B4), the bash factory (B7), the credential store (B5), event subscription (B8), and tool-list construction (B0/B1). The two-bash-tool finding and the factory-level fix. The hardened session composition.
SDD-B12
MITRE ATLAS: The Adversary's Playbook for AI Systems
ATLAS gives you the adversary's playbook. C2B B9 gives you the builder's checklist. SDD-B01 gives you the offensive expansion. ATLAS is the connective tissue — the framework that names every technique an adversary has actually used against an AI system, in the order an adversary actually chains them, from reconnaissance through impact. The ATT&CK equivalent for AI/ML: twelve tactics, ~80+ techniques, dozens of case studies.
SDD-B13
The A2A Protocol — Agent-to-Agent Communication Security
B6 introduced inter-agent trust. This deep-dive goes deep on the A2A protocol that makes inter-agent communication real — and the attack surface it creates. The Agent Card, the Task lifecycle, signed JSON-RPC payloads: every B6 attack now targets a concrete protocol artifact.
Prerequisite: Course 1 and Course 2A recommended. Status: complete.