Course 2B — AI Security

AI
Security

Securing and attacking the harness and the LLM underneath it. Prompt injection, data poisoning, tool/MCP security, agent trust, identity, sandboxes, observability, the OWASP Agentic Top 10, the Microsoft Failure Mode Taxonomy, governance, and security assessments as a service. Every attack is demonstrated against Tau; every defense is retrofitted to Pi.

~25
hours
13
modules
11
deep-dives
2
capstones
0 — Foundations
2 modules
1 — Injection & Poisoning
2 modules
2 — Trust Surfaces
3 modules
B4
Tool and MCP Security
Tools are the agent's hands, and the tool layer is a dual attack surface the model cannot defend. A tool's description is text the model reads as prompt; a tool's output is text the model reads as prompt — but neither carries a server-identity the harness verifies. The Model Context Protocol (MCP) registers third-party tool providers as trusted sources, which is exactly the position a software supply-chain attacker exploits. This module builds the tool-trust gate: signed manifests for tool definitions, provenance attestations for tool outputs, registry pinning for tool sources, and a description-injection scanner for the dual surface no one checks.
90 MIN
B5
Identity and Permission Design
An agent is not a human. Giving an agent a human's credentials means the agent acts with the human's full privilege — and an injected agent with admin credentials is an admin compromise. Agents need their own non-human identities, scoped to the current task, exchanged per action via RFC 8693 token exchange, and isolated from the agent process entirely. The agent never holds a long-lived secret; the harness does.
60 MIN
B6
Inter-Agent Trust and Communication Security
When agents talk to each other, the message channel is a trust boundary. A compromised agent can forge a message that looks like it came from the orchestrator — escalating privileges, impersonating a peer, or triggering a cascade across the mesh. The fix is not better models. It is signed messages, replay protection, and orchestrator-enforced compartment boundaries. B1 rated this the worst blast-radius surface on the map (mesh-wide) because most meshes default to implicit trust. This module closes it.
60 MIN
3 — Controls
2 modules
4 — Frameworks & Governance
6 modules
B9
OWASP Agentic Top 10 as Engineering Checklist
The OWASP Agentic AI Top 10 (2026) becomes a concrete engineering checklist, not a reading list. Each ASI risk (ASI01–ASI10) maps to its defense module (B2–B8), an attack procedure, a defense architecture, and a defined test. Eight rows report PASS/FAIL (deterministic controls); two report MEASURED (probabilistic). The checklist never produces a 'secure' verdict — it produces a scored report with characterized residuals, the same artifact a B12 engagement delivers as scope-complete output.
90 MIN
B10
Microsoft Failure Mode Taxonomy as Red Team Framework
B9 gave you the defense checklist. This is the offense playbook. The Microsoft Failure Mode Taxonomy v2.0 — seven new agentic failure modes plus the zero-click HITL bypass finding — is how systems actually fail in production, distilled from twelve months of deployed red teaming. Where OWASP tells you what to defend, this taxonomy tells you how attackers chain.
45 MIN
B11
Governance and Compliance
B2–B8 built the technical controls. They are necessary but not sufficient. CISOs, governance councils, and boards release budget and approval against frameworks — NIST AI RMF, ISO 42001, the EU AI Act — not against a working guardrail. This module is where a control you built becomes the control an auditor signs off on, the AI BOM procurement accepts, and the audit trail a regulator reads. The governance-to-engineering bridge: policy → control → test → audit.
75 MIN
B12
Harness Security Assessments as a Service
The capstone methodology module. B0–B11 are techniques, controls, and frameworks; B12 is the operational layer that turns them into a scoped, priced, reported, retestable engagement. The six-phase methodology (PTES / NIST SP 800-115 adapted) — Scoping, Reconnaissance, Discovery, Exploitation/Validation, Reporting, Retesting — with each phase tied to the module that supplies its content. The buyer is the CISO / AI security lead, and the deliverable is the scored artifact B9's checklist executor produces, packaged into a five-section report a board can read.
60 MIN
B13
Representation-Level Attacks: The Model as Attack Surface
The seventh surface — the one no B-module covers. C3 FT17 shows you how to steer model representations for alignment control; this module shows you why those same techniques are attack vectors when applied by an adversary. Activation steering (CAA via forward hooks), weight poisoning (BadNets, trojans surviving fine-tuning), checkpoint manipulation (.safetensors prevents code execution, NOT weight backdoors), and the dual-use of interpretability tooling (SAEs, Neuronpedia). The model is a software artifact with a supply chain and an internal state that can be tampered with.
50 MIN
B14
MLSecOps — The ML Security Lifecycle
The term appears in over thirty files across the course corpus, always as a forward reference, never as a taught subject. This module teaches it. Six stages — data, supply chain, training, deployment, inference, incident response — each mapping to content you have already learned. MLSecOps extends DevSecOps: five primitives transfer (SBOM, CI/CD gates, signed artifacts, secrets management, least privilege), five are genuinely new (model scanning, adversarial robustness, drift detection, prompt-injection monitoring, representation-layer analysis). The deployment gate is the single most important practice. The value is the lifecycle, not the content.
60 MIN
Capstones
2 capstones
Deep-Dives
13 studies
SDD-B01
OWASP Agentic AI Top 10 — Offensive Expansion
B9 handed you the defense checklist. This is the red-team playbook for every row. Each OWASP ASI risk (ASI01–ASI10) expanded into its full offensive attack procedure — the attack in depth, real-world exploitation, the evasion techniques that defeat the standard defenses, and the chaining that turns a single-row finding into a compound impact.
SDD-B02
Microsoft Failure Mode Taxonomy v2.0
B10 gave you the framework — seven new agentic failure modes plus the zero-click HITL bypass finding. This deep-dive expands each mode into a full case study: the attack chain, the detection gap (why the OWASP controls did not catch it), and the defense that does. The seven modes distilled from twelve months of deployed red teaming, each one a reproduced chain against an agent built to OWASP.
SDD-B03
InjecAgent: The Bridge Benchmark
The bridge from 2A to 2B. In 2A InjecAgent was the quality gate — is my harness injectable? In 2B it becomes the measurement instrument for every defense in the course — did this defense hold, and by how much? The ~50% baseline is where un-defended agents start; the defended delta is what every module from here on produces. This is the measurement layer for B2's injection defenses.
SDD-B04
CrabTrap Offensive Analysis
The primary attack target. CrabTrap (Course 1 DD-19) is the LLM-as-judge egress governance system — probabilistic enforcement of network policy. This deep-dive dissects the attack surface: prompt injection in the request body to manipulate the judge (the judge is a model, so it is injectable); the response-side gap (no inbound filtering — the judge governs egress, not the content that shapes the agent); the latency/cost tradeoff that makes the judge a bypassable bottleneck. The offensive analysis: how to break CrabTrap, what each break proves, and what the deterministic alternative (IronCurtain) fixes.
SDD-B05
IronCurtain Offensive Analysis
The defense to beat. IronCurtain (Course 1 DD-20) is the deterministic governance + credential quarantine system — the strongest single defense in the roster. This deep-dive dissects its attack surface: the compilation-fidelity attack (the deterministic policy is compiled by a probabilistic LLM, so the build-time compilation is the weak link — a miscompilation enforced consistently); the V8 isolate escape (the sandbox is a V8 isolate with escape CVE history, and the credential quarantine holds only as long as it does); the escalation-fatigue attack (flood the human-in-the-loop with approval requests until they rubber-stamp — the deterministic system still needs a human for edge cases). The offensive analysis of the strongest defense: how to break IronCurtain, what each break proves, and the residual risks that motivate defense-in-depth.
SDD-B06
RedAgent: Black-Box Jailbreaking
RedAgent (2026) finds that most black-box LLMs jailbreak within 5 queries using context-specific attacks. This is the model-level attack — we leave the harness (SDD-B04, SDD-B05) and target the model's refusal training directly. The 5-query chain works because context-specific attacks outperform generic jailbreaks: the attack is engineered to the target's deployed context (its system prompt, its tool surface, its conversation history), not a copy-paste payload. Covers the methodology, the defense-in-depth implications (no single layer suffices — B2's thesis, now at the model), how RedAgent-style testing fits a B12 engagement, and the dual-use disclosure dimension (B0) in its sharpest form.
SDD-B07
Agent SBOM and Supply Chain Assessment
The AI Bill of Materials as a supply-chain security tool. An agent is a dependency graph with a model at the center, and every tool, every MCP server, every checkpoint, and every training corpus is a trust dependency an attacker can reach. Covers the SBOM-to-AI-BOM extension (software bill of materials to AI bill of materials including the model, training data, tools/MCP servers, dependencies), the assessment tools and methodologies (CycloneDX ML/AI and crypto-material extensions, SPDX AI/SAI profiles, NTIA minimum elements), how an agent's supply chain is its attack surface (every trust dependency is a vector for the injection/exfiltration/governance-bypass attacks of B1-B5 and SDD-B04-B06), and the generation and validation practice. Expands B4's supply-chain material and B11's governance material into concrete supply-chain assessment.
SDD-B08
NeMo Guardrails: The Production Guardrail Framework
NeMo Guardrails is NVIDIA's programmable guardrail layer for LLM-based applications — the production-deployed reference for governance-beneath-the-agent. Covers what NeMo is (five rail types: input, output, dialog, retrieval, execution — each governing a distinct stage of the conversation lifecycle), how it enforces policy between the user and the model (programmable rails in Colang and Python, externally evaluated in a separate process the agent cannot reach), its strengths (programmable, multi-rail, production-deployed — NemoClaw scored +4 over OpenClaw for the external governance layer), and its load-bearing attack surface: the guardrail is a model, and models are injectable. External enforcement protects the rails from being DISABLED by a compromised agent (Course 1 Module 0.2); it does NOT protect the rail classifiers from being EVADED by adversarial inputs crafted against their model surfaces. Covers how to red-team a NeMo-guardrailed system: measure per-rail and end-to-end bypass rates over N attempts, demonstrate the correlated-bypass weakness when rails share a classifier model class, and synthesize with the defense-in-depth composition (NeMo is a layer, not a boundary). Expands Course 1 Module 0.2 and Course 1 DD-09 (NemoClaw).
SDD-B09
Prompt Injection Detection Models
B2's Layer 4 uses a secondary model to check 'does this content contain overriding instructions?' before inserting it into the agent's context. This deep-dive expands Layer 4 into the full detection-model analysis: the two families (dedicated classifiers like DeBERTa/RoBERTa fine-tunes, Llama Prompt Guard, ProtectAI; and the secondary-LLM-as-detector approach), their accuracy in practice (benchmark 95-99%+ vs deployment 84-90% vs adversarial out-of-distribution lower still), how attackers evade them (the cat-and-mouse dynamic, the dual-injection problem where a payload must evade the detector AND compromise the primary, the correlated-bypass weakness when detector and primary share a model class), and how to compose them into a defense-in-depth stack where they are a layer, not a silver bullet. The detector is itself a model, and models are injectable. External placement stops DISABLE but not EVADE. Covers the five-step measurement methodology (in-dist + OOD corpus, per-detector bypass, false-positive tax, end-to-end, correlated-bypass test) and the state of the art: detection is necessary but not sufficient; deterministic layers bound the worst case.
SDD-B10
Academic Offensive Harnesses
Academic Offensive Harnesses (Adapted for AI-Target Attacks)
SDD-B11
Tau: The Reference Harness to Attack and Harden
Tau is Hugging Face's educational reimplementation of Pi — ~3,000 LOC across three packages, real enough to attack, small enough to read in an afternoon, and undefended against every surface B2-B8 covers. This deep-dive maps tau's seven attack surfaces to the five explicit extension points where each control attaches: tool-executor wrapping (B2/B4), the bash factory (B7), the credential store (B5), event subscription (B8), and tool-list construction (B0/B1). The two-bash-tool finding and the factory-level fix. The hardened session composition.
SDD-B12
MITRE ATLAS: The Adversary's Playbook for AI Systems
ATLAS gives you the adversary's playbook. C2B B9 gives you the builder's checklist. SDD-B01 gives you the offensive expansion. ATLAS is the connective tissue — the framework that names every technique an adversary has actually used against an AI system, in the order an adversary actually chains them, from reconnaissance through impact. The ATT&CK equivalent for AI/ML: twelve tactics, ~80+ techniques, dozens of case studies.
SDD-B13
The A2A Protocol — Agent-to-Agent Communication Security
B6 introduced inter-agent trust. This deep-dive goes deep on the A2A protocol that makes inter-agent communication real — and the attack surface it creates. The Agent Card, the Task lifecycle, signed JSON-RPC payloads: every B6 attack now targets a concrete protocol artifact.

Prerequisite: Course 1 and Course 2A recommended. Status: complete.