Module B2 — Prompt Injection Defense Engineering

Prompt Injection Defense Engineering

The central engineering module. Prompt injection is not a model bug — it is an architectural property of systems that consume untrusted content and emit privileged actions. LLMs process instructions and data as one token stream; no parser separates them. This module builds the five-layer defense in the harness (tag, isolate, taint-gate, detect, minimize capability), resolves the probabilistic-vs-deterministic enforcement tension (determinism on the taint boundary, probability on the content boundary), and measures residual risk under a harness — never declaring 'fixed.'

minutes

artifacts

sub-sections

Every other module in 2B attacks a surface; this one attacks the mechanism that makes all of them vulnerable. Prompt injection is OWASP ASI01 (goal hijacking) and the engine behind ASI02 (leakage). InjecAgent measured ~50% of agentic tasks vulnerable to indirect injection via tool outputs; RedAgent jailbroke most black-box LLMs within 5 queries. The delivery works half the time; the payload is found in five queries. The window between ship and hijack is short. This module closes it — not by demanding the model refuse harder (the 1.6% trap), but by building five layers in the harness (the 98.4%) where injection is delivered, interpreted, and converted to impact. An attack must bypass all five to reach real-world impact. The residual is measured, not declared zero.

Key Claims

Load-Bearing Claims

Prompt injection is an architectural problem, not a model bug. LLMs process instructions and data as one token stream — no parser separates them. Safety RLHF moves the success rate (60% to 4%); it does not build the boundary. The boundary must be built in the harness: tag the untrusted (Layer 1), isolate the instructions (Layer 2), gate the tainted deterministically (Layer 3), detect the obfuscated probabilistically (Layer 4), and minimize the capabilities (Layer 5). An attack must bypass all five to reach impact.

Indirect injection via tool output is the most dangerous vector and the default vulnerability — InjecAgent measured ~50% of agentic tasks susceptible. The tool faithfully returns attacker-controlled content; the harness inserts it into the context window without an untrusted tag; the model obeys it as instruction. The single most common failure is the missing tag at the tool-output-to-context boundary. Layer 3 (the deterministic taint gate) is where that 50% drops toward zero — even when the model is fooled, the gate blocks high-impact calls with tainted arguments.

The probabilistic-vs-deterministic tension resolves by assigning each defense to the boundary where its failure mode does not matter. Determinism on the structural boundary (Layer 3 taint: tainted? high-impact? — enumerable, no bypass rate; Layer 5 capability: in the allowlist?). Probability on the semantic boundary (Layer 4 content: is this an override attempt? — open space, needs a model). CrabTrap (probabilistic) and IronCurtain (deterministic) are not rivals; they are tools for different boundaries. Using probability where determinism is possible inherits an unneeded bypass rate; using determinism where the space is open inherits unaffordable coverage gaps.

The deliverable is the measured residual risk, not a 'secure' claim. After deploying the five layers, you do not declare the agent secured. You measure the injection success rate under an InjecAgent-style harness and report the number (e.g., 50% to 2%), characterize the residual (the encoded-and-laundered class), and route it to its closure module (B3 for taint laundering, B5 for over-privilege, B7 for sandbox escape). 'We added prompt injection defenses' is not a deliverable; the measured number is.

After This Module

Explain why LLMs cannot reliably distinguish instructions from data — the instructions-are-not-data principle that governs classical software, why it fails architecturally for autoregressive language models, and why safety RLHF moves the success rate but not the category.

Classify any prompt-injection attack into the five-part taxonomy (direct, indirect, multi-step, encoded/obfuscated, context-window flooding) and predict which defense layers each evades.

Cite the empirical motivation — InjecAgent's ~50% agentic-task vulnerability to indirect injection via tool outputs and RedAgent's ~5-query jailbreak of most black-box LLMs — and explain why these make prompt injection a when-not-if engineering problem.

Implement the layered defense — untrusted-content tagging (L1), instruction isolation (L2), taint tracking with propagation and gating (L3), secondary-model injection detection (L4), and output sandboxing/capability minimization (L5) — and state which layer each attack defeats and which layer defeats each attack.

Resolve the probabilistic-vs-deterministic enforcement tension — explain why CrabTrap's (DD-19) probabilistic judge and IronCurtain's (DD-20) deterministic compilation each fail, and design a hybrid using deterministic gating for the taint boundary and probabilistic detection for the content boundary.

Read, extend, and break a real taint-tracking middleware — the TypeScript implementation in this module and the lab — and identify the exact line where an attack bypassing one layer is caught by the next.

Artifacts

Teaching Document

~5,500 words prose (7,370 with code); 4 sub-sections — the injection problem and taxonomy (instructions-vs-data, five attack classes, InjecAgent ~50% and RedAgent ~5-query evidence), the layered defense (L1-L5 with a real TypeScript taint-tracking middleware), probabilistic vs deterministic enforcement (the CrabTrap/IronCurtain tension resolved: determinism on the taint boundary, probability on the content boundary), defense in depth and residual risk (the conjunction, measuring not declaring fixed); with anti-patterns, key terms, lab pointer, 14 references

READ

Diagrams

6 Mermaid diagrams — the instructions-vs-data problem (classical parser boundary vs LLM one-stream), the injection taxonomy tree (five classes with delivery channel + stopping layer), the indirect-injection data-flow with five defense-layer halts, the 5-layer defense stack (matched to Course 1, with bypasses + residuals + closure modules), the taint-tracking flow (mark-propagate-gate + the laundering bypass), the probabilistic-vs-deterministic resolution matrix

READ

Slide Deck

19 slides — reveal.js, dark theme, design-system teal; covers the two empirical numbers, the instructions-vs-data problem, the five attack classes, the five defense layers, the taint-gate load-bearing code, CrabTrap vs IronCurtain failures, the B2.3 hybrid resolution, the conjunction, measuring residual risk, anti-patterns

READ

Teaching Script

~55 min spoken (~3,260 words at 140 wpm); verbatim transcript with [SLIDE N] cues for all 19 slides; longer than B0/B1 (90-min depth module)