Prompt Injection Defense Engineering
Module B2 · Course 2B — Securing & Attacking Harnesses and LLMs
90 minutes · The central engineering module. Every other module attacks a surface; this one attacks the mechanism that makes all of them vulnerable.
Prompt injection is not a model bug you wait for a provider to fix. It is an architectural property of systems that consume untrusted content and emit privileged actions — and it is yours to engineer against.
Pillar 1 — Injection & Poisoning
Why this module is the depth module
B1 gave you the seven surfaces and classify_untrusted_content() — the tag. A tag tells the model "this is data." It does not prevent the model obeying it, and it does not prevent a downstream tool call acting on an injected goal. The tag is Layer 1. This module builds Layers 2 through 5.
The 1.6% trap
The model's refusal training is one layer — and an indirect injection is designed to bypass it by never appearing as a refusal request. Demanding the model "refuse harder" moves the success rate, not the category.
The 98.4% — this module
Injection is delivered through the harness (tool outputs), interpreted by the model, and converted to impact by the harness (tool calls). All three are yours to engineer.
Two numbers that make this mandatory
~50%
InjecAgent — of agentic tasks vulnerable to indirect injection via tool outputs. Not a theoretical risk; the default state of an undefended agent. The bridge benchmark from Course 2A.
~5
RedAgent — queries to jailbreak most black-box LLMs with automated mutation. Sets the offensive floor: an attacker with automation finds a working payload fast.
Delivery mechanism works ~50% of the time; payload found in ~5 queries by automation. The window between "ship an agent" and "it is hijacked" is short. This module closes it.
B2.1 — The injection problem & taxonomy
Why LLMs cannot separate instructions from data · the five attack classes
The instructions-vs-data problem
Classical software. Instructions are not data. A parser boundary (SQL parameterization, NX bit, DEP) ensures data can never become instruction. The entire history of software security is enforcing this boundary.
LLMs. One token stream. System prompt, user input, and tool output are concatenated and processed uniformly by attention. No parser separates instruction from data. The model has one input stream.
Safety RLHF moves the success rate (60% → 4%). It does not build the boundary. A model trained to refuse "ignore your instructions" still complies with an obfuscated version — it has no mechanism to know the obfuscated instruction is data. The architecture is the harness's to fix.
The five attack classes
1 · DIRECT
user → model
crudest; narrow surface
2 · INDIRECT
tool output → model
MOST DANGEROUS · ~50%
3 · MULTI-STEP
across turns
defeats per-turn defense
4 · ENCODED
base64, split, homoglyph
defeats pattern matchers
5 · FLOODING
saturate the window
heap-spray analogue
Indirect is the most dangerous — it crosses a trust boundary invisibly (agent went for data, got instructions), it scales (one poisoned page hits every agent), and it is the InjecAgent ~50% vector. The attacker never needs an account on the target system.
Indirect injection: the canonical path
The path InjecAgent measures: attacker content on a page → fetch_url returns it → enters context untagged → loop's goal is hijacked → tool call executes with real credentials → real-world impact (file read, email sent, fund transfer).
Greshake et al. demonstrated this end-to-end against real LLM-integrated applications in 2023. The single most common failure: the tool output crosses into the context window without an untrusted tag. That one missing tag is the ~50% rate. B2 builds five layers so that one missing tag is no longer fatal.
B2.2 — The layered defense
Five layers · each catches what the layer above misses · matched to Course 1's 6-layer model + 9-layer stack
The five defense layers
L1 · UNTRUSTED TAGGING — wrap inbound content in <untrusted> tags. Stops naive direct/indirect. Bypassed: model ignores tag. Course 1: 6.3
L2 · INSTRUCTION ISOLATION — system prompt in privileged system role; model taught to obey the tag. Stops override attempts. Bypassed: attention domination. Course 1: instruction hierarchy
L3 · TAINT TRACKING + GATE — mark untrusted, propagate taint, block high-impact tool calls with tainted args. DETERMINISTIC CORE. Stops the impact even if the model was fooled. Bypassed: taint laundering → B3.
L4 · SECONDARY-MODEL DETECTION — separate model judges "is this an override attempt?" Decodes + analyzes. Stops encoded class. Advisory, not sole gate. Course 1: CrabTrap
L5 · CAPABILITY MINIMIZATION — tools scoped to minimum, sandboxed. An injected agent cannot call what does not exist. DETERMINISTIC FLOOR. Course 1: IronCurtain
The taint gate — the load-bearing line
// Layer 3 — taint gate. Deterministic. Runs on every tool call.
export async function gateToolCall(call: ToolCall, context: ContextEntry[]): Promise<GateResult> {
// L5 — capability ceiling. tool not in allowlist → blocked.
const cap = TOOL_CAPABILITIES[call.name];
if (!cap) return { allow: false, reason: `tool not in capability allowlist` };
// L5 — per-arg validators (path allowlist, SQL shape, recipient allowlist).
if (cap.argAllowlist) for (const [arg, v] of Object.entries(cap.argAllowlist))
if (!v(call.arguments[arg])) return { allow: false, reason: `arg '${arg}' failed validator` };
// L3 — taint gate. THE LOAD-BEARING LINE.
const tainted = argumentsAreTainted(call, context);
if (tainted && HIGH_IMPACT.has(call.name))
return { allow: false, reason: "high-impact tool args derive from tainted content" };
// L4 — advisory detection. Raises score; does not gate alone.
if (injectionDetector) {
const score = await injectionDetector(JSON.stringify(call.arguments));
if (score > 0.9 && cap.impact !== "low")
return { allow: false, reason: `detector score ${score} on ${cap.impact}-impact`, requiresApproval: true };
}
return { allow: true, reason: "passed all gates" };
}
An injection that fooled the model, defeated the tag, and slipped past the detector is still stopped here — send_email with arguments from a tainted web page is blocked deterministically. No model in the loop.
B2.3 — Probabilistic vs deterministic
The central tension B2 resolves · CrabTrap vs IronCurtain · the hybrid
CrabTrap vs IronCurtain — each fails
CrabTrap (DD-19) — probabilistic. LLM-as-judge egress filter. Flexible, adapts to novel attacks. Fails: judge is a model (foolable — RedAgent finds it in ~5 queries); response-side gap (inbound returns unfiltered); latency budget (skipped under load).
IronCurtain (DD-20) — deterministic. Compiled policy + quarantine. Certain on the enumerated set. Fails: compilation fidelity (poisoned compile → wrong rule enforced faithfully); brittle (alias laundering); coverage gaps (novel attacks pass — no rule matches).
Probability where determinism is possible → unneeded bypass rate. Determinism where the space is open → unaffordable coverage gaps. Both are wrong when misapplied.
The resolution — each on its boundary
Determinism on the TAINT boundary (L3 + L5). The question is structural and enumerable: tainted? high-impact? in the capability allowlist? The rule is simple and the compile is trivial. No model in the loop, no bypass rate.
Probability on the CONTENT boundary (L4). The question is semantic and open: "is this an override attempt?" A model that decodes and analyzes catches obfuscation no rule can enumerate. Advisory — raises score, does not gate alone.
The hybrid: deterministic gating on taint (L3) + probabilistic detection on content (L4) + capability minimization as the floor (L5). An attack must defeat all three kinds of defense — structural, semantic, and capability — to reach impact.
B2.4 — Defense in depth & residual risk
No single layer suffices · measure residual risk · never "fixed"
The conjunction — an attack must bypass all five
| Layer | Bypass condition | Residual → closure |
|---|
| L1 Tagging | model obeys despite tag | → L2 isolation |
| L2 Isolation | attention domination | → L3 gate (doesn't care if fooled) |
| L3 Taint gate | taint laundering (memory/files) | → B3 memory defense |
| L4 Detector | obfuscation fools detector | → L5 capability ceiling |
| L5 Capability | sandbox escape / over-privilege | → B5 identity, B7 sandbox |
Each bypass condition is different (tag-resistance, attention-domination, taint-laundering, detector-fooling, capability-conformance). The conjunction is hard — not a proof of security, but an argument about attacker work factor. The work factor is what you measure.
Measure residual risk — never "fixed"
B0's anti-pattern applies: treating an AI finding as binary "fixed/unfixed." After deploying the five layers, you do not declare the agent "secured." You measure the residual injection success rate under a harness and report the number.
The measurement is InjecAgent-style: agentic tasks with indirect injections planted in tool outputs, run the defended agent, count successes.
The deliverable is the number, not the claim. "Injection success dropped from 50% to 2% under InjecAgent; the residual 2% is the laundered-encoded class, which B3 and L4 are tuned to reduce further." That is an engineering deliverable. "We added prompt injection defenses" is not.
Anti-patterns
"The model's safety training will refuse." The 1.6% trap. Cure: Layers 1–5 are the defense.
Tagging without gating. The tag tells the model; it does not stop the tool call. Cure: L1 + L3 together.
Probabilistic defense as the only gate. A judge has a bypass rate; automation finds it. Cure: deterministic L3 gate for high-impact.
Determinism on the content boundary. The obfuscation space is open. Cure: probability (L4) for content.
Capability over-provisioning "for convenience." An injected agent uses the surplus immediately. Cure: L5 minimal allowlist.
Declaring "fixed" after deploying. Cure: measure residual risk under a harness; report the number.
Lab & what's next
Lab (07): Build the Injection-Defense Stack — (a) an untrusted-content tagger, (b) a taint-tracking middleware that propagates taint and gates high-impact tool calls, (c) a secondary-model injection detector. TypeScript, type-safe, no GPU. Includes a mini-InjecAgent harness to measure your defense's effect.
Next — B3: Memory and Context Poisoning. Closes Layer 3's residual: taint laundering through memory stores. The sleeper attack (poison in session 1, activate in session 2) is exactly the bypass that defeats per-call taint gating. B3 makes memory writes harness-managed and retrieval-time-tagged so laundered values re-enter tainted.
The load-bearing principle
Prompt injection is an architectural problem, not a model bug. LLMs process instructions and data as one token stream — no parser separates them. Safety RLHF moves the success rate; it does not build the boundary. The boundary is the harness's to build: tag the untrusted (L1), isolate the instructions (L2), gate the tainted deterministically (L3), detect the obfuscated probabilistically (L4), and minimize the capabilities (L5). An attack must defeat all five. The residual is measured, not declared zero.
Determinism on the taint boundary. Probability on the content boundary. Capability minimization as the floor. That is what B2 contributes that no prior module does.