Prompt Injection Detection Models

SDD-B09 · Course 2B — Securing & Attacking Harnesses and LLMs

45 minutes · B2's Layer 4 expanded into the full detection-model analysis

A secondary model checks "does this content contain overriding instructions?" before it enters the agent's context. The detector is itself a model — and models are injectable.

Deep-Dives

The Layer 4 pitch — and why it is not a silver bullet

B2's Layer 4: before untrusted content (retrieved doc, email, web page, tool result) enters the agent's context, a secondary model evaluates it for overriding instructions. Flag it, quarantine it. Feels solved.

It is not. The detection model is a model. A dedicated classifier has a probeable decision boundary. A secondary LLM carries the full injection surface of any LLM. Benchmark accuracy is 95–99%+; real-world, adversarially-adapted accuracy is 84–90%. The detector is a layer, not a boundary.

B09.1 — The detection landscape

Two families, what they classify, the accuracy numbers

Two families of detectors

Family 1 — Dedicated classifier
DeBERTa/RoBERTa fine-tunes, Llama Prompt Guard (86M/22M), ProtectAI, Lakera Guard. Fast (19–600ms), cheap, tunable threshold. Fixed boundary — ages with its corpus.

Family 2 — Secondary-LLM-as-detector
A second LLM prompted: "does this contain overriding instructions?" Flexible, re-promptable, feeds human review. Itself an LLM — full injection surface.

Default: dedicated classifier as the high-volume gate; secondary LLM as a second-pass reviewer for flagged-ambiguous content. Neither is a boundary.

What they classify — and the accuracy gap

The signal: overriding instructions — text that changes the model's task, role, or constraints. Direct, indirect, encoded, multi-step.

Number	Range	Source
Benchmark	95–99%+	In-distribution test set (vendor model card)
Deployment	84–90%	Real traffic, tolerable false-positive ceiling
Adversarial	lower, decays	Out-of-distribution, adaptive attacker — the one that matters

No vendor puts the adversarial number on the model card. Prompt Guard AUC ~0.998 on benchmark; one eval found 26.7% detection at 50% FPR. Galileo: 56% of production LLMs successfully injected.

B09.2 — The cat-and-mouse dynamic

Evasion, the dual-injection problem, why the gap is adversarial

The detector is a model — and models are injectable

The SDD-B08 insight, restated one layer down. External placement protects the detector from being DISABLED by a compromised agent. It does NOT protect it from being EVADED by an input crafted against its decision boundary. The detector runs; it classifies the adversarial payload as benign.

Empirical evasion is substantial: arXiv:2504.11168 shows high benchmark accuracy does not translate to robustness against adaptive attackers. Promptfoo catalogs guardrail-evasion as reproducible artifacts.

The dual-injection problem

ADVERSARIAL PAYLOAD
        │
        ▼
  ┌─ Evades DETECTOR? (classified benign)
  │     NO → CAUGHT (quarantined)
  │     YES ↓
  └─ Compromises PRIMARY MODEL? (executes)
        NO → useless (evaded but moved nothing)
        YES → ATTACKER SUCCESS (dual-injection satisfied)

  CORRELATED CASE: detector + primary share model class
  → evasion against one has elevated prob. against other
  → independence assumption FAILS, end-to-end rate > product

The dual constraint helps against naive attackers. Against adaptive ones, it is a targeting signal: probe the detector's boundary, find the false-negative region, craft the payload to sit there.

B09.3 — Composing the detector into the stack

Defense-in-depth, measurement, state of the art

The detector is Layer 4, not the boundary

Layer	What it does	Surface
L1-2 Input rails + IronCurtain	Catches gross violations, bounds worst case	Deterministic
L4 Detection model	Catches the BULK of indirect injections	Model — evasion surface
L5 Output rails + refusal	Backstop if L4 is evaded	Model — evasion surface
Harness scope gate	Hard stop — disallowed action blocked	Deterministic — the floor

Only deterministic layers have no evasion surface. End-to-end bypass = product of residuals, floored by the harness gate. The detector bounds volume; deterministic layers bound the worst case.

Measurement: the five steps

Assemble the corpus — in-distribution + a held-out out-of-distribution set (the number that matters).
Per-detector bypass rate — false-negative rate at the deployed threshold.
False-positive tax — run legitimate content; set threshold where FPR is tolerable. Detection rate at THAT threshold is the real number.
End-to-end bypass rate — full stack; fraction of injections → disallowed action.
Correlated-bypass test — if detector + primary share a model class, test whether evasion transfers.

Defensible measurement: "Detector L4: 91% in-dist / 74% OOD at 2% FPR; end-to-end 3.2% full stack / 5.8% correlated; harness gate is the floor."

State of the art — and where it is heading

Defenders: larger corpora push OOD accuracy up (96.9% frontier); ensemble detectors so evasion against one doesn't evade all; better secondary-LLM output constraints.

Attackers: automated adversarial-prompt generation (RedAgent machinery, applied to the detector surface); dual-injection crafting; shared-model-class correlation makes it easier.

Forecast: detection remains high-value, OOD accuracy improves but never reaches 100% against an adaptive adversary. Winners compose the detector with deterministic layers and measure the residual — they don't deploy a detector and trust the model card. Necessary, not sufficient.

Lab & what's next

Lab (07): build a simulated detection stack — a dedicated classifier proxy, a secondary-LLM heuristic proxy, an in-distribution + out-of-distribution corpus, and a measurement harness reporting per-detector, end-to-end, and correlated-bypass rates. No GPU.

Next — SDD-B10: Academic Offensive Harnesses. PentestGPT, APT-Agent, HPTSA, VulnBot, CAI — re-examined as methodologies for attacking AI systems. HPTSA's hierarchical planning is directly relevant to the zero-click chain. This is the "where is this heading" deep-dive that closes the course's offensive material.