Memory & Context Poisoning

Module B3 · Course 2B — Securing & Attacking Harnesses and LLMs

60 minutes · The only attack surface that survives logout — a poisoned memory is a prompt injection with a fuse, and the fuse crosses the session boundary

B2 built five layers of defense inside a session. Memory breaks the assumption that the attacker and the defense are alive at the same time. You inject today; the payload fires next week.

Pillar 1 — Injection & Poisoning

The two memory surfaces

The distinction is load-bearing. Persistence is the property that makes memory a categorically different attack surface.

Long-term memory

Persists across sessions. Stored outside the model — DB, vector store, files. Loaded into context at session start, before the user types anything. Survives logout/login.

Context window

Per-session. System prompt, conversation history, retrieved context, tool outputs this turn. Born at session start, dies at session end. B2's five layers all live here.

An injection in the context window lives for one session. An injection in long-term memory lives until someone finds it.

Why memory is trusted by default — and why that's wrong

The harness author classifies memory as trusted when they build the loader. So memory reads get no B2 Layer 1 tag and no B2 Layer 3 gate.

B2 layer	What it does	Does it see memory?
L1 untrusted tagging	Tags content crossing into context	No — memory loaded before tagger runs
L3 taint gate	Blocks high-impact calls with tainted args	No — memory reads not marked tainted

The defenses are intact. The attack routes around them — through the memory store, which loads as trusted by default.

B3.1 — The sleeper attack

Inject in session 1 · persist · activate in session 2 (Course 1 Module 4.3)

The sleeper attack across sessions

SESSION 1 — INJECTION          SESSION BOUNDARY — THE FUSE
─────────────────────────       (logout · login · days pass)
attacker delivers payload       user forgets session 1
agent writes SEED to memory              │
seed is INERT this session               ▼
session ends            SESSION 2 — ACTIVATION
                        ─────────────────────────
                        user starts fresh session
                        memory loader runs
                        seed loaded as TRUSTED
                        no tag · no gate · already in window
                                 ▼
                        SEED FIRES — obeyed as if system prompt

The load-bearing failure is the unvalidated READ in session 2. The memory loader trusts its own store — it never re-checks a poisoned entry before promoting it to the context window.

The persistence amplification

A transient injection costs the attacker one delivery per activation. A memory poisoning costs one delivery for N activations.

Vector	Cost per activation	Blast radius
Transient injection (B2)	1 delivery : 1 activation	One session
Sleeper / memory poison	1 delivery : N activations	Every session until found
Shared retrieval poison	1 write : N users × N queries	Every tenant on the index

This is why OWASP ranks Memory Poisoning (ASI04) as a distinct risk from prompt injection (ASI01). Injection is the delivery; memory is the persistence.

B3.2 — Retrieval & context poisoning

RAG poisoning · context-window flooding · cascading hallucination (ASI06)

Retrieval contamination (RAG poisoning)

The retrieval store is writable — and not all write paths are trusted.

Write path	Trusted?
User-uploaded document	Untrusted
Crawled / synced external content	Untrusted (compromisable)
Agent-generated summary (laundered injection)	Untrusted
Cross-tenant leak (weak isolation)	Untrusted

Once poisoned, a chunk persists in the index. Retrieval is the activation — the attacker crafts the chunk to be semantically similar to the victim's queries. Indirect injection with persistence.

Context-window flooding & cascading hallucination

Context-window flooding (MS Failure Mode #5)
Fill the window to displace the system prompt from attention. Attack by volume, not persuasion. Defense: pin system prompt, cap per-source tokens.

Cascading hallucination (ASI06)
One poisoned/hallucinated premise → treated as ground truth → corrupts all downstream reasoning. Faster when the premise is the agent's own memory.

A single hallucination is a localized error the user corrects. A cascade is a contaminated reasoning chain — internally consistent, but the foundation was poisoned.

B3.3 — The defense: harness-managed writes

The model PROPOSES · the harness VALIDATES · only the harness COMMITS

The memory-write gate — three checks

MODEL PROPOSES  ──▶  HARNESS VALIDATES  ──▶  HARNESS COMMITS
(content +              (3 checks, in order)       (tagged with
 justification +                                   provenance + trust)
 provenance)                                          │
   │                                                  ▼
   │            CHECK 1 — PROVENANCE          memory store
   │            CHECK 2 — TRUST LEVEL         (model NEVER
   │            CHECK 3 — CONTENT              touches it)
   ▼                  │
  proposal            ▼
                ALLOW or DENY
   any check fails ─▶ DENY (logged + surfaced)

// The load-bearing line — the sleeper signature:
if (trustLevel === "untrusted" && looksLikeInstruction(content))
    return DENY;   // untrusted-sourced AND instruction-shaped

The conjunction — untrusted provenance AND instruction-like content — is the sleeper seed's signature. A trusted preference passes; an untrusted fact passes. The payload that is both gets blocked.

Provenance on every entry · separation of paths

Field	Why it matters
`source`	user_direct · retrieval:doc · tool · agent_inferred
`trust_level`	trusted · semi_trusted · untrusted
`written_at` / `written_by`	Audit trail back to the injection point
`validation_result`	What the gate decided and why

Write path — model proposes → harness validates → commits with tag. No direct model write.

Read path — load → tag by trust level → untrusted entries get B2 L1 tag → taint gate applies. Defense in depth.

Retrieval defenses — closing the loop with B2

Provenance filtering — down-rank or filter results from untrusted sources before injection. Trust follows the source.
Re-ranking — a secondary model re-ranks by relevance AND instruction-density. Advisory; reduces suspicious volume.
Retrieved-content taint tagging — every retrieved chunk tagged UNTRUSTED (B2 L1) unless provenance proves otherwise. This connects B3 to B2.

A tainted retrieval result that tries to drive a high-impact tool call hits B2's Layer 3 taint gate — even if the model is fooled, the gate is not.

Lab & what's next

Lab (07): (1) the sleeper-attack demo — inject a seed in session 1, watch it activate in session 2 against an undefended agent, then watch the gate block it. (2) build the memory-write gate — validateAndCommit() with provenance, trust, and content checks, plus a provenance-tagging read path.

Next — B4: Tool and MCP Security. Memory is the persistence surface; tools are the action surface. A poisoned memory driving an over-privileged tool is the ASI04 + ASI05 combined worst case. B4 attacks the tool layer the gate-protected memory tries to reach.