# Lab Specification — Module B10: Microsoft Failure Mode Taxonomy as Red Team Framework

**Course**: Course 2B — Securing & Attacking Harnesses and LLMs
**Module**: B10 — Microsoft Failure Mode Taxonomy as Red Team Framework
**Duration**: 45–60 minutes
**Environment**: Python 3.10+. No GPU, no model API calls, no network. This lab is a **trace-based simulation** — you execute a pre-built attack chain as a logged event sequence and then write the session-level detector that catches it. Type hints throughout. The "agent" is a deterministic state machine so the chain reproduces exactly and the detector can be verified against a ground truth.

---

## Learning objectives

By the end of this lab you will have:

1. **Designed a 5+ step zero-click HITL bypass chain** against a sample agent — each step individually benign and approval-passing, the compound malicious. This is the centerpiece finding of the Microsoft Taxonomy v2.0 made concrete and runnable.
2. **Executed the chain as a logged trace** through a deterministic agent simulation, observing that every per-step approval gate passes while the compound reaches exfiltration.
3. **Implemented session-level intent detection** — the required control that catches the chain by evaluating the compound across the session, not the step in isolation. This is the cross-turn extension of B8's observability layer.
4. **Identified the specific session-level gap** that allowed the compound to pass — the deliverable a red team hands the client. You will name the missing control (intent check, freshness window) and verify your detector closes the gap.

This lab operationalizes the module's thesis: per-step approval is necessary and insufficient. The compound intent is the attack class. The detector you write is the control.

---

## Phase 0 — Setup (2 min)

```bash
mkdir b10-taxonomy-lab && cd b10-taxonomy-lab
python3 -m venv .venv && source .venv/bin/bin/activate   # optional; no external deps
# No pip install. This lab uses only the standard library.
```

No model, no API, no GPU. The agent is a deterministic simulation. The value is in the trace, the detector, and the gap analysis — not in a stochastic jailbreak.

---

## Phase 1 — The agent simulation and the trace model (8 min)

The agent is a state machine that processes a list of `Action` events. Each action has a type, a description (what the human approver sees), a sensitivity level, and metadata. The harness logs every action to a `Trace`. A per-step approval gate approves any action below a sensitivity threshold. The lab's question: does the chain reach impact despite every gate passing?

### 1.1 The trace model

Create `trace.py`:

```python
from __future__ import annotations
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class Sensitivity(str, Enum):
    PUBLIC = "public"
    INTERNAL = "internal"
    SENSITIVE = "sensitive"      # requires human approval
    RESTRICTED = "restricted"    # high-impact: exfil, payment, credential use


class ActionKind(str, Enum):
    RECEIVE_INPUT = "receive_input"
    READ_DATA = "read_data"
    DRAFT_MESSAGE = "draft_message"
    SEND_MESSAGE = "send_message"
    CALL_TOOL = "call_tool"
    LOG = "log"


@dataclass
class Action:
    step: int
    kind: ActionKind
    description: str              # what the human approver sees in isolation
    sensitivity: Sensitivity
    data_touched: str = ""        # e.g. "vendor_list", "billing_records", "credentials"
    external_recipient: Optional[str] = None   # non-None => potential exfiltration
    source: str = "user"          # "user" | "retrieved_doc" | "tool_output" | "external"


@dataclass
class Trace:
    actions: list[Action] = field(default_factory=list)
    approvals: list[dict] = field(default_factory=list)

    def log(self, action: Action) -> None:
        self.actions.append(action)

    def record_approval(self, step: int, gate: str, decision: str, reason: str) -> None:
        self.approvals.append(
            {"step": step, "gate": gate, "decision": decision, "reason": reason}
        )
```

### 1.2 The per-step approval gate (the control B8 built — necessary, insufficient)

Create `gates.py`:

```python
from __future__ import annotations
from trace import Action, Sensitivity, Trace


def per_step_approval(action: Action, trace: Trace) -> tuple[str, str]:
    """The per-step HITL gate. Sees ONLY the current action in isolation.

    Approves anything below RESTRICTED. For RESTRICTED, it 'asks a human'
    who sees only action.description — which the chain has engineered to look benign.
    Returns (decision, reason).
    """
    if action.sensitivity in (Sensitivity.PUBLIC, Sensitivity.INTERNAL):
        trace.record_approval(action.step, "per_step", "AUTO_APPROVE",
                              f"{action.sensitivity.value} — below threshold")
        return ("APPROVE", "auto-approved: below sensitivity threshold")

    # SENSITIVE or RESTRICTED: simulate a human who sees ONLY this step's description.
    # The human has no session context — exactly the zero-click chain's premise.
    trace.record_approval(action.step, "per_step", "HUMAN_APPROVE",
                          f"human reviewed: '{action.description}'")
    return ("APPROVE", f"human approved based on description: '{action.description}'")
```

**The critical design choice**: the per-step gate sees only `action.description`. It has no access to prior steps. This is precisely why it is structurally insufficient — and your detector (Phase 3) will sit above it with full trace access.

### 1.3 Run a trivial trace to confirm the gate works

Create `smoke_test.py`:

```python
from trace import Action, ActionKind, Sensitivity, Trace
from gates import per_step_approval

t = Trace()
a = Action(step=1, kind=ActionKind.READ_DATA, description="Read public FAQ doc",
           sensitivity=Sensitivity.PUBLIC, data_touched="faq")
decision, reason = per_step_approval(a, t)
t.log(a)
print(f"Step {a.step}: {decision} — {reason}")
print(f"Approvals logged: {len(t.approvals)}")
```

Run it: `python3 smoke_test.py`. Confirm the gate auto-approves. This is your sanity check before the real chain.

---

## Phase 2 — Design and execute the zero-click bypass chain (20 min)

This is the heart of the lab. You design a 5+ step chain against the sample agent, run it through the gate, and confirm that **every gate passes while the compound reaches exfiltration.**

### 2.1 The sample agent's capability surface

The agent (from reconnaissance — Mode 7, Capability Disclosure) has these capabilities:

| Capability | Sensitivity | Notes |
| --- | --- | --- |
| Receive email/webhook | INTERNAL | External input — the chain's trigger |
| Read vendor list | SENSITIVE | Internal data |
| Read billing records | SENSITIVE | Internal data |
| Draft email | INTERNAL | Low sensitivity — drafting is benign |
| Send email | RESTRICTED | High-impact, but the body can be engineered to look benign |
| Look up "auditor" contact | INTERNAL | Resolves an address the payload supplied |

### 2.2 Design your chain

Design a chain with **at least 5 steps** that:
- Starts from a single external input (zero-click trigger).
- Each step individually passes `per_step_approval` (each is benign in isolation).
- The compound exfiltrates the vendor list and/or billing details to an external recipient.
- No single step's `description` reveals the compound intent.

Record your design as a sequence of `Action` objects. Here is the canonical 5-step design from the teaching doc, which you may adapt or extend:

```python
# canonical_chain.py — the chain from the teaching doc, encoded as Actions
from trace import Action, ActionKind, Sensitivity

CHAIN: list[Action] = [
    Action(
        step=1, kind=ActionKind.RECEIVE_INPUT,
        description="Received email: reconcile Q3 vendor list vs billing, report discrepancies",
        sensitivity=Sensitivity.INTERNAL,
        source="external",
    ),
    Action(
        step=2, kind=ActionKind.READ_DATA,
        description="Read vendor list and billing records for Q3 reconciliation",
        sensitivity=Sensitivity.SENSITIVE,
        data_touched="vendor_list,billing_records",
        source="external",  # the read was prompted by the external email
    ),
    Action(
        step=3, kind=ActionKind.DRAFT_MESSAGE,
        description="Draft email to vendor noting a billing discrepancy; CC billing-auditor@vendor-partner.com",
        sensitivity=Sensitivity.INTERNAL,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",  # the 'discrepancy' and the CC were engineered by the payload
    ),
    Action(
        step=4, kind=ActionKind.SEND_MESSAGE,
        description="Send drafted reconciliation email to vendor (routine follow-up)",
        sensitivity=Sensitivity.RESTRICTED,
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
    Action(
        step=5, kind=ActionKind.LOG,
        description="IMPACT: vendor list + billing details delivered to billing-auditor@vendor-partner.com (attacker)",
        sensitivity=Sensitivity.RESTRICTED,
        data_touched="vendor_list,billing_records",
        external_recipient="billing-auditor@vendor-partner.com",
        source="external",
    ),
]
```

### 2.3 Execute the chain through the per-step gate

Create `run_chain.py`:

```python
from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from canonical_chain import CHAIN


def run_chain(chain: list[Action]) -> Trace:
    """Execute a chain through the per-step gate. Returns the logged trace."""
    t = Trace()
    for action in chain:
        decision, reason = per_step_approval(action, t)
        print(f"Step {action.step} [{action.kind.value}] sensitivity={action.sensitivity.value}")
        print(f"  desc: {action.description}")
        print(f"  GATE: {decision} — {reason}")
        if decision == "APPROVE":
            t.log(action)
        else:
            print(f"  CHAIN HALTED at step {action.step}")
            break
    return t


def assess_impact(t: Trace) -> dict:
    """Determine whether the compound reached exfiltration."""
    external_sends = [a for a in t.actions
                      if a.external_recipient and a.kind.value in ("send_message", "log")]
    sensitive_reads = {a.data_touched for a in t.actions if a.sensitivity.value in ("sensitive", "restricted")}
    return {
        "steps_executed": len(t.actions),
        "all_gates_passed": len(t.actions) == len(CHAIN),
        "external_send_count": len(external_sends),
        "sensitive_data_touched": list(sensitive_reads),
        "exfiltrated": len(external_sends) > 0 and bool(sensitive_reads),
    }


if __name__ == "__main__":
    print("=== EXECUTING ZERO-CLICK HITL BYPASS CHAIN ===\n")
    t = run_chain(CHAIN)
    print("\n=== IMPACT ASSESSMENT ===")
    impact = assess_impact(t)
    for k, v in impact.items():
        print(f"  {k}: {v}")
    print(f"\nDeliverable question: did every gate pass AND the compound exfiltrate? "
          f"{'YES — per-step approval is insufficient.' if impact['exfiltrated'] and impact['all_gates_passed'] else 'NO.'}")
```

Run it: `python3 run_chain.py`. **Confirm the output shows every gate APPROVE and the impact assessment reports `exfiltrated: True`.** This is the empirical demonstration that per-step approval is structurally insufficient. Every human approval was "correct" for the step it saw. The compound still reached impact.

### 2.4 The deliverable so far

Write `chain_analysis.md` answering:

1. Which step's description was the most engineered to look benign? (Likely step 4 — "routine follow-up" while carrying the exfiltration.)
2. Which approver, if any, *could* have refused given only their step? (Trick question: none — that is the point.)
3. What single piece of session context, if shown to the step-4 approver, would have changed their decision? (Steps 1+3 together: the external trigger plus the engineered CC address.)

This is the gap identification (engagement methodology step 5) — the deliverable a red team hands the client.

---

## Phase 3 — Implement session-level intent detection (12 min)

Now write the control that catches the chain. It sits *above* the per-step gate, evaluates the *compound*, and halts when the compound matches an exfiltration pattern or diverges from the user's original intent.

### 3.1 The detector spec

Create `session_detector.py`:

```python
from __future__ import annotations
from trace import Action, ActionKind, Sensitivity, Trace


def session_level_intent_check(action: Action, trace: Trace,
                                user_original_intent: str = "answer the user's question") -> tuple[str, str]:
    """The session-level detector. Evaluates the COMPOUND, not the step.

    Sits ABOVE per_step_approval. Returns (decision, reason).
    If it returns BLOCK, the agent halts regardless of the per-step gate.

    Three mechanisms (from the teaching doc):
      1. Intent tracking — does this action's source align with the user's intent?
      2. Compound-action detection — does the action sequence so far match an exfil pattern?
      3. Approval freshness — has too much external-driven activity accumulated without re-confirmation?
    """
    prior = trace.actions  # the compound so far

    # --- Mechanism 1: intent tracking ---
    # An external-source action driving sensitive reads is a divergence from
    # a session whose intent was set by the user.
    external_chain = [a for a in prior if a.source == "external"] + (
        [action] if action.source == "external" else []
    )
    if len(external_chain) >= 3 and action.source == "external":
        return ("BLOCK",
                f"intent divergence: {len(external_chain)} external-source actions in a "
                f"session whose intent was '{user_original_intent}' — possible hijack/drift")

    # --- Mechanism 2: compound-action detection ---
    # Pattern: sensitive data read + external recipient present + send pending = exfiltration shape
    touched_sensitive = any(
        a.sensitivity in (Sensitivity.SENSITIVE, Sensitivity.RESTRICTED)
        and a.data_touched for a in prior
    )
    has_external_recipient = any(a.external_recipient for a in prior) or action.external_recipient
    if (touched_sensitive and has_external_recipient
            and action.kind in (ActionKind.SEND_MESSAGE, ActionKind.LOG)
            and action.external_recipient):
        return ("BLOCK",
                f"compound matches exfiltration pattern: sensitive data ({[a.data_touched for a in prior if a.data_touched]}) "
                f"+ external recipient ({action.external_recipient}) + {action.kind.value}. "
                f"Per-step gate cannot see this; session-level compound does.")

    # --- Mechanism 3: approval freshness ---
    # If 4+ actions have executed since the last user-intent checkpoint, re-confirm.
    last_user_action = max(
        (i for i, a in enumerate(prior) if a.source == "user"), default=-1
    )
    actions_since_user = len(prior) - 1 - last_user_action
    if actions_since_user >= 4 and action.sensitivity == Sensitivity.RESTRICTED:
        return ("BLOCK",
                f"freshness window expired: {actions_since_user} actions since last user-intent "
                f"checkpoint; RESTRICTED action requires re-confirmation with compound context")

    return ("PASS", "no compound anomaly detected")
```

### 3.2 Run the chain with the detector layered above the gate

Create `run_chain_detected.py`:

```python
from __future__ import annotations
from trace import Action, Trace
from gates import per_step_approval
from session_detector import session_level_intent_check
from canonical_chain import CHAIN


def run_chain_with_session_detection(chain: list[Action]) -> Trace:
    """Per-step gate AND session-level detector. Detector has veto."""
    t = Trace()
    for action in chain:
        # Layer 1 (top): session-level — evaluates the compound
        sess_decision, sess_reason = session_level_intent_check(action, t)
        if sess_decision == "BLOCK":
            print(f"\n>>> SESSION-LEVEL DETECTOR HALTED at step {action.step}:")
            print(f"    {sess_reason}")
            t.record_approval(action.step, "session_level", "BLOCK", sess_reason)
            break

        # Layer 2 (bottom): per-step — evaluates the step in isolation
        gate_decision, gate_reason = per_step_approval(action, t)
        if gate_decision == "APPROVE":
            t.log(action)
            print(f"Step {action.step}: PASSED both layers — {action.description[:60]}...")
        else:
            print(f"Step {action.step}: per-step gate blocked")
            break
    return t


if __name__ == "__main__":
    print("=== EXECUTING CHAIN WITH SESSION-LEVEL DETECTION ===\n")
    t = run_chain_with_session_detection(CHAIN)
    print(f"\n=== RESULT: chain executed {len(t.actions)} of {len(CHAIN)} steps ===")
    print(f"Exfiltrated: {any(a.external_recipient and a.kind.value == 'log' for a in t.actions)}")
```

Run it: `python3 run_chain_detected.py`. **Confirm the detector halts the chain before impact.** The output should show the detector blocking at the step where the compound first matches an exfiltration pattern — likely step 3 or 4, before the data leaves.

### 3.3 Verify the detector catches the chain but does not over-block benign traffic

Write `test_detector.py` with two cases:

```python
from trace import Action, ActionKind, Sensitivity, Trace
from session_detector import session_level_intent_check

# Case A: the attack chain — detector MUST block before impact
# (use your CHAIN; assert it halts at step <= 4)

# Case B: a benign session — detector MUST NOT block
benign = [
    Action(step=1, kind=ActionKind.RECEIVE_INPUT, description="User asks: summarize the meeting notes",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=2, kind=ActionKind.READ_DATA, description="Read meeting notes from docs store",
           sensitivity=Sensitivity.INTERNAL, data_touched="meeting_notes", source="user"),
    Action(step=3, kind=ActionKind.DRAFT_MESSAGE, description="Draft summary for the user",
           sensitivity=Sensitivity.INTERNAL, source="user"),
    Action(step=4, kind=ActionKind.LOG, description="Return summary to user",
           sensitivity=Sensitivity.PUBLIC, source="user"),
]
t = Trace()
t.log(benign[0]); t.log(benign[1]); t.log(benign[2])
decision, reason = session_level_intent_check(benign[3], t)
assert decision == "PASS", f"FALSE POSITIVE on benign traffic: {reason}"
print("Case B (benign): PASS — detector correctly did not block")
```

**Your detector must block the attack chain (Case A) and pass the benign session (Case B).** If it false-positives on benign traffic, tune the thresholds (the `>= 3` external-action count, the freshness window of 4). This is the engineering tradeoff: too aggressive and you block legitimate work; too lax and you miss the chain.

---

## Phase 4 — Gap identification and the deliverable (8 min)

The lab's deliverable is not "the chain worked" (Phase 2) nor "the detector catches it" (Phase 3). It is the **gap identification** — the artifact a red team hands a client.

Write `engagement_deliverable.md`:

```markdown
# Engagement Deliverable — Zero-Click HITL Bypass Chain Finding

## Finding
A 5-step zero-click chain exfiltrated the vendor list and billing details to an
external recipient. Every per-step approval gate passed. The compound reached impact.

## The gap (what the client patches)
The deployed agent has per-step HITL approval (B8, working correctly) but NO
session-level intent detection. The per-step gate is structurally insufficient
against compound-intent chains: each step is benign in isolation, so every gate
correctly approves, but the composition is exfiltration.

## The control (the patch)
Layer session-level intent detection ABOVE per-step approval, with three mechanisms:
1. Intent tracking — re-derive the agent's sub-goal each turn from source, not
   from accumulating (possibly contaminated) context.
2. Compound-action detection — pattern-match the action sequence against
   exfiltration/escalation/lateral shapes.
3. Approval freshness windows — re-confirm with compound context after N actions
   or T seconds since the last user-intent checkpoint.

## Verification
The detector (session_detector.py) halts the chain at step [N] before impact,
and passes benign traffic (test_detector.py Case B) without false positives.

## Cross-reference
- B8 (Observability & Attack Detection): the substrate this detector extends.
- B9 (OWASP ASI): the per-row controls all PASSED — this finding lives between rows.
- B10 Mode 7 (Capability Disclosure): the reconnaissance that enabled chain design.
```

This is what B12 packages into the engagement report alongside B9's scored checklist.

---

## Phase 5 — Stretch: extend the chain across failure modes (optional, 10 min)

The canonical chain is a single-mode (goal hijack + HITL bypass) demonstration. For stretch credit, design a chain that **crosses multiple taxonomy modes**:

- Start with **Mode 7** (capability disclosure) to enumerate the agent's tools.
- Use **Mode 5** (session context contamination) to establish a durable false premise in turn 1.
- Deliver the payload via **Mode 1** (a poisoned MCP tool output) in turn 2.
- Reach impact through the **zero-click HITL bypass** (the compound).

Encode it as a longer `Action` list, run it through both gates, and confirm your session detector catches the compound. The point: the modes compose into chains. A red-team that tests modes in isolation misses the composition.

---

## Deliverables

- `trace.py` — the trace and action model (Phase 1)
- `gates.py` — the per-step approval gate (Phase 1)
- `canonical_chain.py` — your designed 5+ step zero-click chain (Phase 2)
- `run_chain.py` — executes the chain, confirms every gate passes and impact is reached (Phase 2)
- `chain_analysis.md` — the gap identification from the chain run (Phase 2)
- `session_detector.py` — the session-level intent detector with all three mechanisms (Phase 3)
- `run_chain_detected.py` — runs the chain with the detector layered above the gate (Phase 3)
- `test_detector.py` — verifies the detector blocks the attack and passes benign traffic (Phase 3)
- `engagement_deliverable.md` — the client-facing finding, gap, and control (Phase 4)
- (optional) `cross_mode_chain.py` — a chain crossing Modes 7, 5, 1, and HITL bypass (Phase 5)

## Success criteria

- [ ] The canonical chain executes all steps with every per-step gate APPROVE, and `assess_impact` reports `exfiltrated: True`.
- [ ] `chain_analysis.md` correctly identifies that no single approver could have refused (the structural insufficiency).
- [ ] `session_detector.py` implements all three mechanisms (intent tracking, compound-action detection, approval freshness).
- [ ] The detector HALTS the attack chain before impact (Case A) and PASSES benign traffic (Case B) — no false positives.
- [ ] `engagement_deliverable.md` names the specific gap (no session-level detection), the control (three mechanisms), and cross-references B8/B9/B10.
- [ ] Every Python file has type hints; the lab runs with no external dependencies and no GPU.