{
  "module": "S05 — Meta-Harness Architecture for Offensive Operations",
  "course": "2A — Building AI Harnesses for Cybersecurity",
  "version": "1.0.0",
  "duration_minutes": 35,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is CSI (Common Scaffold Interface), developed at Alias Robotics?", "options": ["An offensive tool suite that competes with CAI", "A scaffold combiner that routes between CAI, Claude Code, Codex CLI, and other harnesses through a local proxy", "A benchmark suite for testing injection resistance", "A vector database for engagement memory"], "answer_index": 1, "rationale": "CSI is a scaffold combiner — it does not implement offensive capability itself. It routes work between existing specialized harnesses through a local proxy that every sub-harness talks to. The proxy is the only endpoint any sub-harness can reach." },
    { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the five security primitives that MUST live at the meta-harness level.", "options": ["Classification, selection, telemetry, cost, routing", "Scope enforcement, evidence collection, kill chain state, signal/noise filtering, RoE enforcement", "Detection, exploitation, triage, reporting, retention", "Recon, weaponize, deliver, exploit, post-exploit"], "answer_index": 1, "rationale": "The five primitives are scope enforcement (every outbound call checked), evidence collection (hash-chained, scope_ref stamped), kill chain state (phase-appropriate gating), signal/noise filtering (one triage pipeline for all sub-harnesses), and RoE enforcement (rate caps + prohibited actions at the proxy)." },
    { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the four responsibilities of the meta-harness.", "options": ["Scan, attack, report, archive", "Task classification, harness selection, telemetry aggregation, cost control", "Recon, exploit, post-exploit, cleanup", "Routing, logging, alerting, backup"], "answer_index": 1, "rationale": "The meta-harness classifies the task (emits domain label), selects the sub-harness (deterministic lookup), aggregates telemetry (one ledger per engagement), and controls cost (per-engagement budget). Anything outside these four belongs in a sub-harness." },
    { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "Your team lets each sub-harness (CAI, Claude Code, Codex CLI) load its own scope file. What failure mode results?", "options": ["No failure — each tool knows its own scope best", "Scope drift — the three files diverge, leading to out-of-scope tool calls when one is updated and others are not", "Performance degradation from redundant scope checks", "The evidence log becomes corrupted"], "answer_index": 1, "rationale": "The 'three harnesses, three scope files' failure. If each sub-harness loads its own scope, they drift — the operator cannot keep them synchronized. Drifted scope is how out-of-scope tool calls happen. Cure: one scope file, owned by the meta-harness, read-only to every sub-harness." },
    { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "A finding is produced by CAI. Where should the evidence record be written?", "options": ["To CAI's internal evidence store", "To the meta-harness-owned evidence log via the local proxy, transparently — same schema as findings from Claude Code or Codex CLI", "To a separate per-sub-harness log, reconciled later", "Nowhere — CAI findings do not need evidence"], "answer_index": 1, "rationale": "The evidence log is meta-harness-owned. The proxy intercepts CAI's tool call and writes the record transparently — CAI does not opt in. One log, one schema, one engagement_id. A finding from CAI and one from Claude Code land in the same log; a client query is a single lookup." },
    { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You are implementing the security primitive middleware. A sub-harness emits a tool call. What runs FIRST?", "options": ["Execute the call, then check scope on the result", "Scope enforcement — check target::action against scope.json before anything else", "Evidence collection — log the request", "Cost accounting — charge the tokens"], "answer_index": 1, "rationale": "The middleware path is: scope check → RoE check → kill chain gate → execute → evidence log → triage → cost. Scope enforcement runs first because an out-of-scope call must be blocked before any further processing. If scope fails, the call is blocked, a scope_ref is stamped, and the block is logged." },
    { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Your meta-harness routes a code-review task to Codex CLI. Codex CLI has no concept of scope. What enforces scope?", "options": ["Nothing — code review does not need scope enforcement", "The local proxy intercepts every Codex CLI tool call and checks it against scope.json before execution, transparently", "Codex CLI must be modified to add scope checks", "The operator manually reviews each Codex CLI action"], "answer_index": 1, "rationale": "The whole point of the security primitive layer. Codex CLI has no native scope model, but it does not need one — the proxy enforces scope on every outbound call regardless of which sub-harness made it. This is why the primitives MUST live at the meta-harness level: you do not control the sub-harnesses." },
    { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You run InjecAgent against your meta-harness. One injected command is executed. What do you do?", "options": ["Note it and continue benchmarking — one failure is acceptable", "FAIL the gate. Fix injection resistance (S01.3 reader-actor separation, S02.1 structured-summary-only memory) before benchmarking anything else", "Increase the model's temperature to resist injection", "Disable the security primitive layer to avoid the failure"], "answer_index": 1, "rationale": "InjecAgent is a pass/fail gate, not a performance metric. A single execution of an injected command is a fail, regardless of how the harness scored on the six metrics, because the failure mode is catastrophic — the harness will exfiltrate scope, modify evidence, or call out-of-scope systems. Fix injection resistance first." },
    { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Your benchmark lab uses a live bug bounty target. Why is this wrong?", "options": ["It is illegal", "Live targets change, invalidating the baseline. The lab must use isolated Docker/VM targets with seeded, documented vulnerabilities for repeatability", "Live targets are too slow", "Live targets do not have vulnerabilities"], "answer_index": 1, "rationale": "A benchmark is only useful if repeatable. Live targets change (patches, config, traffic) and invalidate the baseline — you cannot compare harness versions against a moving target. The lab uses isolated targets with seeded vulnerabilities, a consistent scoring script, and a fixed compute envelope." },
    { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is harness selection a deterministic lookup table rather than a second model call?", "options": ["Model calls are too expensive for selection", "Once task classification is done, the mapping from domain label to sub-harness configuration is deterministic — no second decision needed. Classification is the only model decision; selection is mechanical", "Deterministic lookup is faster to implement", "Models cannot be trusted with selection"], "answer_index": 1, "rationale": "Classification is the model decision (read task, emit domain label). Once that label exists, the mapping to a sub-harness is a lookup: bug bounty → S02 harness, CTF → S04 harness, code review → Codex CLI. Adding a second model call for selection adds latency and non-determinism without value — the mapping is already known." },
    { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why can cost control NOT live in a sub-harness?", "options": ["Sub-harnesses are too slow to track cost", "A sub-harness only sees its OWN consumption. Only the meta-harness sees the TOTAL across all sub-harnesses in an engagement. Cost control requires the aggregate view", "Sub-harnesses do not have access to token counts", "Cost control is a legal requirement, not a technical one"], "answer_index": 1, "rationale": "Cost control requires the aggregate. CAI sees its tokens; Claude Code sees its tokens; neither sees the sum. The meta-harness aggregates telemetry into one ledger and enforces the per-engagement budget — re-routing, degrading, or halting when exhausted. This is why it is a meta-harness responsibility, not a sub-harness one." },
    { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "CAI ships an update that silently breaks its internal scope model. Your meta-harness uses CSI with the security primitive layer. What happens?", "options": ["Out-of-scope calls execute because CAI's scope check is broken", "Nothing — the proxy's scope enforcement is independent of CAI's internal model. The proxy checks every outbound call against scope.json regardless. The primitives are not in CAI's code path", "The meta-harness crashes", "You must roll back the CAI update immediately"], "answer_index": 1, "rationale": "This is the core argument for meta-harness-level primitives. The proxy's scope check is independent of the sub-harness's internal model. Because the primitives are applied at the proxy boundary — not in the sub-harness's code path — a sub-harness update that breaks its internal checks does not affect the meta-harness's enforcement. The wrap holds." },
    { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Your harness scores 95% detection rate but fails InjecAgent (one injected command executed). Is this harness operable against real targets?", "options": ["Yes — 95% detection is excellent", "No. Failing InjecAgent means the harness executes attacker-controlled actions from target output. It will exfiltrate scope, modify evidence, or call out-of-scope systems. The detection rate is irrelevant — the failure mode is catastrophic", "Only against targets with no injection risk", "Yes, if the operator monitors closely"], "answer_index": 1, "rationale": "InjecAgent is a pass/fail gate precisely because its failure mode is catastrophic. A harness that executes injected commands cannot be operated against real targets regardless of its detection rate — it will eventually betray the operator by acting on attacker-controlled input. Fix injection resistance (S01.3, S02.1) before any offensive benchmarking." },
    { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "A benchmark change improves detection rate by 5 points but doubles cost per finding. Is this a win?", "options": ["Yes — higher detection is always better", "No — it is a tradeoff, not a win. A change that improves one metric while doubling another can only be evaluated by measuring ALL six metrics consistently. Without measuring cost per finding, the 'improvement' would have been mistaken for a win", "Yes, if the client pays the extra cost", "It depends on the engagement type"], "answer_index": 1, "rationale": "The 'it feels faster/better' trap. Without measuring all six metrics (detection, exploit success, FPR, evidence completeness, time-to-first-finding, cost per finding), an unmeasured change is mistaken for an improvement. A 5-point detection gain at 2x cost is a tradeoff that may or may not be acceptable — you can only see it if you measure both." },
    { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is the evidence log described as making the meta-harness 'legally defensible'?", "options": ["Because it uses encryption", "Because it is owned by the meta-harness and aggregates findings from ALL sub-harnesses into ONE log with one schema and one hash chain. When the client asks how a finding was obtained, the answer is one query — not a reconciliation across three tools' separate, incompatible evidence stores", "Because it is stored in the cloud", "Because it uses Pydantic schemas"], "answer_index": 1, "rationale": "Legal defensibility comes from having one unified, hash-chained, scope-referenced evidence log. If each sub-harness kept its own log (different schemas, different hash chains, some missing entirely), reconciling them into a defensible legal record would be impossible. The proxy-owned single log makes every finding traceable to its authorization via scope_ref." }
  ]
}