{ "module": "DD-01 — Pi: The Minimal Baseline", "course": "Master Course — Harness Engineering", "version": "1.0.0", "duration_minutes": 30, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Name Pi's complete tool registry.", "options": [ "bash, read_file, write_file, search — the minimal capable set, validated by the Vercel finding that cutting tools improves selection accuracy.", "bash, edit, grep, glob, search — the standard coding-assistant set.", "run, read, write, exec, net — the five-tool Unix-style set.", "shell, file_read, file_write, web — the four-tool I/O set with renamed semantics." ], "answer_index": 0, "rationale": "Pi's complete tool registry is bash, read_file, write_file, and search — four tools, schema-first. The Vercel finding (Module 2) showed that cutting 80% of tools IMPROVED selection accuracy: the minimum capable set is the lowest-noise set, and a 5th tool adds decision noise at the dispatch layer. This is the '4-tool philosophy' — a noise-reduction strategy, not a capability limitation." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What is Pi's permission model, stated in three words, and what does it mean?", "options": [ "Gate-every-action — every tool call requires explicit human approval before execution.", "Trust-the-model — the model is trusted to act; there are no per-action approval gates.", "Sandbox-by-default — all tool execution occurs inside an ephemeral container.", "Scope-and-allow — tools execute only against an explicit allowlist of paths and URLs." ], "answer_index": 1, "rationale": "Pi's permission model is 'trust-the-model' (three words). The model is trusted to act; there are no per-action approval gates. This is correct for a trusted single-user environment (a personal assistant on your own machine) and is a vulnerability catalog for any multi-user, multi-tenant, or untrusted-input context. Pi has no sandbox, no filesystem scoping, and no untrusted-content tagging — the omissions are deliberate tradeoffs for thinness, not oversights." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "How many of the 5 stop conditions (Module 1.2) does Pi implement, and which ones?", "options": [ "All 5: end_turn, max-iterations, token-budget, error-threshold, and human-interrupt.", "2 of 5: end_turn (model emits no tool call — natural stop) and max-iterations (hard cap — safety net). Missing: token budget, error threshold, human interrupt.", "3 of 5: end_turn, max-iterations, and token-budget.", "1 of 5: end_turn only. Pi has no max-iterations guard." ], "answer_index": 1, "rationale": "Pi implements 2 of the 5 stop conditions from Module 1.2: end_turn (the model emits no tool call — the natural stop) and max-iterations (the hard cap — the safety net). The missing three are token budget, error threshold, and human interrupt. These absences are acceptable for a personal assistant but are scored findings (Pi accepts the risk of a runaway bill, a compounding-error loop, and no mid-stream cancellation). The token budget is the cheapest missing one to add." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You fork Pi to run inside a CI pipeline that operates on an untrusted codebase (PRs from external contributors). Without modifying the harness, which OWASP ASI risk is Pi MOST vulnerable to on the very first tool call?", "options": [ "ASI09 Resource Exhaustion — the model will loop until max-iterations trips.", "ASI01 Goal Hijacking via indirect injection — read_file pulls attacker-controlled content into context as raw, untagged text; no taint gate exists; the model may comply. This is Pi's defining vulnerability 'by design.'", "ASI03 Excessive Agency — Pi has too many tools for the CI task.", "ASI08 Supply Chain — the CI runner's dependencies are unsigned." ], "answer_index": 1, "rationale": "Pi has no untrusted-content tagging (Module 2.4). When read_file pulls a file from an untrusted PR into context, that content enters as raw text. If the file contains injected instructions, the model may comply — there is no taint gate to stop it. This is OWASP ASI01 (Goal Hijacking via indirect injection) and it is Pi's defining vulnerability 'by design': Pi is a thin harness that trusts the model. This is the exact surface Course 2B's SDD-B01 offensive expansion lands on first. Running unmodified Pi against untrusted input is the textbook misuse — Pi is correct only for trusted single-user environments." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You are asked to harden Pi for a team deployment but can add ONLY ONE of the three proposed changes (token budget, per-turn observability, basic compaction). Which delivers the highest safety-per-line return, and why?", "options": [ "Basic compaction — it extends Pi's effective session length and indirectly reduces cost.", "Per-turn observability — the 8-field payload makes long sessions debuggable, which is Pi's biggest operational gap.", "Token-budget stop condition — it is the cheapest missing stop condition and prevents the worst outcome (a runaway bill from a hijacked loop), closing a Module 1.2 gap for minimal code.", "None of these — hardening requires the full security layer (sandbox + scope + gates), so a single addition is pointless." ], "answer_index": 2, "rationale": "The token-budget stop condition is the highest safety-per-line return. Pi implements only 2 of the 5 stop conditions (end_turn + max-iterations). The token budget is the cheapest missing one to add — a few lines that cap cumulative token spend and hard-stop the loop when exceeded. It prevents the worst single outcome: a hijacked or runaway loop producing an unbounded bill. Observability and compaction improve operability and range, but neither prevents the catastrophic-cost outcome the way a token budget does. (The 'none of these' answer is wrong because the three changes are explicitly the 'minimal production' fork that preserves thinness — the full security layer is a different fork for a different use case.)" }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "Trace a read_file({path: 'auth.ts'}) tool call through Pi's dispatch. What are the steps in order?", "options": [ "execute → validate name → schema-check → return to user.", "Model emits tool_use → harness validates name against the 4-tool registry → schema-validates {path: string} → executes readFileSync → returns content → appends to history as {role: 'tool', content} → next model call.", "Schema-check → permission gate → execute → verification gate → append.", "User approves → execute → sanitize output → append to history." ], "answer_index": 1, "rationale": "The 7-step dispatch (Module 2): (1) the model emits tool_use: read_file({path: 'auth.ts'}); (2) the harness validates the name against the 4-tool registry; (3) the harness schema-validates {path: string}; (4) the harness executes readFileSync('auth.ts'); (5) it returns the content; (6) it appends the result to history as {role: 'tool', content: ...}; (7) the next model call sees the appended result. Clean, minimal — exactly Module 2's 7-step dispatch with 4 tools. Note what is ABSENT: no permission gate (step 2.5 in a hardened harness), no output sanitization, no verification." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "A colleague argues: 'Pi's <1,000-token system prompt is lazy — we should write a detailed 5,000-token prompt that specifies behavior precisely.' Using the future-proof test, what is the correct response?", "options": [ "Agree — a detailed prompt produces more predictable behavior across model versions.", "Disagree — the thin prompt is deliberate delegation. The future-proof test asks whether performance improves on model upgrade WITHOUT harness changes. A thick prompt over-specifies behavior the model already handles and rots when the model upgrades; the <1k prompt gets out of the way and co-evolves. Add prompt content only when the model demonstrably fails without it.", "Compromise — a 2,500-token prompt balances specificity and flexibility.", "Defer — prompt length should be determined by A/B testing, not principle." ], "answer_index": 1, "rationale": "The thin prompt is a co-evolution mechanism, not laziness. The future-proof test (Module 1): does performance improve on model upgrade without harness changes? Pi passes by design because the dumb-loop and thin prompt add no cleverness that can rot. A 5,000-token prompt over-specifies behavior the model already handles; when the model upgrades, the over-specified behavior may conflict with the model's new capabilities and the future-proof test FAILS. The cure: add prompt content only when the model demonstrably fails without it, and remove it when the model outgrows it. Treat the prompt as the delegation layer, not the control layer." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Pi is deployed as a personal assistant on your laptop. After a long debugging session, its responses degrade — it forgets earlier context and repeats itself. Which scored finding from the 12-module audit explains this, and what is the minimal fix?", "options": [ "Module 4 (Memory) — Pi has no persistent store. Add a database.", "Module 3 (Context Management) — Pi has no compaction; raw tool output accumulates until context rots. The minimal fix is a threshold-triggered summarizer that compacts history when it exceeds a limit, extending Pi's effective range without compromising thinness.", "Module 8 (State) — Pi is stateless. Add checkpointing.", "Module 11 (Observability) — Pi's console.log hides the cause. Add structured logging." ], "answer_index": 1, "rationale": "The symptom (forgets earlier context, repeats itself on a long session) is context rot — Module 3. Pi has no compaction: raw tool output goes straight into history with no masking, truncation, or summarization. On a long bash session, the accumulated output displaces earlier context until the model loses it. The minimal fix is a threshold-triggered summarizer: when history exceeds a limit, summarize the oldest portion. This is one of the three proposed 'minimal production' changes — it extends Pi's effective range without compromising thinness. (Memory (M4) is about cross-SESSION persistence, not within-session rot; State (M8) is about interruption/resume; Observability (M11) would help diagnose but would not fix the cause.)" }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You are scoping a Course 2B (Attacking Harnesses) engagement and choose Pi as the un-hardened baseline. Which three OWASP ASI offensive procedures (from SDD-B01) will produce findings against Pi 'by default,' and why?", "options": [ "ASI03 (excessive agency), ASI08 (supply chain), ASI09 (resource exhaustion) — Pi has too many tools and no circuit breaker.", "ASI01 (goal hijacking via indirect injection — no taint gate), ASI07 (output-as-input — no sanitizer), ASI10 (inter-agent trust escalation — no principal binding). Each maps to a layer Pi deliberately omits; each is trivial against Pi 'by design.'", "ASI02 (prompt leakage), ASI04 (memory poisoning), ASI06 (cascading hallucination) — Pi's thin prompt and ephemeral memory are the weak points.", "ASI05 (tool/skill abuse), ASI07 (output handling), ASI08 (supply chain) — Pi's tool layer is the weak point." ], "answer_index": 1, "rationale": "Pi is the un-hardened baseline for Course 2B because it omits the three layers those procedures target. ASI01 (Goal Hijacking via indirect injection): Pi has no taint gate — read_file content enters as raw, untagged text, so drift is trivial. ASI07 (Insecure Output Handling / output-as-input): Pi has no sanitizer — agent outputs are consumed downstream with no escape of the natural-language instruction boundary. ASI10 (Broken Access Control / inter-agent trust escalation): Pi has no principal binding — though Pi is single-agent, the trust assumption (the model is trusted to act) is the same surface ASI10 exploits in multi-agent systems. Each maps to a layer Pi deliberately omits for thinness; each is trivial 'by design.' This is why Pi is the correct baseline: the procedures land on what Pi does not have." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the deep-dive argue that 'Pi scores 25/60 on the rubric, and that low score is the EXPECTED result, not a failure'? What is the distinction between 'what Pi IS' and 'what Pi IS NOT'?", "options": [ "The rubric is wrong — it should weight legibility and co-evolution more heavily so Pi scores higher.", "The rubric scores production-readiness across ALL 12 dimensions. Pi is DELIBERATELY thin — it is not trying to be production-grade across all dimensions. The 25/60 reflects what Pi IS NOT (the deliberate omissions: sandbox, memory, state, verification), not a failure of what Pi IS (the irreducible core: loop, tools, thin prompt, co-evolution). Pi's value is in dimensions the production-readiness rubric does not measure.", "Pi IS a failure for production; the score is honest about that, and Pi should only be used as a teaching tool.", "The score is low because the rubric penalizes single-agent designs; if subagents were scored generously, Pi would score 30/60." ], "answer_index": 1, "rationale": "The distinction is load-bearing. The rubric scores production-readiness across all 12 modules. Pi is deliberately thin — it does not attempt to be production-grade across all dimensions. The 25/60 score reflects what Pi IS NOT: the deliberate omissions (sandbox (M5: 1/5), memory (M4: 1/5), state (M8: 1/5), verification (M9: 1/5)). It does NOT reflect a failure of what Pi IS: the irreducible core Pi does well (loop M1: 4/5, tools M2: 5/5, prompt M12: 5/5) plus the value the rubric does not measure (legibility, co-evolution, the future-proof test). A thick harness scores higher because it attempts all dimensions; Pi's value is in the dimensions the production-readiness rubric does not capture. This is why the score must be read alongside the use case." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Pi passes the 'future-proof test' (performance improves on model upgrade without harness changes) by design. Explain the mechanism, and identify which Pi design decisions are load-bearing for passing it.", "options": [ "Pi passes because it uses the latest API version, which auto-upgrades.", "The dumb-loop philosophy (no lookahead/planning/reflection that could conflict with new model capabilities) and the ultra-thin prompt (no over-specified behavior that rots when the model upgrades) are the load-bearing decisions. The harness adds no cleverness that can degrade; the model upgrade flows through unimpeded.", "Pi passes because its 4 tools are so simple they never need to change.", "Pi passes because it is stateless — no persisted state needs migration on upgrade." ], "answer_index": 1, "rationale": "The future-proof test (Module 1) asks whether performance improves on model upgrade without harness changes. Pi passes BY DESIGN because of two load-bearing decisions. (1) The dumb-loop philosophy: the ReAct loop has no lookahead, no planning, no reflection — harness-level cleverness that could conflict with, override, or duplicate the new model's improved capabilities. (2) The ultra-thin prompt: the <1k-token prompt does not over-specify behavior the model already handles, so it does not rot or conflict when the model upgrades. Together: the harness adds no cleverness that can degrade, so the model upgrade flows through unimpeded and Pi benefits for free. This is Pi's reason for being — co-evolution — and it is the opposite of over-engineering." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team forks Pi and adds only a sandbox (containerized tool execution), leaving everything else unchanged. They deploy it for a multi-user internal tool. Using the deep-dive's reasoning, what is wrong with this fork?", "options": [ "Nothing — a sandbox is the single highest-value security addition and makes Pi production-ready.", "Adding a sandbox without the full security layer (filesystem scoping (M5.3), per-action approval gates (M6), untrusted-content tagging (M2.4), and observability (M10)) gives the COST of the sandbox without the BENEFIT. The sandbox stops one vector (host compromise) but leaves indirect injection (ASI01), unrestricted file paths inside the container, and zero debuggability. The threat model changed (multi-user) but only one layer was added — fork coherently or not at all.", "The sandbox is the wrong choice; they should have added compaction first.", "The fork is correct but they should also increase the max-iterations cap." ], "answer_index": 1, "rationale": "This is the 'add a sandbox and it is production-ready' anti-pattern. Adding a sandbox alone changes Pi's threat model (the team moved to multi-user) but closes only ONE vector: host compromise via bash/write_file. It leaves every other gap open. No filesystem scoping (M5.3): the sandboxed process can still write anywhere inside the container. No per-action approval gates (M6): a hijacked model still acts without confirmation. No untrusted-content tagging (M2.4): indirect injection still flows through read_file into context. No observability (M10): a compromise is still invisible. The sandbox gives the cost (initialization overhead, operational complexity) without the benefit of a coherent security posture. The cure: if you need a sandbox, you need the FULL security layer (Course 2A's stack), not one component. Fork Pi coherently." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "How does Course 3 (LLM Fine-Tuning) use Pi's loop, and why does the dumb-loop philosophy make the fine-tuned-model swap seamless? What would break if Pi's loop contained model-specific lookahead logic?", "options": [ "Course 3 rewrites Pi's loop for each fine-tuned model — the dumb loop is insufficient for fine-tuned capabilities.", "Course 3 drops the fine-tuned model INTO Pi's UNCHANGED loop. The dumb-loop philosophy (no model-specific lookahead/planning) means the harness has no logic that assumes the old model's behavior, so the swap needs no harness changes. Model-specific lookahead logic would encode assumptions about the base model that the fine-tuned model may violate — the swap would break or degrade.", "Course 3 uses Pi only as inspiration; it deploys a different harness for fine-tuned models.", "Course 3 bypasses the loop entirely and calls the fine-tuned model directly." ], "answer_index": 1, "rationale": "Course 3's contribution is INSIDE the model; Pi's contribution is the stable loop AROUND it. The fine-tuned model drops into Pi's UNCHANGED loop because the dumb-loop philosophy means the harness contains no model-specific lookahead, planning, or reflection logic — no assumptions about the base model's behavior that the fine-tuned model could violate. If Pi's loop contained model-specific lookahead logic (e.g., 'if the model produces a plan, validate it against schema X'), that logic would encode assumptions about the base model's planning behavior. The fine-tuned model may produce different plans, or no plans, or better plans — and the lookahead logic would either break, degrade, or fight the new behavior. The dumb loop's absence of such logic is exactly what makes the swap seamless. This is the future-proof test applied to fine-tuning: the harness does not need to know the model changed." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Construct the argument for why Pi — scoring 25/60, with no sandbox, no memory, and no verification — is the MOST IMPORTANT deep-dive in the curriculum, rather than the least. What makes it load-bearing?", "options": [ "It is the simplest, so it is the best pedagogical starting point before the complex harnesses.", "Pi is the BASE ARCHITECTURE every subsequent course builds on, varies, attacks, or deploys. Course 2A builds the security layers Pi omits; Course 2B attacks Pi's trust-the-model posture; Course 3 drops fine-tuned models into Pi's unchanged loop; Course 4 deploys Pi's loop at scale. Understanding WHY Pi is thin (the deliberate omissions and the co-evolution value) is the prerequisite for understanding what every thicker harness adds and why. Pi is the reference, not the warm-up.", "Pi is the most widely deployed harness, so it has the highest practical impact.", "Pi is the only open-source harness, making it the only one students can study." ], "answer_index": 1, "rationale": "Pi is load-bearing because it is the base architecture the entire curriculum threads through. The deep-dive's central claim: 'Pi is the base architecture every course in this curriculum builds on.' Course 2A builds security tools FOR it (the sandbox/scope/gate Pi omits). Course 2B ATTACKS it (the OWASP ASI offensive procedures find Pi's trust-the-model posture by default). Course 3 drops fine-tuned models INTO its unchanged loop (the future-proof test makes the swap seamless). Course 4 DEPLOYS it at scale (the same ~80-line loop, replicated, with observability and state layers on top). The 25/60 score and the deliberate omissions are precisely what make Pi the reference: every thicker harness is measured against Pi's minimalism ('how much thickness does X add, and why?'), and every subsequent course assumes you understand why Pi is thin. Read Pi once; refer back to it in every course. It is the reference, not the warm-up." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "The deep-dive proposes three changes to move Pi from 'minimal baseline' to 'minimal production' (token budget, per-turn observability, basic compaction). Why are THESE three chosen, and why do they preserve thinness while adding a token-budget + an 8-field log + a summarizer would NOT compromise Pi's core value?", "options": [ "They are chosen because they are the cheapest to implement in lines of code.", "They are chosen because each closes a scored gap (M1.2 stop condition, M10 observability, M3 context) WITHOUT adding cleverness that could rot on model upgrade (the co-evolution value) and WITHOUT changing the threat model (the thinness value). They add infrastructure (a cap, a log payload, a summarizer) not model-specific logic — so the future-proof test still passes and the trust-the-model posture is preserved. The result is a harness that is still Pi-shaped but operationally viable.", "They are chosen because they are the only three modules where Pi scores below 3/5.", "They are chosen because they are required by the OWASP ASI checklist." ], "answer_index": 1, "rationale": "The three changes are chosen because each closes a specific scored gap (the token budget closes the M1.2 missing stop condition; the 8-field payload closes the M10 observability gap; the summarizer closes the M3 context-rot gap) WITHOUT compromising the two properties that make Pi valuable. (1) Co-evolution (the future-proof test): each change adds infrastructure — a spending cap, a structured log payload, a threshold-triggered summarizer — NOT model-specific lookahead/planning/reflection logic. The harness adds no cleverness that can rot on model upgrade. (2) Thinness (trust-the-model posture): none of the three changes alters the permission model, adds approval gates, or re-introduces the layers Pi deliberately omits for its single-user use case. The threat model is unchanged. The result is 'minimal production' — a harness that is still Pi-shaped (minimal loop, 4 tools, thin prompt) but operationally viable (capped cost, debuggable, longer effective range). This is the fork worth making, and it is deliberately distinct from the 'add the full security layer' fork (Course 2A) which serves a different use case." } ] }