{ "id": "dd-13-exam", "title": "DD-13 OpenHarness — Academic Baseline (Inspectability as Product)", "deep_dive": "dd-13-openharness", "course": "master-course", "duration_minutes": 30, "pass_threshold": 0.70, "question_count": 15, "bloom_distribution": {"recall": 3, "application": 6, "analysis": 6}, "questions": [ { "id": "q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is OpenHarness's defining objective function, and how does it differ from a production harness?", "options": [ "Capability under constraints (cost, latency, safety) — the same as a production harness, just with fewer tools.", "Inspectability under reproducibility — can a researcher point at the exact mechanism that produced this decision, vary it, and measure the effect. Production harnesses optimize capability under constraints; OpenHarness optimizes a different objective.", "Cost minimization — the cheapest harness to run at scale.", "Speed of inference — the lowest-latency agent loop in the roster." ], "answer_index": 1, "rationale": "OpenHarness optimizes for inspectability under reproducibility. The diagnostic test: can a researcher point at the exact mechanism that produced this decision, vary it, and measure the effect? If the answer is 'it's buried in the framework,' the harness has failed. A production harness optimizes for capability under constraints (cost, latency, safety). OpenHarness optimizes a different objective function, not a weaker version of the same one." }, { "id": "q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the four parts of OpenHarness's auto-compaction mechanism.", "options": [ "Token counter, retry loop, model call, response cache.", "Trigger predicate (declared condition for when), selection policy (what summarized/preserved/dropped), summarization step (model-based, inputs and outputs logged), non-destructive record (original retained for analysis).", "Read path, write path, eviction policy, garbage collector.", "Context window, sliding window, summarizer model, output buffer." ], "answer_index": 1, "rationale": "The four parts: (1) trigger predicate — a declared condition (token threshold, message count, custom predicate) that decides when compaction fires; (2) selection policy — which messages/tool results get summarized, preserved verbatim, or dropped, named and parameterized; (3) summarization step — the model-based summarization call with its own logged inputs and outputs; (4) non-destructive record — the original context is retained for analysis even after the compacted version is what the model sees." }, { "id": "q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is ohmo, and what role does it play?", "options": [ "Ohmo is OpenHarness's sandboxing layer — it isolates bash/exec calls.", "Ohmo is OpenHarness's built-in personal agent — the reference demonstration that exercises every component so a researcher can vary a parameter and measure the effect. ohmo is to OpenHarness what Pi (DD-01) is to the minimal-harness category.", "Ohmo is OpenHarness's permission model — it gates tool calls.", "Ohmo is OpenHarness's observability dashboard — it renders the decision log." ], "answer_index": 1, "rationale": "ohmo is OpenHarness's built-in personal agent. It exercises every component so a researcher can run it, vary a parameter (compaction threshold, memory tier, tool set), and measure the effect on a benchmark without rebuilding the harness. ohmo is to OpenHarness what Pi (DD-01) is to the minimal-harness category: the readable instance that proves the architecture." }, { "id": "q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are writing a paper claiming 'a different summarization policy preserves more security-critical instructions during compaction.' Which harness do you run the experiment against, and why?", "options": [ "Any production harness with rich telemetry — the telemetry will let you measure the effect.", "OpenHarness — because the summarization step is a first-class, named, parameterized component with logged inputs and outputs, you can swap the policy and audit what the summarizer saw versus what it produced. Only against OpenHarness is the mechanism studiable rather than merely present.", "DD-21 (Tau) — because it has the most complete session tree.", "DD-01 (Pi) — because it has the most legible code." ], "answer_index": 1, "rationale": "The experiment requires a harness where the summarization step is individually examinable. OpenHarness implements compaction as a designed four-part mechanism: trigger, selection policy, summarization step, non-destructive record. You can swap the selection policy, run the experiment, and audit what the summarizer was shown versus what it produced. Production harnesses entangle compaction with session store, token accounting, and retry logic — examinable only with instrumentation. This is why Module 3 cites OpenHarness as the compaction reference." }, { "id": "q05", "bloom": "application", "type": "multiple_choice", "prompt": "You need to ship an agent that calls bash to install packages. Can you use OpenHarness as the production harness, and what is the load-bearing reason?", "options": [ "Yes — OpenHarness's inspectability means you can audit every bash call.", "No — OpenHarness scores 1/5 on Module 5 (Sandbox): bash/exec run without containment. The inspectability is a research property, not a security property. A visible bash call is not a safe bash call. Use a harness with the security modules implemented.", "Yes — OpenHarness is academic-grade, which is higher quality than production-grade.", "Yes, but only if you enable ohmo's personal-agent mode." ], "answer_index": 1, "rationale": "OpenHarness is the academic baseline, not a production harness. Its sandbox is minimal (Module 5: 1/5), its permission model is below floor (Module 6: 2/5), and its security model is absent (Module 11: 1/5). If you ship it, you are running an agent with no isolation, no gates, and no security model. Inspectability is a research property ('I can see what happens'), not a security property ('what happens is safe'). For production with bash, use a harness with sandboxing implemented." }, { "id": "q06", "bloom": "application", "type": "multiple_choice", "prompt": "A colleague says 'OpenHarness scores 30/60, so it is the weakest harness in the roster.' How do you correct the interpretation?", "options": [ "You agree — 30/60 is below the pass threshold and OpenHarness is the weakest.", "You correct the interpretation: the rubric scores production-readiness across all eleven modules. OpenHarness is deliberately not optimizing for all dimensions — it optimizes for inspectability under reproducibility. The score reflects what it is NOT (production-ready), not a failure of what it IS (the cleanest mechanism reference in the roster). Same pattern as DD-01 (Pi).", "You argue the rubric is wrong and OpenHarness should be re-scored.", "You argue the score is irrelevant and should be ignored." ], "answer_index": 1, "rationale": "As with DD-01 (Pi), the absolute score is low because the rubric scores production-readiness across all dimensions. OpenHarness is not trying to be production-grade across all dimensions — it is deliberately inspectable. The score reflects what OpenHarness is not (production-ready), not a failure of what it is (the cleanest mechanism reference in the roster). Its value is pedagogical and methodological: it is the harness that makes every other harness's claims falsifiable. You cannot falsify a claim against a black box; you can falsify a claim against OpenHarness." }, { "id": "q07", "bloom": "application", "type": "multiple_choice", "prompt": "You want to investigate whether a malicious tool output could write a poisoned fact to semantic memory that survives compaction. Which harness do you use, and which property makes the investigation possible?", "options": [ "Any production harness — memory poisoning is a general problem.", "OpenHarness — its memory is implemented as separate tiered stores (working, episodic, semantic) with declared interfaces, so you can trace a write across tiers without production conflation blurring the path. The question is only answerable against a harness where the tiers are separate stores.", "DD-21 (Tau) — because it has the most complete event union.", "DD-01 (Pi) — because it has the fewest moving parts." ], "answer_index": 1, "rationale": "Memory-tier poisoning — can a write to semantic memory survive compaction? — is only answerable against a harness where the memory tiers are separate stores with declared interfaces. OpenHarness pays the integration friction to buy this property. Production harnesses conflate the tiers into one buffer (faster, cheaper, blurred), so the read path and write path are not individually auditable. This is the Module 4.3 (write-gating) property that production harnesses usually blur." }, { "id": "q08", "bloom": "application", "type": "multiple_choice", "prompt": "You are benchmarking two compaction policies and want to know whether either policy changed the outcome of the run. Which property of OpenHarness's design makes this answerable?", "options": [ "The decision log is human-readable.", "The non-destructive record — the original context is retained for analysis even after the compacted version is what the model sees. You can replay the run with the original context and compare.", "ohmo's personal-agent mode.", "The tool count is research-minimal." ], "answer_index": 1, "rationale": "The non-destructive record (Part 4 of the four-part mechanism) is the load-bearing detail. The original context is retained for analysis even after the compacted version is what the model sees on the next turn. The on-disk truth is never rewritten, only the in-context view is swapped. This lets a researcher ask 'did compaction change the outcome?' — a question a production harness cannot answer without instrumentation. The design mirrors DD-21's CompactionEntry with replaces_entry_ids and Module 8's event-sourcing principle." }, { "id": "q09", "bloom": "application", "type": "multiple_choice", "prompt": "A team is choosing a harness. They need (a) to ship to production with bash tool calls and (b) to publish a reproducible research paper on compaction. Which harnesses do they pick?", "options": [ "OpenHarness for both — it is the cleanest implementation.", "OpenHarness for the research paper (inspectability under reproducibility), and a production harness with sandboxing (e.g., the Agents SDK DD-11 for the 7-provider sandbox abstraction, plus NemoClaw DD-09 for governance) for the production deployment. Different objective functions require different harnesses.", "A production harness for both — research can be done anywhere.", "Pi (DD-01) for both — it is the most legible." ], "answer_index": 1, "rationale": "OpenHarness and production harnesses optimize for different objective functions. OpenHarness is the right choice for the research paper because it is the cleanest mechanism reference and the only harness where compaction is studiable rather than merely present. For production with bash tool calls, use a harness with the security modules implemented — the Agents SDK (DD-11) for sandboxing, NemoClaw (DD-09) for governance. The methodological rule: treat OpenHarness as a research and pedagogical tool, never as a production harness." }, { "id": "q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "OpenHarness and DD-21 (Tau) are the two most inspectable harnesses in the roster. What is the axis that distinguishes them?", "options": [ "OpenHarness is for research; Tau is for production.", "OpenHarness optimizes for legibility-of-BEHAVIOR (what the harness did — decisions logged at source with inputs). Tau optimizes for legibility-of-CODE (what the harness is — code structure, session tree, machine-readable event union). Both inspectability-first, complementary properties. The shared design is the non-destructive record.", "OpenHarness uses compaction; Tau does not.", "OpenHarness is academic; Tau is commercial." ], "answer_index": 1, "rationale": "The inspectability axis: OpenHarness = legibility-of-behavior (what the harness did — decisions logged at source with inputs, human-readable). Tau = legibility-of-code (what the harness is — code structure, session tree, machine-readable event union). Both are inspectability-first but optimize for complementary properties. The shared design is the non-destructive record — both refuse to overwrite the on-disk truth. OpenHarness retains the original context; Tau retains the CompactionEntry replacement chain. The event-sourcing principle (Module 8) as a convergent design." }, { "id": "q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does OpenHarness keep its tool count research-minimal, given that production harnesses have the opposite incentive (more tools = more capability)?", "options": [ "Because the developers could not build more tools.", "Because a large tool set is noise for research — it confounds 'what did the model do?' with 'what did the tools enable?' OpenHarness keeps the surface small so the model's decisions are the signal, not the tool surface.", "Because the rubric penalizes tool count.", "Because more tools would break the compaction mechanism." ], "answer_index": 1, "rationale": "The tool set is research-minimal: small, stable, and chosen so capability claims can be attributed to the model rather than to a rich tool surface. This is the opposite of a production harness's incentive (more tools = more capability = better benchmarks). For research, a large tool set is noise — it confounds the question 'what did the model do?' with 'what did the tools enable?' OpenHarness keeps the surface small so the model's decisions are the signal." }, { "id": "q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "OpenHarness logs every decision at source with the inputs that produced it. Why is this stronger than after-the-fact trace reconstruction, and what is the precise distinction?", "options": [ "It is faster to render.", "It uses less disk space.", "'We can reproduce the run' (trace reconstruction) means you can replay the sequence. 'We can defend why the run went the way it did' (decision-at-source logging) means you can point at the decision AND the inputs that produced it — a stronger epistemic property needed for falsifiable research claims.", "It is more secure against tampering." ], "answer_index": 2, "rationale": "The distinction: 'we can reproduce the run' (trace reconstruction) means you can replay the sequence — weaker. 'We can defend why the run went the way it did' (decision-at-source logging) means you can point at the decision AND the inputs that produced it — stronger. OpenHarness logs every decision at the point it is made, with the inputs that produced it, as a first-class record. This is the epistemic property research needs: you can falsify a claim by pointing at the exact decision and inputs." }, { "id": "q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "DD-01 (Pi) and OpenHarness (DD-13) both have low absolute scores but are both described as 'valuable.' What is the shared pattern, and why is it load-bearing for how you read the roster?", "options": [ "Both are weak harnesses that should be avoided.", "Both are early prototypes that have been superseded.", "Both share the low-score-for-legibility pattern: a low absolute number that reflects 'not trying to be production,' not a design failure. Pi is the most legible minimal harness; OpenHarness is the most inspectable academic harness. The pattern is load-bearing because it teaches you to read the score against the objective function, not against the maximum.", "Both are academic harnesses from the same lab." ], "answer_index": 2, "rationale": "The shared pattern: a low absolute score that reflects 'not trying to be production,' not a failure of the design. Pi (25/60) is the most legible minimal harness — the readable instance of the harness category. OpenHarness (30/60) is the most inspectable academic harness — the cleanest mechanism reference in the roster. The pattern is load-bearing for how you read the roster because it teaches you to interpret the score against the objective function the harness optimizes for, not against the production-readiness maximum. A harness optimized for inspectability should not be scored as if it failed at production." }, { "id": "q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "OpenHarness's decision logs are human-readable. DD-21 (Tau) and DD-14 (Mastra) produce machine-readable structured events. What is the practical consequence, and what would OpenHarness need to do to close the gap?", "options": [ "No consequence — human-readable and machine-readable are interchangeable.", "OpenHarness would need to add a sandbox.", "Human-readable logs are inspectable by a person but not analyzable at scale (no structured query, no automated alerting, no pipeline integration). To close the gap, OpenHarness would need to emit a machine-readable decision-log schema — this is the third thing to fix if misused as production, and it would bring Module 10 observability up to the DD-21/DD-14 event standard.", "OpenHarness would need to add more tools." ], "answer_index": 2, "rationale": "Human-readable logs are inspectable by a person but not analyzable at scale — no structured query, no automated alerting, no pipeline integration, no replay against a benchmark harness. Machine-readable structured events (DD-21's event union, DD-14 Mastra's observability standard) are analyzable at scale. To close the gap, OpenHarness would need to emit a machine-readable decision-log schema. This is the third thing to fix if misused as production, and it is the gap that keeps OpenHarness at 4/5 (not 5/5) on Module 10 Observability." }, { "id": "q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "OpenHarness's non-destructive compaction record retains the original context even after the compacted version is what the model sees. Which other harness in the roster shares this design, and what course module formalizes the principle?", "options": [ "No other harness shares this design — it is unique to OpenHarness.", "DD-21 (Tau) shares the design via CompactionEntry with replaces_entry_ids — the on-disk truth is never rewritten, only the in-context view is swapped. Module 8 (State/Checkpointing) formalizes the underlying event-sourcing principle: the log is the source of truth, and views are projections over it.", "DD-01 (Pi) shares the design via its minimal loop.", "DD-09 (NemoClaw) shares the design via its governance layer." ], "answer_index": 1, "rationale": "DD-21 (Tau) shares the non-destructive record design via CompactionEntry with replaces_entry_ids: the on-disk truth is never rewritten, only the in-context view is swapped. OpenHarness retains the original context for analysis; Tau retains the replacement entry chain. Module 8 (State/Checkpointing) formalizes the underlying event-sourcing principle: the log is the source of truth, and the in-context view is a projection over it. This convergent design — two harnesses independently arriving at non-destructive compaction — is evidence the principle is load-bearing for inspectability." } ] }