120 min · Design · Build · Benchmark · Publish
Prerequisites: Pillars 0, 1, 2 complete
Offensive (Pillar 1) + AppSec (Pillar 2) + Design/Security (Pillar 0) assembled into one shippable harness. The deliverable is a publishable benchmark, not a script.
Three properties separate a real harness from a prompt wrapper: scope enforcement, evidence chains, adversarial-output defense.
Twenty minutes of design saves sixty of rework. Every decision is load-bearing: domain, scope, memory, tools, evidence, autonomy, threat model.
| Track | Target | Deliverable | Scope |
|---|---|---|---|
| Bug bounty | Authorized program / vuln lab | Triaged findings + PoC, submission-ready | Engagement boundary (hosts, endpoints) |
| AppSec gate | Codebase in CI | Pass/fail gate + findings report | Repository, dependency set, code paths |
Architecture is identical: scope middleware, memory, tools, evidence, triage, report. Domain changes the tool suite and report format, not the skeleton.
You do not write "stay in scope" in the system prompt and trust the LLM. You write a scope-check function that intercepts every tool call and rejects any target outside the allow-list.
Blocked calls return a structured result the LLM sees as "blocked, here is why" — never a silent failure. Defense in depth: tool checks AND registry checks.
Without memory, every scan cold-starts. With memory, each finding enriches the target-state model; the next tool call reasons over what was already found.
context_for(tool, args) loads host state + prior findings into LLM context before the tool runs. record() writes findings + evidence back after.
A recon tool discovers /admin; the next active-probe call sees /admin in context without being told. That is the difference between a harness that builds a picture and a script firing isolated requests.
| # | Tool | Role | high_impact |
|---|---|---|---|
| 1 | Recon | Discovers target surface | False |
| 2 | Active probe | Crafted requests / mutations | True |
| 3 | Static analyzer | Linter / SAST / pattern match | False |
| 4 | Exploit verifier | Controlled PoC to confirm | True |
| 5 | Evidence recorder | Captures request/response/diff | False |
Each: Pydantic input schema · extract_target · high_impact flag · run(args, trace_id). Uniformity is what makes the middleware hold.
Build order: scope, memory, tools, evidence, triage, report. Each layer depends on the one below.
A finding without evidence is an opinion. Every confirmed finding links: finding → evidence (sha256) → tool call → log line (trace_id).
The hash lets a reviewer verify the recorded output matches what the tool produced. The trace ID is the join key that makes the chain replayable. This is what makes a finding submissible to a bounty program or acceptable to an AppSec gate reviewer.
The harness is an agentic system — it is itself an attack surface. Your threat model must address prompt injection from tool output, tool misuse, excessive agency, memory poisoning, and unbounded consumption.
Defense is structural: tool output is data (never instructions), scope middleware on every tool, autonomy policy + approval gates, schema-validated memory writes, global call budget + per-tool rate limits.
Four tests convert "my harness is safe and effective" into falsifiable, publishable evidence.
| Test | Measures | Pass bar |
|---|---|---|
| 1. Vuln lab recall | Of N known bugs, how many confirmed? | Effectiveness — report the number |
| 2. OOS call block | Inject OOS targets into tool-call args | 100% blocked — any success = scope bug |
| 3. Served injection | Serve response with prompt injection | Behavior unchanged (data, not instructions) |
| 4. InjecAgent | % injection attempts that failed | > 90% = publishable security claim |
Client report (HTML + JSON) + published benchmark = two portfolio assets. Report proves it works; benchmark proves it is safe.
The benchmark is the portfolio asset. Recall, OOS block rate, injection result, InjecAgent score, report link.
GitHub README · LinkedIn post · Deepthreat.ai demonstration asset. The harness is the engine; the benchmark is the proof.
Next: Capstone 2 — same architecture, smart contract or cloud domain, same publishable benchmark.