# Module S02 — Bug Bounty Harness Engineering **Course**: 2A — Building AI Harnesses for Cybersecurity **Module**: S02 — Bug Bounty Harness Engineering **Duration**: 90 minutes **Level**: Senior Engineer and above **Prerequisites**: S00, S01 complete > *A harness that operates across multi-day engagements with persistent state, structured evidence, and client-ready output. Persistent engagement memory, the offensive tool suite, evidence chain engineering, and the signal/noise triage pipeline.* --- ## Learning Objectives 1. Design and implement a persistent target-state memory store using ChromaDB or Qdrant, and defend the choice against alternatives. 2. Wrap offensive tools (nmap, httpx, nuclei, ffuf) as harness tools with Pydantic schemas, scope-check wrappers, and rate limiting. 3. Implement an evidence logger as harness middleware that produces a tamper-evident, scope-referenced evidence chain. 4. Build a signal/noise triage pipeline that filters raw scanner output into a prioritized, deduplicated finding list. 5. Measure triage performance (precision, recall, F1) against a labeled vulnerability dataset. --- # S02.1 — Persistent Engagement Memory Bug bounty engagements span days or weeks. The context window is a session primitive, not a persistence mechanism — it fills, it compacts, and when it compacts, discovered subdomains from day one disappear. A bug bounty harness needs memory that survives across sessions: discovered infrastructure, prior vulnerability hypotheses, exploitation attempts and their results. This is the concrete instantiation of the offensive state machine from S01.1. S01.1 defined the loop's *shape* (observe → orient → decide → act, with scope enforcement at every transition); S02 builds the *substrate* that loop reads and writes. Without persistent memory, the state machine is amnesiac — every session re-discovers the target from zero and re-trips the rate limits S00.3 warned about. With it, day two begins where day one ended. ## The target-state schema The engagement memory is not a chat log. It is a structured representation of the target's attack surface and the harness's accumulated knowledge of it: ```typescript interface TargetState { // Infrastructure discovered domains: DomainRecord[]; subdomains: SubdomainRecord[]; open_ports: PortRecord[]; services: ServiceRecord[]; endpoints: EndpointRecord[]; // Engagement knowledge credentials_found: CredentialRecord[]; vulnerability_hypotheses: HypothesisRecord[]; exploitation_attempts: AttemptRecord[]; confirmed_findings: FindingRecord[]; // Metadata last_updated: string; engagement_id: string; scope_ref: string; } ``` Every field is a structured record, not free text. The harness queries this store before each recon step ("what subdomains do I already know?") and updates it after each tool call. This is the attack graph from S01.1, expanded to cover the full engagement surface — and, as we will see in S03.1, it is the flat precursor to the dependency-weighted attack graph a red team engagement requires. ## Vector store selection: ChromaDB vs Qdrant vs LanceDB The persistent store for offensive memory is a vector database — not because we need embeddings of prose, but because we need semantic similarity search across findings, hypotheses, and target records. ("Have I seen a similar vulnerability pattern before on this engagement?") | Store | Strengths | Weaknesses | When to choose | | --- | --- | --- | --- | | **ChromaDB** | Python-native, simple embedding API, good for prototyping | Limited scale, single-node default | Engagement-scoped memory (<1M records); most bug bounty harnesses | | **Qdrant** | Production-grade, horizontal scale, rich filtering | More setup overhead | Large-scale continuous monitoring; fleet engagements | | **LanceDB** | Columnar storage, good for analytical queries over findings | Newer ecosystem, less tooling | When you need aggregation queries ("count findings by severity over time") | For most bug bounty harnesses, ChromaDB is the right default: it runs in-process, needs no server, and handles the engagement-scoped volume (typically tens of thousands of records per target) without issue. Move to Qdrant when you are running continuous monitoring across many targets simultaneously. The reasoning is worth making explicit, because "it's simpler" is a hand-wave that produces regret. Three concrete factors: 1. **Process model matches the engagement lifecycle.** ChromaDB's embedded model means the store lives and dies with the engagement directory — collections materialize on disk under that folder, and when the engagement closes you archive or destroy the directory. No server to provision, no credentials to rotate, no orphaned Qdrant instance on a forgotten box. The engagement's memory is *co-located* with its scope file, evidence chain, and report — one directory, one engagement, one blast radius. Qdrant's server model is right when the store outlives any single engagement, which is the fleet case, not the per-engagement case. 2. **Query patterns fit single-node.** Bug bounty memory queries are two kinds: exact-key lookups ("give me the record for `api.target.com:443`") and nearest-neighbor over a small corpus ("which prior hypotheses resemble this finding?"). Both are within ChromaDB's single-node envelope at tens of thousands of records. Qdrant's distributed planner pays off at millions of records across shards — fleet scale a single engagement will not reach unless deduplication has failed (next sub-section). 3. **Operational surface area is a security property.** Every additional service in an offensive harness is an additional thing a compromised agent can attack or exfiltrate (S03.3). A Qdrant server reachable from the agent container is a network endpoint with credentials; a ChromaDB embedded store is a file on disk with filesystem permissions. Fewer reachable services means a smaller inbound attack surface. This is judgment, not a benchmark — but where S01.3 established the agent may be compromised by target content, reducing reachable services is a defensible default. The caveat: ChromaDB's single-node model is a ceiling, not a floor. If you move a harness from per-engagement work to continuous monitoring, that is the moment to revisit Qdrant — and the migration is mechanical (embeddings and schemas port directly) *if* you designed memory access through a repository interface rather than scattering raw ChromaDB calls through the tool code. Otherwise it is a rewrite. Design through a repository interface. ## The ChromaDB memory store, concretely The store is a typed repository with one collection per record type, each enforcing the deduplication contract at write time: ```python import chromadb from chromadb.config import Settings from datetime import datetime, timezone class EngagementMemory: """Persistent target-state memory for a single engagement. One instance per engagement directory. ChromaDB is embedded (PersistentClient) — no server, no network port. Collections are typed: one per record kind, each with its own dedup key. """ def __init__(self, engagement_dir: str, engagement_id: str, scope_ref: str): self.engagement_id = engagement_id self.scope_ref = scope_ref self.client = chromadb.PersistentClient( path=f"{engagement_dir}/memory", settings=Settings(anonymized_telemetry=False), # no phoning home ) self.subdomains = self.client.get_or_create_collection( "subdomains", metadata={"hnsw:space": "cosine"}, ) self.findings = self.client.get_or_create_collection("findings") self.hypotheses = self.client.get_or_create_collection("hypotheses") def upsert_subdomain(self, record: dict) -> dict: """Write-time dedup: same fqdn updates, never duplicates.""" fqdn = record["fqdn"] existing = self.subdomains.get(ids=[fqdn]) ts = datetime.now(timezone.utc).isoformat() if existing["ids"]: # Merge: preserve first_seen, update last_seen and source list prev = existing["metadatas"][0] record = { **prev, **record, "first_seen": prev["first_seen"], "last_seen": ts, "sources": sorted(set(prev.get("sources", [])) | set(record.get("sources", []))), "scan_count": prev.get("scan_count", 0) + 1, } else: record = {**record, "first_seen": ts, "last_seen": ts, "scan_count": 1} self.subdomains.upsert( ids=[fqdn], metadatas=[record], documents=[self._summarize(record)], # for semantic search ) return record def similar_findings(self, finding_summary: str, k: int = 5) -> list[dict]: """Semantic nearest-neighbor over past findings. Used by the triage pipeline (S02.4) and plan correction (S03.2): 'have we seen something like this before, and what did we conclude?'""" res = self.findings.query(query_texts=[finding_summary], n_results=k) return res["metadatas"][0] if res["metadatas"] else [] def _summarize(self, record: dict) -> str: # Reader-actor boundary (S01.3): only structured fields are embedded, # never raw target output, so an injected payload cannot poison the # embedding space directly. return " ".join(f"{k}:{v}" for k, v in record.items() if isinstance(v, (str, int, float))) ``` Two decisions in that snippet are load-bearing. First, `anonymized_telemetry=False` — an offensive harness must not phone home to a third-party telemetry endpoint, for operational stealth (S00.3) and because the payload could leak engagement metadata. Second, `_summarize` is the reader-actor boundary made concrete: raw tool output is parsed into structured fields upstream, and only those fields are embedded. An injection payload in an HTTP response body never reaches the embedding model as raw text. ## The accumulation vs. deduplication problem Offensive memory wants to accumulate — every recon scan adds subdomains, every port scan adds services, every exploit attempt adds a result. A naive implementation accumulates noise: after a week, the memory holds 50,000 records, half duplicates or stale, and the harness spends more time managing memory than finding vulnerabilities. The solution is **structured deduplication at write time**: every record is checked against existing records, and if a duplicate exists (same subdomain, same port, same finding signature), the existing record is updated, not duplicated. The store grows with *unique* knowledge, not raw tool output volume. This is what makes the store queryable at decision time: when the triage model in S02.4 asks "have I seen a finding matching this signature before?", the answer is trustworthy only if each unique finding exists once. A store with twelve near-duplicate records of the same exposed `.git` directory produces twelve conflicting answers — one confirmed, one false-positive, ten undispositioned — and the triage model reconciles them under uncertainty. Dedup at write time collapses that to one record with an accumulated disposition. The cost is a hash lookup per write; the benefit is that downstream queries reason over a set, not a multiset. The dedup key is per record type and must survive rescans: `fqdn` for subdomains, `host:port:proto` for ports, `target:signature` for findings where `signature` is a stable hash of (template-id, matched-url, matched-pattern). A signature that includes a volatile field defeats dedup, because every rescan produces a "new" finding. Hash the stable signal, not the noise. ## A multi-day engagement, watched through memory To make accumulation concrete, walk a hypothetical engagement against `*.example-corp.com`: - **Day 1 — recon seeding.** `subfinder` and `amass` write 240 subdomains (fresh inserts, scan_count=1). `httpx` probes them, adding 180 `EndpointRecord`s. `nmap` against the ten most interesting hosts adds 320 `PortRecord`s. End of day one: ~740 records, all unique. - **Day 2 — rescan and web testing.** The operator reruns `subfinder` as a sanity check. Without dedup this adds another 240 records — 240 duplicates. With dedup, the 240 existing records upsert (`scan_count` increments, `last_seen` updates) and three genuinely *new* subdomains appear as fresh inserts. The store grows by 3, not 240. `ffuf` produces 600 raw paths; after dedup against `endpoints`, 412 are new. End of day two: ~1,155 records. Day-three resume begins with a memory query, not a rescan. - **Day 3 — vulnerability testing.** `nuclei` fires against the 412 live endpoints and returns 200 raw findings. These flow into the triage pipeline (S02.4), which dedups, applies the secondary model, and surfaces 3 actionable reports. Across three days, ~1,300 records — not the ~2,500 a naive accumulator would hold, and every record unique and dispositioned. The ratio to watch is `scan_count` to unique records. If `scan_count` climbs but unique records are flat, the harness is rescanning stable infrastructure. If unique records climb faster than scans, a dedup key includes a volatile field. That ratio is the memory store's health metric. ## Memory poisoning via deceptive target responses S01.3 established that target content may contain injection payloads. The same risk applies to memory: if the harness writes a deceptive target response into the engagement memory verbatim, the poisoned data persists across sessions and degrades all future decisions. The defense is the same as S01.3's reader-actor separation: the model that ingests raw target output produces a structured summary; only the structured summary (not the raw response) is written to memory. An injection in the raw response can affect the summary's content, but it cannot inject arbitrary records into the structured store. The `_summarize` method above is this boundary: the embedding is over structured fields, and the raw response lives only in the evidence chain (S02.3), retained as evidence but never re-ingested as reasoning input. A subtler poisoning path: a target that *correctly* reports its own state can still poison memory if the harness writes the target's self-description as ground truth. A banner that says `Server: Apache/2.4.49` is the target's claim, not a verified fact; writing it as `service.version = "2.4.49"` elevates a claim to a finding. The cure is to tag every field with its epistemic source: `observed` (the harness measured it — a port is open because nmap connected), `claimed` (the target asserted it — a version banner), or `inferred` (the model hypothesized it). The triage model in S02.4 weights `observed` above `claimed` above `inferred`, and a finding built entirely on `claimed` fields carries lower confidence. This is judgment, not a standard — but it is the judgment that prevents a banner-spoofing target from steering the engagement. --- # S02.2 — The Bug Bounty Tool Suite The offensive tool suite is not a list of binaries. It is a set of **harness tools** — each wrapped with a Pydantic schema, a scope-check wrapper, rate limiting, and structured output. The wrapping is what turns a CLI scanner into a harness tool the model can call safely. This is where the scope file from S00.1 meets the offensive state machine from S01.1. The scope file is the *input* — the machine-checkable authorization boundary. The state machine is the *consumer* — it calls tools to observe and act. The harness tool is the *joint*: the only path from the state machine to the network, and at that joint it enforces scope (S01.2), rate limits (S00.3), and evidence capture (S02.3) on every call. A tool reachable without wrapping is a hole in the authorization chain. ## The four tiers | Tier | Tools | Purpose | | --- | --- | --- | | **Recon** | nmap, amass, subfinder, httpx, nuclei | Discover and enumerate the target's attack surface | | **Web application** | Burp Suite (headless), sqlmap, ffuf, gobuster | Test web application vulnerabilities | | **Evidence** | Playwright (screenshots), request/response capture, NVD CVE lookup, CVSS calculator | Capture and enrich findings | | **Report** | Structured finding generator | Produce client-ready output | ## Wrapping a tool as a harness tool Every offensive tool follows the same wrapping pattern. The example here is the full nmap tool — input/output schemas, the four-step body (scope check, rate limit, execute, parse), and automatic evidence logging. This is the template every other tool in the suite copies. ```python from pydantic import BaseModel, Field from typing import Literal import asyncio, json, shlex class NmapScanInput(BaseModel): target: str = Field(..., description="The host to scan") scan_type: Literal["fast", "default", "intense"] = "default" ports: str | None = Field(None, description="Port range, e.g. '1-1000'") rate: int = Field(100, description="Max packets/sec; capped by RoE in scope file") class NmapScanOutput(BaseModel): host: str open_ports: list[dict] services: list[dict] raw_output: str # retained for evidence but not for model context duration_seconds: float scope_ref: str # carried into the output so the caller can cite it async def nmap_scan( input: NmapScanInput, scope, # ScopeFile from S00.1 / S01.2 rate_limiter, # per-RoE limiter from the scope file evidence_logger, # S02.3 chain executor, # the subprocess runner (single chokepoint) ) -> NmapScanOutput: # 1. Scope check (belt-and-suspenders; middleware does this too). # scope_ref is stamped here so every downstream record can cite it. scope_ref = scope.stamp_ref(input.target, "scan") if not scope.is_in_scope(input.target, "scan"): # A blocked call is evidence too — log it. evidence_logger.record( tool="nmap", target=input.target, request="BLOCKED: out_of_scope", response="", scope_ref=scope_ref, ) return NmapScanOutput( host=input.target, open_ports=[], services=[], raw_output="BLOCKED: out_of_scope", duration_seconds=0, scope_ref=scope_ref, ) # 2. Rate limit: min(RoE cap, requested). Stealth control (S00.3) # as much as a legal one — never exceeds RoE even if asked. effective_rate = min(input.rate, scope.roe.max_packets_per_second) await rate_limiter.acquire(effective_rate) # 3. Execute through the executor — never subprocess directly. The # executor is the chokepoint where middleware, scope, and evidence # logging are wired in. subprocess.run() bypasses all three. cmd = ["nmap", "-sS", f"--max-rate={effective_rate}"] if input.scan_type == "fast": cmd += ["-T4", "-F"] if input.ports: cmd += ["-p", input.ports] cmd += [input.target] raw, duration = await executor.run(cmd, scope_ref=scope_ref) # 4. Parse. Model reasons over `parsed`; `raw` goes to evidence only # (reader-actor boundary, S01.3). parsed = parse_nmap_output(raw) # 5. Evidence — automatic, not optional. # tool call produces an evidence record linked to its authorization. evidence_logger.record( tool="nmap", tool_version=executor.tool_version("nmap"), target=input.target, request=shlex.join(cmd), response=raw, scope_ref=scope_ref, ) return NmapScanOutput( host=input.target, open_ports=parsed.open_ports, services=parsed.services, raw_output=raw, # retained; the model sees the structured fields duration_seconds=duration, scope_ref=scope_ref, ) ``` **Key design points:** the model sees the Pydantic schema, not CLI construction; the scope check runs in both the middleware and the tool (defense in depth); rate limiting is capped to the RoE maximum even if the model requests more; raw output goes to evidence only, never to model context (reducing injection risk and context bloat); and the executor is the single chokepoint — a tool that calls `subprocess.run` directly has left the authorized path. ## The tradeoffs of the wrapping pattern The wrapping is not free. **Latency:** every call carries a scope check, a rate-limiter acquire, a subprocess spawn (the dominant cost), a parse, and an evidence write — overhead negligible next to the subprocess, but the rate limiter is the *point*: a wrapped tool cannot exceed the RoE cap, where an unwrapped `subprocess.run` could. "Optimizing" by skipping the wrapper optimizes away the legal control. **Schema rigidity:** a Pydantic schema forces structured intent, preventing `nmap --script=malicious-category` or smuggled arguments — but novel invocations the schema did not anticipate cannot be expressed. The cure is not to loosen the schema; add a typed field when a legitimate need appears (a schema accreting twenty `Optional` fields does too much and should be split). **Raw output retention:** retaining `raw_output` in evidence but not in model context is the central tension of the reader-actor boundary (S01.3) — the model needs *enough* to reason, the chain needs *all* to be reproducible. The split: structured fields to the model, full raw to evidence, a one-way arrow via a parser the harness controls. A model that reads raw output directly is one prompt-injection away from being steered. ## Tool design for offensive contexts Three principles distinguish offensive tool design from general tool design. **Idempotency:** scanning the same target twice produces the same result (modulo target changes), making retry safe and enabling deduplication — it is also what makes the write-time dedup from S02.1 sound, since a tool returning different `signature` fields for the same state defeats dedup and re-pollutes the store. **Rate limiting that doesn't trigger WAFs:** offensive tools respect rate caps not just for legal reasons (S00.3) but for operational ones — tripping a WAF gets your IP banned and ends the engagement. Rate limiting is a stealth control as well as a legal one; the cap comes from the RoE and the tool caps itself to that maximum regardless of what the model requests. **Scope-check wrappers on every outbound call:** not just in the middleware — in the tool itself. If the middleware is bypassed, the tool's own scope check is the last line of defense, and `scope_ref` is stamped *before* the scope decision so even a blocked call produces an evidence record citing the authorization it would have needed. ## Worked scenario: a recon pass through the suite The four tiers chain. A day-one recon pass against a fresh target: `subfinder`/`amass` enumerate subdomains (upserting into the `subdomains` collection via S02.1's repository, rate-limited to the passive-recon cap); `httpx` probes them for live HTTP services (each live host becomes an `EndpointRecord`, dead hosts retained with `state=dead`); `nmap` runs against the live hosts (the wrapper stamps `scope_ref`, rate-limits, parses into `PortRecord`s and `ServiceRecord`s, raw XML to evidence only); then `nuclei` fires its templates against the live endpoints and emits raw findings. That last handoff is the load-bearing one. Recon output is structured fact (a port is open because nmap measured it). Scanner output is *candidate* signal (nuclei matched a template, but the match may be a false positive). The raw findings do *not* go to memory directly — they go to the triage pipeline in S02.4, the only path by which a raw finding becomes a `FindingRecord`. The triage pipeline promotes candidate signal to confirmed finding under a separate model's judgment. --- # S02.3 — Evidence Chain Engineering Every finding must have a reproducible, client-presentable trail. The evidence chain is the harness's legal protection (S00.2) and the client's compliance requirement. The link to S00.2 is direct: S00.2 defined *what* evidence you must preserve (timestamp, exact request, exact response, tool and version, scope reference) and *what* you must not retain (PII, credentials, customer data). S02.3 builds the mechanism that enforces both automatically. A harness that relies on the operator to remember to log evidence will produce an incomplete chain the one time a finding is disputed. The evidence logger is middleware, not a discipline. ## The evidence record schema ```typescript interface EvidenceRecord { // Identification id: string; trace_id: string; session_id: string; // What happened timestamp: string; // UTC ISO 8601 tool: string; tool_version: string; target: string; action: string; // What was sent and received request: string; // exact command or HTTP request response: string; // exact output or HTTP response // Legal anchor scope_ref: string; // which scope entry authorized this call // Finding linkage finding_id: string | null; cvss_draft: string | null; // Retention retention_class: "public" | "redacted" | "restricted"; retention_expires: string | null; } ``` Every field serves a legal or operational purpose. The `scope_ref` is the load-bearing field — it ties this evidence record to the specific authorization that permitted the call. The `retention_class` is what prevents the store from becoming the compliance liability S00.2 warned about: `restricted` records (real PII or credentials) carry an expiry and are destroyed on schedule, not retained indefinitely. ## Append-only, tamper-evident The evidence log is **append-only**. No overwriting, no deletion of individual records (destruction happens per the retention policy, batch-level, with the destruction itself recorded). Every attempt is captured, including failures — a failed exploit is as much evidence as a successful one. Tamper-evidence is achieved through **hash chaining**: each record includes a hash of the previous record. Tampering with any record breaks the chain: ```python import hashlib, json from datetime import datetime, timezone class EvidenceChain: """Append-only, hash-chained evidence log. One chain per engagement. Records are linked by SHA-256 over the canonical JSON of (record_data + previous_hash). Tampering with any record breaks the chain at that point and is detectable on verify. """ def __init__(self, path: str): self.path = path self._head_hash = "GENESIS" # sentinel for the first record self._load_head() def record(self, record_data: dict) -> dict: """Append a record. The caller cannot set record_hash or previous_hash — the chain vouches for integrity, not the caller.""" if "record_hash" in record_data or "previous_hash" in record_data: raise ValueError("Caller must not set hash fields") ts = datetime.now(timezone.utc).isoformat() record = { **record_data, "timestamp": record_data.get("timestamp", ts), "previous_hash": self._head_hash, } record["record_hash"] = self._hash(record) self._append_to_disk(record) self._head_hash = record["record_hash"] return record def verify(self) -> tuple[bool, str | None]: """Walk the chain. Returns (ok, first_broken_record_id_or_None). A broken chain means a record was altered after it was written. Run before any report ships; logged as part of the report build.""" prev = "GENESIS" for record in self._iter_records(): if record["previous_hash"] != prev: return (False, record["id"]) recomputed = self._hash({k: v for k, v in record.items() if k != "record_hash"}) if recomputed != record["record_hash"]: return (False, record["id"]) prev = record["record_hash"] return (True, None) @staticmethod def _hash(record: dict) -> str: # sort_keys=True => canonical hash. An independent auditor with the # evidence directory and this algorithm reproduces the verification. return hashlib.sha256( json.dumps(record, sort_keys=True).encode() ).hexdigest() ``` The `verify()` method is not decorative. It runs before any report ships, and the result (the `(ok, broken_id)` tuple) is logged as part of the report build. If a finding is disputed, the audit trail shows not only the evidence records but the verification that confirmed the chain was intact at report time. An unverified chain is an unverified claim. The `sort_keys=True` makes this independently auditable: a client's auditor, given the evidence directory and the hash algorithm, can re-run `verify()` and reproduce the same pass/fail. If the hash were over Python's `repr` or dict insertion order, verification would be implementation-dependent. Canonical JSON is the portable choice. ## Chain of custody The evidence chain answers: who ran what, when, from which IP, in which session. For a client-facing report or a legal proceeding, this chain of custody is the difference between credible evidence and an assertion. The harness maintains it automatically — every tool call gets a trace_id and session_id, the operator's identity is recorded, the source IP is stamped, and every record links to the previous one via hash chain. The "harness's egress IP, not the operator's personal IP" matters: an engagement run from a developer laptop that also carries personal traffic stamps records with a residential IP — an operational-security exposure (the target now has the operator's home IP) and a chain-of-custody weakness. The S03.3 dual-containment model addresses this at the network layer; in S02 the discipline is simpler: the harness egresses through a declared engagement IP, and that IP is what every record carries. ## Worked scenario: a finding from discovery to client report To make the chain concrete, trace one finding end to end. Suppose `nuclei` matches the `git-config` template against `https://repo.example-corp.com/.git/config`: 1. **Discovery.** `nuclei` (wrapped per S02.2) executes. The wrapper calls `evidence_logger.record(...)` before returning, producing `ev-047`: `tool=nuclei v3.3.0`, `target=repo.example-corp.com`, `request` = the exact command line, `response` = the raw matched output, `scope_ref` = the scope entry authorizing `*.example-corp.com`, `finding_id=null`, `retention_class=public`. Its `previous_hash` links to `ev-046`; its `record_hash` becomes the new head. 2. **Triage.** The raw match enters the S02.4 pipeline. The secondary model evaluates it: `exploitable=true`, `in_scope=true`, `false_positive=false`, `severity=high`. A `FindingRecord` is created, and `ev-048` is written — a finding-link record whose `finding_id` points to the new finding and whose fields capture the verdict. Two records now tie to this finding. 3. **Confirmation.** The operator fetches `.git/config` to confirm exploitability. The fetch tool wraps the request, producing `ev-049`: `request` = `GET /.git/config`, `response` = the config body, `finding_id` = the finding's id, `retention_class=redacted` (the config may reference internal paths). This is the reproducible proof. 4. **Report.** The report generator (S03.4) renders the finding, linking `evidence_refs: ["ev-047","ev-048","ev-049"]`. Before rendering it calls `evidence_logger.verify()`; the chain is intact, so the citations are trustworthy. The client receives a finding whose every claim is one click from the exact request and response. 5. **Retention.** Thirty days after the report is accepted, the retention policy fires on `ev-049` (redacted class, 30-day tail). The record is destroyed in a batch, and a destruction record `ev-120` is appended noting what was destroyed, when, and under which policy clause. The chain remains verifiable: `ev-049` is gone, but `ev-120` records its absence, and `ev-121`'s `previous_hash` links to `ev-120`. Destruction is recorded, not silent. Step 5 is the one most teams omit, and the one S00.2 insists on. A store that accumulates forever is a compliance liability; one that deletes silently is a chain-of-custody break. Recording destruction preserves verifiability while honoring the retention obligation. --- # S02.4 — Signal/Noise Filter and Triage Pipeline The harness's job is to triage findings, not dump raw scanner output. A nuclei scan can produce 200 raw findings, of which 15 are real vulnerabilities, of which 3 are exploitable, of which 1 is worth reporting. The triage pipeline turns raw scanner output into a prioritized finding list. This connects forward to S03.4 — the report generator renders *triaged* findings, not raw matches — and back to S01.1: triage is the `orient` phase of the offensive state machine, where raw observations become structured decisions about what to act on next. ## The triage pipeline ``` Raw finding (from scanner) ↓ Dedup (against engagement memory + known false positives) ↓ Severity score (CVSS draft) ↓ Context enrichment (CVE lookup, CWE mapping, exploitability check) ↓ Model-judged triage (secondary LLM evaluates exploitability + scope relevance) ↓ Surface / Batch / Discard ``` ## Model-judged triage The key innovation: a **secondary LLM call** evaluates each raw finding for exploitability and scope relevance. The primary model found the finding; the triage model judges it. This separation prevents the primary model's enthusiasm (which led it to report the finding) from biasing the severity assessment. The full function, with prior-disposition injection from memory: ```python from pydantic import BaseModel, Field from typing import Literal import json class TriageVerdict(BaseModel): exploitable: bool in_scope: bool false_positive: bool severity: Literal["critical", "high", "medium", "low"] reasoning: str = Field(..., description="One-paragraph justification") confidence: Literal["low", "medium", "high"] async def triage_finding( finding: "RawFinding", scope: "ScopeFile", memory: "EngagementMemory", # S02.1 — for similar-past-finding lookup triage_model, # a DIFFERENT model instance than the primary ) -> "TriagedFinding": # Prior dispositions make triage accumulate across the engagement # rather than re-judging every match from scratch. priors = memory.similar_findings(finding.signature_summary(), k=5) priors_block = "\n".join( f"- {p.get('title')} -> {p.get('disposition','unknown')}" for p in priors ) or "(no prior similar findings on this engagement)" prompt = f""" You are a vulnerability triage analyst. Evaluate this finding. Finding: {finding.title} Target: {finding.target} Evidence: {finding.evidence_summary} CVSS (draft): {finding.cvss_draft} Prior dispositions on similar findings in this engagement: {priors_block} Questions: 1. Is this finding exploitable in THIS specific context? (not theoretically, actually) 2. Is it in scope? (check against the scope description) 3. Is it a known false positive pattern? 4. What is the real severity? (may differ from the draft CVSS) Respond as JSON matching the TriageVerdict schema. """ result = await triage_model.complete(prompt) verdict = TriageVerdict(**json.loads(result)) return TriagedFinding(original=finding, verdict=verdict, priors=priors) ``` Two things in that function are load-bearing. First, `triage_model` is a *different model instance* — different session, ideally a different provider, so a context-poisoning or temperature artifact in the primary cannot carry into the triage judgment. Second, the `priors_block` injects the engagement's accumulated dispositions: if five similar `.git`-exposure matches were already confirmed, the sixth is judged with that context. This is how triage quality improves across an engagement without retraining. ## Why the secondary model prevents confirmation bias — concretely The single-model triage anti-pattern is real, and worth a concrete scenario. Suppose the primary model, running nuclei, matches `tech-detect` and reports "Spring Boot Actuator detected on `api.example-corp.com`." The primary model is *invested* — it surfaced the finding, it is mid-reasoning, its next step is to probe the actuator. If the same model now judges severity, it judges its own discovery, and LLMs are measurably more lenient toward claims they generated. It rates it `high`, asserts `exploitable=true`, and proceeds — even if the "actuator" is a health-check endpoint with no sensitive routes. The secondary model, given the same finding, has no investment. It did not discover it; its only job is to evaluate. It asks "is this *actually* exploitable in this context?" and, finding the actuator exposes only `/health` and `/info` (no `/env`, no `/heapdump`), rates it `low` and flags it for batching. The primary model's enthusiasm is checked by a model with no enthusiasm to project. The bias is not a flaw in the primary model; it is a property of asking the same reasoner to both generate and evaluate. The cure is structural: separate the roles. This is judgment, supported by the general finding that LLM self-evaluation is more confident than cross-model evaluation — but the operational test is the one that matters: run both configurations against a labeled dataset and measure precision. The dual-model configuration shows higher precision at comparable recall, because it surfaces fewer enthusiasm-driven false positives. ## Worked scenario: 200 raw findings down to 3 reports Walk the pipeline over a real-shaped nuclei run. Against 412 live endpoints (from the S02.2 scenario), nuclei returns 200 raw matches: - **Dedup.** 200 matches collapse against the `findings` collection's signatures. 60 are duplicate matches of the same template on the same URL (nuclei emits a finding per matched instance). 40 are duplicates across URLs behind a load balancer. After dedup: 100 unique candidates. - **Severity + enrichment.** Each gets a draft CVSS and CVE/CWE lookup. 30 are informational (`tech-detect`, `favicon-hash`) — batched. 20 are known FP patterns (e.g., a `robots.txt` "exposure"). After filtering: 50 candidates with real severity. - **Model-judged triage.** Of the 50: 12 are marked `false_positive` (a CSP "missing directive" the RoE excludes). 20 are `in_scope=true` but `exploitable=false` or `severity=low` — batched. 18 remain. - **Operator review.** The 18 surfaced findings get human eyes. 3 are confirmed as client-ready reports. 9 merge into 3 aggregate findings (same vulnerability class across endpoints — one report, not nine). 6 are dropped after verification. End state: 200 raw matches, 3 client-ready findings, plus a batched low-severity set and a logged-but-discarded FP set. Every disposition is recorded in the evidence chain (S02.3) and reflected in memory (S02.1), so the next engagement against the same target begins with these dispositions as priors. ## When to surface, batch, or discard - **Surface immediately:** Critical and high-severity exploitable findings in scope. The operator is notified now. - **Batch:** Medium and low-severity findings, or findings needing human verification. Accumulated for the session report. - **Discard:** Confirmed false positives and out-of-scope findings. Logged but not surfaced. The "discard" path is misleadingly named. A discarded finding is not deleted — it is logged in the evidence chain with its disposition, and its signature is added to the engagement's false-positive list so future matches auto-discard without re-triage. This is how the pipeline gets faster over an engagement: the auto-discarded signature set grows, and the secondary model is invoked only on genuinely novel matches. ## Measuring triage performance The triage pipeline is measurable. Against a labeled vulnerability dataset (e.g., a deliberately vulnerable lab target like DVWA with known vulnerabilities tagged): - **Precision:** Of findings surfaced, how many were real? (False positive rate = 1 - precision) - **Recall:** Of real vulnerabilities, how many did the triage surface? (Miss rate = 1 - recall) - **F1:** Harmonic mean of precision and recall. A good triage pipeline targets >90% precision and >80% recall. Below 80% precision means the operator is buried in false positives. Below 70% recall means real vulnerabilities are being missed. The measurement is not one-time. Run it per engagement against the engagement's own labeled subset (the findings the operator confirms or rejects become the labels), and track precision/recall over time. Drift in precision means the FP list is stale or the threshold shifted; drift in recall means the dedup signature is too aggressive and collapsing distinct findings. Both are fixable, but only if measured. --- ## Anti-Patterns ### The raw-dump report Surfacing raw nuclei output as the finding list. 200 raw findings, no triage, no dedup. The client sees noise, not signal. Cure: the triage pipeline (this section). ### The un-scope-checked tool A tool that calls subprocess directly without the scope wrapper. The middleware catches most cases, but a tool that bypasses the executor bypasses the middleware. Cure: every tool goes through the executor; the scope check is in both places. ### The context-window memory Storing engagement knowledge in the model's context window rather than a persistent store. It compacts, it disappears, the harness re-discovers what it already knew. Cure: ChromaDB or equivalent persistent store. ### The single-model triage Using the same model that found the finding to judge its severity. Confirmation bias. Cure: separate triage model. ### The unbounded subdomain scan `subfinder` + `amass` + a recursive `ffuf` of a 50,000-word wordlist, run on every session resume, written to memory without dedup. The store balloons, the harness spends its budget on rescans, and the operator cannot find real findings in the noise. "More words = more coverage" is the bias that produces the flood. Cure: passive sources first (certificate transparency, DNS), bounded wordlists scoped to the target's naming patterns, and write-time dedup (S02.1). If `scan_count` climbs and unique records are flat, the harness is rescanning — stop it. ### The context-window evidence store Storing evidence as messages in the model's context ("here is the nmap output from earlier") rather than the hash-chained log. It compacts, the chain is gone, and at report time there is no reproducible trail — only the model's recollection. This is the evidence-chain equivalent of the context-window memory anti-pattern, and it fails for the same reason: the context window is not durable. Cure: the evidence logger is middleware (S02.3) that writes to disk on every tool call; the report generator reads from the log, not from the model's memory. ### The single-pass triage Running the pipeline once, at engagement end, over accumulated raw findings. The false-positive list was never built, every match judged from scratch, and findings discovered late are triaged without the context of findings discovered early. Cure: triage is incremental, not batch. Every raw finding enters the pipeline as produced, dispositions are written back to memory, and the FP list grows across the engagement so later matches auto-discard. The end-of-engagement report is a render of accumulated triage, not a triage pass. --- ## Key Terms | Term | Definition | | --- | --- | | **Target-state memory** | Persistent structured representation of the target's attack surface and engagement knowledge | | **ChromaDB** | Default vector store for engagement-scoped offensive memory | | **Structured deduplication** | Write-time check that updates existing records rather than duplicating | | **Harness tool** | An offensive tool wrapped with Pydantic schema, scope check, rate limiting, and structured output | | **Evidence chain** | Append-only, hash-chained, scope-referenced record of every tool call and result | | **Triage pipeline** | Raw finding → dedup → severity → enrichment → model-judged triage → surface/batch/discard | | **Model-judged triage** | A secondary LLM call that evaluates exploitability and scope relevance | | **Reader-actor boundary** | The split between the model that ingests raw target output and the structured summary written to memory/evidence | | **Scope_ref** | The legal anchor field tying an evidence record to the specific authorization that permitted the call | --- ## Lab Exercise See `07-lab-spec.md`. Four labs: (1) persistent target-state memory seeded from an nmap scan, defended against poisoning; (2) three offensive tools with Pydantic schemas and scope wrappers; (3) evidence logger middleware with hash chaining; (4) nuclei triage pipeline measured against a known-vulnerability lab. --- ## References 1. **CAI (Cybersecurity AI)** — the bug bounty tool suite architecture. SDD-01 covers the full study. 2. **ChromaDB** — vector database for engagement-scoped memory. 3. **nmap, nuclei, httpx, ffuf** — the core recon and scanning tools. 4. **DVWA** — labeled vulnerability dataset for triage measurement. 5. **S00.1** — the scope file; this module's tools consume it as authorization input. 6. **S00.2** — evidence obligations and retention policy; the evidence chain is the engineering realization. 7. **S00.3** — rate limiting as a legal and stealth control; enforced in every wrapped tool. 8. **S01.1** — the offensive state machine; this module builds the tools that populate it. 9. **S01.2** — scope enforcement middleware; this module's tools run through it. 10. **S01.3** — prompt injection from target output; the reader-actor boundary is the defense. 11. **S03.1** — the attack graph; the flat target-state store here is its precursor. 12. **S03.4** — report generation; consumes the triaged findings and evidence records produced here.