{
  "id": "DD-24",
  "title": "Cost-Aware Agents: Budget as a First-Class Constraint",
  "pillar": "Deep-Dives",
  "duration": 50,
  "level": "senior",
  "prerequisites": ["Modules 0-12", "DD-01 (Pi)"],
  "subtitle": "Why agents cost 1,000x more than chat, and how to add budget enforcement to any agent loop. The @token_meter interceptor pattern: throttle at 80%, rollback at 100%, circuit-break on retry storms. Prompt caching (90% discount). The prime directive. With a real TypeScript cost-aware loop wrapper.",
  "lede": "Pi has no budget enforcement. An agent without budget controls is a budget fire — a single runaway loop can burn thousands of dollars in an afternoon. This deep-dive adds the missing layer: budget as a first-class constraint, tracked at the provider-call boundary, enforced via throttle/rollback/circuit-break, surfaced through HITL at thresholds, and designed into the loop from the start.",
  "claims": [
    "Agents cost up to 1,000x more than equivalent chat (Stanford-MIT, arXiv 2604.22750) because five mechanisms compound: re-sent context (62% of inference bills per Stanford Digital Economy Lab), tool-call fan-out, retry loops, quadratic attention (O(n²)), and context rot (30%+ degradation at mid-window, which triggers more retries, which costs more).",
    "LLM inference is only ~20% of total cost of ownership. The majority lives in context management (the 62% re-sent-context problem), retrieval/RAG, and orchestration/eval/governance — the hidden 80% that teams don't budget for. Optimizing only Layer 1 (switching models) addresses a fraction of spend.",
    "The @token_meter interceptor pattern (Microsoft EvalAgentic): wraps every provider call, tracks cumulative token usage against a per-task budget, enforces throttle at 80% (switch to cheaper model, reduce context), rollback at 100% (abort + restore prior state), and circuit-break on retry storms (stop the loop). Budget is checked BEFORE the call — prevention, not detection.",
    "Prompt caching is the highest-ROI optimization: up to 90% reduction on cached tokens (Anthropic), zero quality loss. The barrier is prefix stability — dynamic content (timestamps, session IDs, framework-injected UUIDs) in the first tokens invalidates the cache. Audit the prefix; move dynamic content after the stable portion.",
    "The prime directive: cutting cost 80% while dropping success rate 50% produces a broken-cheaper agent, not an optimized one. Cost optimization must hold success rate constant (RouteLLM: 85% cut at 95% quality; prompt caching: 90% cut at 100% quality). Blind throttling without quality monitoring is a prime-directive violation.",
    "The cost-aware loop is a wrapper, not a rewrite. You add the @token_meter interceptor at the provider-call boundary. The loop structure stays the same. Pi's loop + meter.check + meter.record = cost-aware loop. This is how you turn a personal-scale harness into an enterprise-scale harness without redesigning the loop.",
    "HITL at cost thresholds converts uncontrolled spending into controlled spending: log at 50%, warn+choice at 80%, hard stop at 100%. But HITL must be at meaningful decision points (one or two per session), not every turn — frequent alerts produce fatigue and rubber-stamping.",
    "The circuit breaker prevents runaway loops — the highest-risk cost failure. Four trip conditions: consecutive retries (e.g., >= 5), total session retries (e.g., >= 20), fan-out depth (e.g., >= 3), tokens-per-minute. Reset must be MANUAL (automatic reset produces a flapping breaker that never stops the storm)."
  ],
  "objectives": [
    "Explain why agents cost 1,000x more than chat via the five-mechanism multiplier stack.",
    "Quantify the invisible cost layers: re-sent context (62%), the hidden 80%, and why Layer-1-only optimization fails.",
    "Implement the @token_meter interceptor: throttle at 80%, rollback at 100%, circuit-break on retry storms.",
    "Design cost-aware tool selection and model routing (the RouteLLM pattern).",
    "Implement HITL at cost thresholds without triggering alert fatigue.",
    "Build a circuit breaker for runaway loops and explain why reset must be manual.",
    "Write a cost-aware execution loop in TypeScript and connect it to Course 4 Module E03."
  ],
  "subsections": [
    {
      "id": "DD-24.1",
      "title": "Why Agents Cost 1,000x More Than Chat",
      "duration": "8 min",
      "desc": "The multiplier stack: re-sent context (62% of bills), tool-call fan-out (5-deep tree = 243 leaf calls), retry loops (3 retries triple cost; unbounded = unbounded cost), quadratic attention (O(n²) in sequence length), context rot (30%+ degradation → more retries → more cost). Five mechanisms compounding to 1,000x."
    },
    {
      "id": "DD-24.2",
      "title": "The Invisible Cost Layers",
      "duration": "5 min",
      "desc": "The cost iceberg: Layer 1 LLM inference (~20% TCO — what teams track), Layer 2 context management (the 62% re-sent-context problem), Layer 3 retrieval/RAG (embeddings, vector queries, reranking), Layer 4 orchestration/eval/governance (the hidden 80%). Why optimizing only Layer 1 underdelivers."
    },
    {
      "id": "DD-24.3",
      "title": "The @token_meter Interceptor Pattern",
      "duration": "8 min",
      "desc": "The core pattern. Full TypeScript implementation of the TokenMeter class: beforeCall (check projected budget, return ok/throttle/rollback), afterCall (record usage, track consecutive retries, throw RetryStormError on storm). Budget checked BEFORE the call — prevention, not detection."
    },
    {
      "id": "DD-24.4",
      "title": "Cost-Aware Tool Selection",
      "duration": "5 min",
      "desc": "Annotating tools with cost estimates. Model routing: 27x price spread GPT-4o vs mini. RouteLLM (UC Berkeley, ICLR 2025) routes cheap queries to cheap models — 85% cost cut at 95% quality. Switching models mid-session based on task phase (strong for planning, cheap for execution)."
    },
    {
      "id": "DD-24.5",
      "title": "Human-in-the-Loop at Cost Thresholds",
      "duration": "5 min",
      "desc": "Three tiers: log at 50% (no interruption), warn+choice at 80% (human decides: continue? cheaper model? abort?), hard stop at 100% (task paused, not killed). Alert-fatigue avoidance: set thresholds at meaningful decision points, not every turn."
    },
    {
      "id": "DD-24.6",
      "title": "Circuit Breakers for Runaway Loops",
      "duration": "5 min",
      "desc": "The runaway loop is the highest-risk cost failure. Four trip conditions: consecutive retries (>= 5), total session retries (>= 20), fan-out depth (>= 3), tokens-per-minute. Manual reset (automatic = flapping breaker). The breaker is a forcing function for diagnosis."
    },
    {
      "id": "DD-24.7",
      "title": "A Cost-Aware Execution Loop in TypeScript",
      "duration": "8 min",
      "desc": "Real, compilable code. Wraps Pi's loop with TokenMeter: beforeCall budget check, HITL threshold trigger, model selection (strong vs cheap based on throttle flag), afterCall usage recording + retry tracking, circuit-breaker catch, hard iteration cap as fail-safe."
    },
    {
      "id": "DD-24.8",
      "title": "Prompt Caching: The 90% Discount You're Probably Missing",
      "duration": "4 min",
      "desc": "Prefix-based caching: 90% reduction on cached tokens. Cache-killers: timestamps, session IDs, per-request UUIDs in the prefix. Frameworks (some LangChain configs) inject dynamic IDs silently. Fix: audit the prefix, move dynamic content after the stable portion."
    },
    {
      "id": "DD-24.9",
      "title": "The Prime Directive and the C4 Connection",
      "duration": "4 min",
      "desc": "Cutting cost 80% while dropping success 50% = broken-cheaper, not optimized. Hold quality constant. Throttle must pair with quality check. Course 4 Module E03 (Cost-Aware Agent Architectures) builds the fleet: per-tenant budgets, chargeback, cost dashboards, hidden-80% instrumentation."
    }
  ],
  "prev": {
    "title": "DD-23 — DeerFlow: Deep Research Agent Architecture",
    "url": "../dd-23-deerflow/",
    "label": "Previous"
  },
  "next": null,
  "course1_index": "../../",
  "nav_code": "DD-24",
  "footer": "Harness Engineering Master Course · Deep-Dives · Course 1 Complete",
  "artifact_descs": {
    "01": "Teaching document — 9 sub-sections covering the multiplier stack (1,000x), the cost iceberg (4 layers, 20% visible), the @token_meter interceptor (full TypeScript), cost-aware tool selection + RouteLLM, HITL at thresholds, circuit breakers, the cost-aware TypeScript loop, prompt caching (90%), the prime directive + C4 E03 connection; with anti-patterns, key terms, references",
    "02": "6 Mermaid diagrams — multiplier stack (5 mechanisms), cost iceberg (4 layers), interceptor flow (throttle/rollback/break), HITL thresholds (3 tiers), runaway loop + circuit breaker (4 trip conditions), Pi vs cost-aware loop; design-system colors",
    "03": "10 slides — reveal.js, dark theme, design-system teal; multiplier stack, iceberg, interceptor, HITL, caching, tool selection, prime directive, the loop, recap",
    "04": "Verbatim teaching transcript with [SLIDE N] cues, ~2,800 words across 10 slides",
    "05": "22 flashcards (TSV) — mix of recall, application, analysis; covers multiplier stack, cost iceberg, interceptor, HITL, caching, circuit breaker, prime directive, RouteLLM, prompt caching",
    "06": "15 questions, 20/40/40 Bloom distribution (3 recall / 6 application / 6 analysis), 70% pass; validated JSON with rationale per question",
    "07": "Build a cost-aware agent loop — TypeScript lab: TokenMeter class (throttle/rollback/circuit-break), costAwareLoop function, mock provider (normal/storm/expensive/throttle), four demos (normal, retry storm with circuit breaker, budget exceeded with rollback, throttle), re-sent-context visualization; 5 stretch goals (HITL, model routing, prompt caching, tool selection, fan-out)"
  }
}