# DD-24 — Teaching Script: Cost-Aware Agents *Verbatim teaching transcript. Read aloud at ~140 wpm. Maps to the 10-slide deck in 03-slide-deck.html. ~2,800 words across 10 slide cues. Total runtime ~50 minutes.* --- [SLIDE 0 — Title] Welcome to DD-24 — the final deep-dive in Course 1. This one is about money. Specifically, about why agents cost so much more than chat, and how to add budget enforcement to any agent loop so you don't burn the company's API budget. Here is the thesis. Pi has no budget enforcement. This deep-dive shows how to add it — turning Pi's personal-scale loop into an enterprise-scale loop that won't burn the company's API budget. This deep-dive connects to Course 4 Module E03, Cost-Aware Agent Architectures, which takes these patterns to fleet scale. Here we build the single-session cost-aware loop. E03 builds the enterprise version with per-tenant budgets and chargeback. --- [SLIDE 1 — The problem] Let's start with the numbers. They are stark. Agentic tasks consume up to 1,000 times more tokens than equivalent chat tasks. That's from Stanford and MIT research, published on arXiv. Chat is one request, one response — maybe 500 tokens. An agent is a loop. Each iteration of the loop is a provider call, and each provider call re-sends the entire context: the system prompt, the full conversation history, all tool definitions. A 10-turn agent session sends tens of thousands of tokens to the provider, not because the task is harder, but because the context is re-sent on every turn. Re-sent context accounts for 62% of total agent inference bills — that's from the Stanford Digital Economy Lab. The majority of the bill is the same tokens, billed again and again. And it's invisible because it doesn't feel like new spending. 95% of enterprise generative-AI pilots produce no measurable impact — that's MIT, 2025. 40% of agent projects will be cancelled by 2027 on cost grounds — that's Gartner. Token costs have been doubling every 45 days during the AI build-out. The implication: an agent without budget enforcement is a budget fire. A single runaway loop — the model retries a failing tool call 500 times, each retry re-sending the full context — can burn thousands of dollars in an afternoon. Pi has no defense against this. Neither does DeerFlow. Neither do most agent harnesses. --- [SLIDE 2 — The multiplier stack] Why 1,000x? Five mechanisms compound. Let me walk through them. One: re-sent context. Every turn re-sends the full history. 62% of inference bills. Turn 1 sends the system prompt. Turn 2 sends the system prompt plus turn 1. Turn 3 adds turn 2. By turn 10, you're re-sending 9 turns of history with every call. The same tokens, billed again and again. Two: tool-call fan-out. When the model calls a tool, the tool's result goes back into context, and the model calls again. A task that needs 5 tool calls generates 5 additional provider calls. If agents spawn sub-agents, the fan-out is multiplicative. A 5-deep tree with 3 branches per node is 3 to the 5th power — 243 leaf calls, each with its own context stack. Three: retry loops. When a tool call fails, the model retries. Three retries triple the cost. When retries are unbounded — the model doesn't know the tool is permanently broken — cost is unbounded. This is the runaway loop. Four: quadratic attention. Transformer attention is O of n squared in sequence length. Doubling context more than doubles compute cost per token. A 100K context is not 10 times the cost of 10K — it's more. Five: context rot. As context fills, accuracy degrades — 30% or more at mid-window, per Stanford. Longer sessions cost more per token AND produce worse results, which triggers more retries, which costs more. Context rot is a cost amplifier disguised as a quality problem. These five compound. That's the 1,000x. --- [SLIDE 3 — The cost iceberg] Now here's the part most teams miss. The LLM inference bill — the tokens you pay the provider — is only about 20% of the total cost of ownership. The full cost has four layers. Layer 1: LLM inference. The token bill. About 20% of TCO. This is what most teams track. It's the visible part of the iceberg. Layer 2: Context management. Storage, retrieval, re-injection of context across turns. This is where the 62% re-sent-context problem lives. The largest invisible cost. Layer 3: Retrieval and RAG. Embedding generation, vector store queries, reranking. Every retrieval step costs, and agents retrieve on most turns. Layer 4: Orchestration, eval, and governance. The infrastructure around the agent. This is called the hidden 80% because teams budget for Layer 1 and are surprised by Layers 2 through 4. The trap: optimizing only Layer 1 — say, switching to a cheaper model — addresses 20% of the cost. The majority of agent cost lives in Layers 2 through 4, and most teams don't instrument it. A cost-optimization effort that only looks at the model-choice lever will underdeliver. --- [SLIDE 4 — The @token_meter interceptor] Now the core pattern. The @token_meter interceptor. This comes from Microsoft's EvalAgentic research. The interceptor wraps every provider call. It tracks cumulative token usage against a per-task budget, and it enforces budget at three levels. Throttle at 80%. The session is approaching budget. Actions: switch to a cheaper model — there's a 27x price spread between GPT-4o and GPT-4o-mini — reduce the context window by summarizing and evicting older turns, or skip non-essential tool calls. The session continues, but at lower cost. Rollback at 100%. The session has exceeded budget. Actions: abort the current operation, restore the agent to its state before the current step, surface to the operator. The session does not continue past this point without explicit intervention. Circuit-break on retry storms. Consecutive retries exceed a threshold — say 5. The same tool failing 5 times in a row is a storm, not a transient error. The breaker trips, the loop stops immediately, the operator is notified. This is the defense against the runaway loop. The teaching document has the full TypeScript implementation of the TokenMeter class and the cost-aware loop. It compiles. It is the budget-enforcement layer that Pi lacks. --- [SLIDE 5 — HITL at thresholds] Some spending decisions should not be automated. When a session crosses a cost threshold, the right action is not to throttle or rollback silently. It's to pause and ask a human. Three tiers. At 50% of budget: log. Surface to the operator dashboard. No interruption. At 80%: warn and offer a choice. Notify the user: "This task has used 80% of its budget. Continue? Switch to a cheaper model? Abort?" The human decides. At 100%: hard stop. No more spending without explicit override. The task is paused, not killed — the human can authorize more budget if the task is worth it. This converts uncontrolled spending into controlled spending. The agent doesn't get to burn unlimited budget on a single task. The human is in the loop at the points where the cost of continued spending exceeds the cost of asking. One warning: alert fatigue. If you set HITL at every turn, or at every 10% of budget, the human stops reading the alerts and rubber-stamps everything. Set thresholds at meaningful decision points — typically one or two per session. Not at frequent intervals. --- [SLIDE 6 — Prompt caching] Now the highest-ROI optimization in this entire deep-dive. Prompt caching. Anthropic's prompt caching gives a 90% reduction on cached tokens. Your system prompt and tool definitions are re-sent on every call. If they total 2,000 tokens and you make 100 calls, that's 200,000 billable tokens without caching. With caching: 20,000. A 90% discount, just for structuring your prompt correctly. The catch: caching is prefix-based. The cache hits only if the beginning of the prompt is identical across calls. This means static prefixes — a system prompt that never changes — cache perfectly. But dynamic prefixes kill the cache. A timestamp in the system prompt. A session ID. A per-request UUID. Any of these in the prefix invalidates the cache. And some frameworks — certain LangChain configurations — inject dynamic IDs silently, destroying the 90% benefit without the developer knowing. The fix: audit your prompt prefix. Remove anything dynamic from the first N tokens. Put timestamps, session IDs, and per-request data AFTER the stable prefix. This single change can cut your inference bill by up to 90% with zero quality loss. It is the highest-ROI optimization in agent engineering, and most teams miss it because the cache invalidation is invisible. --- [SLIDE 7 — Cost-aware tool selection] Not all tools cost the same. A read_file tool is free — it reads the local filesystem. A web_search tool costs a per-query fee. A run_analysis tool that calls a paid API costs per call. A generate_chart tool that calls an image model costs per image. Cost-aware tool selection means the orchestrator knows the cost of each tool and factors it into the decision. At the planning step, choose the cheaper path when multiple tools could accomplish the same goal. At the throttle threshold, skip expensive tools and fall back to cheaper alternatives. Model routing is the same principle. A 27x price spread exists between GPT-4o and GPT-4o-mini. RouteLLM — from UC Berkeley, published at ICLR 2025 — routes cheap queries to cheap models, cutting cost 85% while maintaining 95% of quality. Most agent tasks don't need the expensive model on every turn. The first few turns — planning, decomposition — benefit from the strong model. The execution turns — tool calls, simple reasoning — can run on the cheap model. Cost-aware routing switches mid-session based on the task phase. --- [SLIDE 8 — The prime directive] Now a warning. There is a trap in cost optimization, and the research literature calls it the prime directive. Cutting cost 80% while dropping success rate 50% does not produce an optimized agent. It produces a broken agent that is cheaper to run. An agent that costs 80% less but succeeds half as often is a worse product. The customers who relied on the 50% of tasks that now fail are not saved money — they are failed tasks. Cost optimization that sacrifices success rate is a regression, not an improvement. The correct framing: cost optimization must hold success rate constant — or improve it — while reducing cost. That's what RouteLLM does: 85% cost cut at 95% of quality. That's what prompt caching does: 90% cost cut at 100% of quality. Those are correct optimizations. The trap is most dangerous in the throttle action. When you switch from the strong model to the cheap model at 80% budget, the cheap model may fail at tasks the strong model handled. The throttle must be paired with a quality check. If the cheap model's output confidence is low, fall back to the strong model for that turn and accept the cost. Blind throttling without quality monitoring is the prime-directive violation. You're making the agent cheaper and worse, and you won't even know it. --- [SLIDE 9 — The loop] Let me show you the structural change. Pi's loop: call model, tools, repeat. No budget tracking, no throttle, no circuit breaker, no HITL. It runs until done, until the model stops, or until the budget burns. The cost-aware loop: meter.check, call model, meter.record. Budget tracked per session. Throttle at 80%. Rollback at 100%. Circuit-break on retry storms. HITL at thresholds. Here is the key architectural point: the cost-aware loop is a wrapper, not a rewrite. You add the @token_meter interceptor at the provider-call boundary. The loop structure stays the same. You don't redesign Pi's loop — you wrap it. This is how you turn Pi's personal-scale loop into an enterprise-scale loop without starting from scratch. The teaching document has the full TypeScript implementation — the TokenMeter class and the costAwareLoop function. It compiles against standard TypeScript. The lab asks you to build it and run the runaway-loop demo so you can see the circuit breaker trip. Course 4 Module E03 builds the fleet version: per-tenant budgets, cost allocation across teams, the cost dashboard that surfaces the hidden 80%, and the chargeback model that makes agent cost visible to the business units consuming it. --- [SLIDE 10 — Recap] To recap. Agents cost 1,000 times more than chat because five mechanisms compound: re-sent context, tool-call fan-out, retry loops, quadratic attention, and context rot. LLM inference is only 20% of cost — the majority is context management at 62% of bills plus the hidden 80% of orchestration, eval, and governance. The core pattern is the @token_meter interceptor: throttle at 80%, rollback at 100%, circuit-break on retry storms. HITL at thresholds — but only at decision points, not every turn, or you get alert fatigue. Prompt caching gives a 90% discount if your prefix is stable. Audit for dynamic content — timestamps, session IDs — in the first tokens. This is the highest-ROI optimization, and most teams miss it. And the prime directive: cheaper-and-broken is not optimized. Hold quality constant while you reduce cost. RouteLLM and prompt caching do this correctly. Blind throttling does not. The lab asks you to build a TypeScript cost-aware loop wrapper with TokenMeter, budget enforcement, HITL simulation, and a runaway-loop demo. The code in the teaching document is real and compiles. Start there. That's DD-24. The final deep-dive of Course 1. The layer that turns a personal harness into an enterprise harness. Budget as a first-class constraint. --- *End of transcript. Runtime ~50 minutes at 140 wpm.*