{ "module": "DD-24", "title": "Cost-Aware Agents: Budget as a First-Class Constraint", "course": "Harness Engineering Master Course", "version": "1.0.0", "duration_minutes": 30, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Approximately how much more do agentic tasks cost compared to equivalent chat tasks?", "options": [ "10x more", "100x more", "Up to 1,000x more", "About the same" ], "answer_index": 2, "rationale": "Stanford-MIT research (arXiv 2604.22750) found agentic tasks consume up to 1,000x more tokens than chat. Chat is one request/response (~500 tokens). Agents are loops that re-send full context every turn, compounding via five mechanisms." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What percentage of total agent inference bills is re-sent context?", "options": [ "About 20%", "About 40%", "About 62%", "About 90%" ], "answer_index": 2, "rationale": "Stanford Digital Economy Lab found re-sent context is 62% of total agent inference bills. Every provider call re-sends the system prompt, conversation history, and tool definitions — the same tokens billed repeatedly." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the @token_meter interceptor pattern?", "options": [ "A billing API that charges clients per token", "An interceptor that wraps provider calls, tracks cumulative token usage against a budget, and enforces throttle at 80%, rollback at 100%, circuit-break on retry storms", "A caching layer that reduces token counts by 90%", "A model router that selects the cheapest model for each query" ], "answer_index": 1, "rationale": "The @token_meter (from Microsoft EvalAgentic) is the core pattern: wraps every provider call, tracks cumulative usage against a per-task budget, and enforces three actions: throttle (80%, switch to cheaper model), rollback (100%, abort + restore), circuit-break (retry storm, stop loop)." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "Your agent's TokenMeter reports cumulative usage at 85% of the task budget. The beforeCall check returns 'throttle'. What should the cost-aware loop do?", "options": [ "Continue normally — 85% is close enough to 100%", "Switch to a cheaper model, reduce the context window (summarize-and-evict older turns), and/or skip non-essential tool calls. The session continues at lower cost.", "Immediately abort the session and rollback all changes", "Send an alert to the user and wait for approval before every subsequent call" ], "answer_index": 1, "rationale": "Throttle at 80% means the session approaches budget. Actions: switch to a cheaper model (27x price spread GPT-4o vs mini), reduce context window, skip non-essential tools. The session CONTINUES at lower cost. Rollback is at 100%; HITL is at decision points, not every call." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "Your agent has retried the same failing tool call 6 times consecutively (threshold is 5). What happens?", "options": [ "The agent retries indefinitely until the tool succeeds", "The circuit breaker trips: the loop stops immediately, the operator is notified, and the breaker requires manual reset", "The agent switches to a cheaper model and retries", "The TokenMeter logs a warning but allows the retry" ], "answer_index": 1, "rationale": "Consecutive retries >= threshold (5) is a retry storm — the circuit breaker trips. The loop stops immediately, the operator is notified. Reset is MANUAL (automatic reset produces a flapping breaker). This prevents a single broken tool from burning the entire budget." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You audit your agent's system prompt and find a timestamp injected at character position 50 ('Current time: 2026-07-11T...'). Prompt caching is enabled but you're seeing 0% cache hits. What is the fix?", "options": [ "Increase the cache TTL", "Move the timestamp to AFTER the stable portion of the system prompt. Caching is prefix-based — any dynamic content in the first tokens invalidates the entire cache. Static content must come first.", "Switch to a different model provider", "Disable caching and rely on throttle instead" ], "answer_index": 1, "rationale": "Prompt caching is prefix-based. A timestamp at position 50 means the prefix changes every call, invalidating the cache. Fix: put static content (system prompt, tool defs) first; move dynamic content (timestamps, IDs) after the stable prefix. This restores the 90% discount with zero quality loss." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Your team's cost optimization effort reduced the LLM inference bill by 30% by switching to a cheaper model, but total agent costs barely moved. Why?", "options": [ "The cheaper model is not actually cheaper", "LLM inference is only ~20% of total cost of ownership. The majority lives in context management (62% of inference bills), retrieval/RAG, and orchestration/eval/governance (the hidden 80%). Optimizing Layer 1 alone addresses a small fraction.", "The agent is running fewer sessions", "The model switch increased quality, causing more usage" ], "answer_index": 1, "rationale": "The cost iceberg: LLM inference is ~20% of TCO. Optimizing it (even by 30%) moves only ~6% of total cost. The majority is in context management (62% of bills), retrieval, and orchestration/eval/governance (the hidden 80%). A Layer-1-only optimization underdelivers because it ignores 80% of the spend." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Your cost-aware loop throttles at 80% budget by switching from GPT-4o to GPT-4o-mini. After throttling, the agent's task success rate drops from 92% to 45%. What went wrong?", "options": [ "The throttle threshold is too high", "The prime directive was violated. Blind throttling without quality monitoring produces a cheaper-but-broken agent. The cheap model can't handle tasks the strong model handled. Throttle must be paired with a quality check — if output confidence is low, fall back to the strong model for that turn.", "GPT-4o-mini is not a real model", "The circuit breaker should have tripped" ], "answer_index": 1, "rationale": "The prime directive: cutting cost while dropping success rate produces a broken-cheaper agent. Throttle must be paired with quality monitoring. If the cheap model's output confidence is low, fall back to the strong model for that specific turn. Blind throttle = prime-directive violation." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You set HITL alerts at every 10% of budget (10%, 20%, 30%, ... 100%). After a week, the operators are rubber-stamping every alert without reading them. What is the fix?", "options": [ "Make the alerts louder (push notifications, SMS)", "Reduce HITL to meaningful decision points — typically one or two per session (e.g., warn+choice at 80%, hard stop at 100%). Frequent alerts produce fatigue; operators stop reading them.", "Fire the operators for not reading alerts", "Increase the budget so alerts trigger less often" ], "answer_index": 1, "rationale": "Alert fatigue. HITL at every 10% produces 10 interruptions per session. Operators rubber-stamp to make them stop. Fix: set thresholds at meaningful decision points — typically the 80% warn+choice and the 100% hard stop. One or two human decisions per session, not ten." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does context rot act as a cost amplifier rather than just a quality problem?", "options": [ "Because longer contexts use more memory", "Because degraded accuracy (30%+ at mid-window) triggers more retries, each retry re-sends the full context, each retry costs more. Context rot drives retries, retries drive cost. It is a cost amplifier disguised as a quality issue.", "Because the model charges more for longer contexts", "Because context rot only happens with expensive models" ], "answer_index": 1, "rationale": "Context rot degrades accuracy at mid-window (Stanford: 30%+ degradation). Worse results trigger more retries as the model tries to correct. Each retry re-sends the full context — more tokens, more cost. Context rot → retries → cost. It amplifies cost indirectly through the retry mechanism." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team argues: 'We cut our agent cost 80% by switching to the cheapest model. Our success rate dropped from 90% to 45%, but we're saving money, so it's a win.' What is the flaw?", "options": [ "There is no flaw — saving money is always good", "The prime directive violation. An agent that costs 80% less but succeeds half as often is a worse product. The 45% of tasks that now fail are failed customers. Cost optimization must hold success rate constant (or improve it). This is cheaper-and-broken, not optimized.", "The flaw is that they didn't cut cost enough", "The flaw is that 90% success was already too low" ], "answer_index": 1, "rationale": "The prime directive: cutting cost while dropping success rate produces a broken-cheaper agent. The 45% of tasks that now fail represent failed customers — not savings. Correct optimization holds quality constant: RouteLLM (85% cut at 95% quality), prompt caching (90% cut at 100% quality). This team's switch is a regression." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why must the budget check happen BEFORE the provider call, not after?", "options": [ "Because the provider charges more for after-call checks", "Because by the time the call completes, the tokens are already spent. Checking projected usage before the call allows prevention (throttle or rollback BEFORE paying). Checking after only allows detection — you've already burned the budget.", "Because before-call checks are faster", "Because the API requires it" ], "answer_index": 1, "rationale": "The @token_meter's beforeCall checks PROJECTED usage (current cumulative + estimated prompt tokens) against budget. If projected > 100%, rollback before the call happens. If > 80%, throttle before paying full price. After-call tracking is for recording; before-call checking is for prevention. You cannot un-spend tokens." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Compare the cost-aware loop to Pi's original loop. What is the architectural relationship?", "options": [ "The cost-aware loop is a complete rewrite of Pi's loop with a different structure", "The cost-aware loop is a WRAPPER around Pi's loop. The @token_meter interceptor sits at the provider-call boundary. The loop structure (call model → tools → repeat) stays the same. You add meter.check before and meter.record after each call. Pi's loop + meter = cost-aware loop.", "The cost-aware loop replaces Pi's tools with cheaper alternatives", "They are unrelated architectures" ], "answer_index": 1, "rationale": "The cost-aware loop is a wrapper, not a rewrite. The @token_meter interceptor sits at the provider-call boundary — meter.beforeCall() checks budget before the call, meter.afterCall() records usage after. The loop structure is unchanged. This is how you turn Pi's personal-scale loop into an enterprise-scale loop without redesigning the loop." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is automatic circuit-breaker reset dangerous, and why must the reset be manual?", "options": [ "Manual reset is faster than automatic", "Automatic reset produces a flapping breaker: it trips, auto-resets, the storm resumes, it trips again — forever. Manual reset forces an operator to investigate the CAUSE of the storm before resuming. The storm is a symptom; the breaker ensures someone diagnoses the disease.", "Automatic reset voids the warranty", "Manual reset uses fewer tokens" ], "answer_index": 1, "rationale": "A flapping breaker (auto-reset) never truly stops the storm — it trips and resets in a loop, and the underlying cause (a broken tool, a pathological sub-agent tree) continues. Manual reset forces investigation: what is causing the retries? Fix the cause, then reset. The breaker is a forcing function for diagnosis, not just a stop button." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Prompt caching offers a 90% discount on cached tokens. Why do many agent deployments see 0% cache hits despite enabling caching?", "options": [ "Prompt caching doesn't work for agents", "Caching is prefix-based. Dynamic content in the first tokens (timestamps, session IDs, per-request UUIDs) invalidates the cache. Some frameworks (certain LangChain configurations) inject dynamic IDs silently. The prefix changes every call, so the cache never hits.", "The 90% figure is marketing, not reality", "Cache hits require a paid subscription tier" ], "answer_index": 1, "rationale": "Caching is prefix-based — the cache hits only if the beginning of the prompt is identical across calls. Any dynamic content in the prefix invalidates it: timestamps, session IDs, UUIDs. Frameworks that inject dynamic IDs (some LangChain configs) silently destroy the 90% benefit. The fix: audit the prefix, move dynamic content after the stable portion." } ] }