# DD-22 — MCP: Building Production Tool Servers **Course**: Master Course · **Deep-Dive**: DD-22 · **Duration**: 50 min · **Level**: Senior engineer · **Prerequisites**: Modules 0-12, DD-01 (Pi), DD-11 (OpenAI Agents SDK) **Pillar**: Deep-Dives > A tool is a function the model can call. A tool *server* is a process that exposes tools, resources, and prompts over a wire protocol — discoverable, versioned, independently deployable. MCP is the wire protocol. This deep-dive is about building production tool servers: the lifecycle, the capability negotiation, the schema validation, the registry, and the failure modes that turn a clever integration into a production outage. --- ## Learning Objectives 1. State the MCP thesis — tool definitions externalized into a discoverable, versioned service — and explain why this is the difference between a personal harness and a platform. 2. Describe the three primitives (tools, resources, prompts), the two transports (stdio, HTTP+SSE), and the JSON-RPC 2.0 message envelope that carries them. 3. Walk the full server lifecycle: initialization, capability negotiation, the message loop, graceful shutdown — and explain why each phase exists. 4. Write a JSON Schema for a tool input, and explain why input validation is the server's responsibility, not the client's or the model's. 5. Implement a production MCP server in TypeScript with: tool registration, schema validation, error handling, health checks, and graceful shutdown on SIGTERM. 6. Explain the registry/discovery pattern — why a client needs to know what a server offers *before* calling it — and how capability negotiation encodes this. 7. Diagnose the four production failure modes: partial tool failure, transport disconnect, schema drift across versions, and the runaway resource subscription that floods the client. --- ## The Thesis Pi (DD-01) has four tools hardcoded in `tool-registry.ts`. The model calls `read_file`, `write_file`, `edit_file`, `bash`. Each tool is a function reference. Adding a tool means editing the harness source, shipping a new build, restarting the process. This is fine for a personal harness. It is fatal for a platform. A platform needs tools to be **external**. A new database connector should not require redeploying the agent. A vendor integration should arrive as a server the agent discovers at runtime, not a library the agent links against. The team that maintains the CRM should ship their MCP server on their own release cadence, and the agent should pick it up without a rebuild. MCP (Model Context Protocol) is the wire protocol that makes this possible. It defines: - **Three primitives** — tools (model-invoked functions), resources (server-exposed data the client can read), prompts (server-authored prompt templates the user can invoke). Three primitives, three trust boundaries, three lifecycles. - **Two transports** — stdio (local process, the agent spawns the server as a subprocess) and HTTP+SSE (remote server, the agent connects over the network). Same protocol, different pipes. - **A JSON-RPC 2.0 envelope** — every message is a request, a response, or a notification, with a method name, parameters, and an optional ID for correlation. - **A capability negotiation handshake** — the client and server exchange their capabilities at initialization, so neither side sends a message the other cannot handle. The result: a tool becomes a service. A service has a version, a health check, a deployment pipeline, and an owner. This is the structural difference between "my agent has four tools" and "my platform integrates twelve vendor systems." --- ## DD-22.1 — The Protocol: JSON-RPC, Transports, and the Three Primitives ### The Envelope Every MCP message is JSON-RPC 2.0. Three message types: - **Request** — has an `id`, a `method`, and `params`. The recipient must reply with a response carrying the same `id`. - **Response** — has the `id` from the request, plus either `result` (success) or `error` (failure with code and message). - **Notification** — has a `method` and `params` but no `id`. Fire-and-forget. The recipient must not reply. This is not exotic. JSON-RPC 2.0 is the substrate of the Language Server Protocol (LSP), the Debug Adapter Protocol (DAP), and a generation of editor integrations. MCP reuses it because the problem is the same: two processes, one with capabilities the other wants to consume, needing a structured conversation over a pipe. ### The Transports Two transports, chosen by deployment topology: **stdio** — the agent spawns the server as a child process. Requests go to the server's stdin; responses come back on stdout. stderr is reserved for logs (never protocol messages — a common bug is logging JSON to stdout and corrupting the stream). This is the local-integration transport: the filesystem tool, the git tool, the shell tool. The agent and server share a machine, often a filesystem, sometimes a working directory. **HTTP+SSE** (Streamable HTTP in newer revisions) — the agent connects to an HTTP endpoint. The server can push updates over a Server-Sent Events stream. This is the remote-integration transport: the database connector running in a different availability zone, the vendor API proxied through a server the vendor operates. The same JSON-RPC messages, carried over HTTP instead of pipes. The protocol is transport-agnostic. The same server logic handles both; only the framing layer changes. This is deliberate — a server written for local use can be deployed remotely without rewriting the tool implementations. ### The Three Primitives **Tools** are model-invoked functions. The server advertises a tool with: a name, a description, and a JSON Schema for its input. The model sees the name and description, decides to call it, produces arguments conforming to the schema. The server executes, returns a result. This is the primitive that replaces Pi's hardcoded `tool-registry.ts`. **Resources** are server-exposed data. A file, a database row, a configuration object, a log tail. The server exposes a URI scheme; the client can read a resource by URI, list available resources, and (with the right capability) subscribe to updates. Resources are *passive* — the model does not "call" a resource. The client surfaces resources to the user (or to the model as context), and the user or client decides what to read. This is the primitive for "context the server knows about that the agent might want." **Prompts** are server-authored templates. A prompt is a named, parameterized message the server provides. The user (via the client UI) can invoke a prompt by name, fill in parameters, and inject the result into the conversation. This is the primitive for "the server knows the right way to ask for X." A database server might ship a prompt for "explain this query plan" — the user picks it from a menu rather than figuring out how to phrase the request. The three primitives map to three trust boundaries: | Primitive | Who initiates | Trust model | | --- | --- | --- | | Tool | The model decides to call | Untrusted input — schema-validated, must be safe to execute | | Resource | The user/client decides to read | Semi-trusted — the server vouches for the data | | Prompt | The user decides to invoke | Trusted template — the server authored it | Confusing these boundaries is a bug. A tool the model can call whenever it wants is different from a resource the user must opt into reading. A prompt the user selects from a menu is different from a tool description the model reads as an instruction. The protocol separates them precisely because the security implications differ. --- ## DD-22.2 — The Lifecycle: Initialization, Negotiation, and the Message Loop An MCP session is not "send a request, get a response." It is a stateful conversation with a defined beginning, middle, and end. Skipping the beginning (initialization) is the most common integration bug. ### Phase 1 — Initialization The client sends `initialize` with: - The protocol version it speaks. - Its own capabilities (what client-side features it supports). - Client identification (name, version). The server replies with: - The protocol version it will use (may be lower than what the client offered). - Its capabilities — which primitives it offers, which optional features it supports. - Server identification. The client then sends `notifications/initialized` — a notification (no reply expected) confirming the handshake is complete. Only after this notification may the client send any other request. This is not ceremony. The capabilities exchanged here determine what messages are valid for the rest of the session. If the server did not advertise the `tools.list` capability in its initialize response, the client must not send `tools/call`. If the server did not advertise `resources.subscribe`, the client must fall back to polling `resources/read`. The handshake is the contract; everything after is the contract in force. ### Phase 2 — Capability Negotiation (the part most teams skip) Capabilities are not booleans. They are structured objects describing the *shape* of support. The server's `tools` capability might include `{ listChanged: true }` — meaning the server will send `notifications/tools/list_changed` when its tool set changes. The client's `roots` capability (the client telling the server about the filesystem roots it can access) might include `{ listChanged: true }`. The negotiation rule: **only use what both sides advertised**. If the server offers `listChanged` but the client didn't advertise it understands list-changed notifications, the server should not send them. If the client offers `roots` but the server didn't advertise `roots` support, the server cannot query the client's filesystem roots. This is the difference between MCP and a typical REST API. A REST API documents its endpoints and assumes you read the docs. MCP *negotiates* the endpoint set at runtime, so a client written against version N degrades gracefully against a server speaking version N-1. You build a client once; it works against every server, using whatever subset of features that server offers. ### Phase 3 — The Message Loop After initialization, the client and server exchange messages until the session ends. Typical patterns: - On startup or when needed: `tools/list` → server returns the tool catalog → client surfaces to model. - Model decides to act: `tools/call` with name and arguments → server executes → server returns result or error. - User wants context: `resources/list` → `resources/read` → server returns content. - Server's state changes: `notifications/resources/updated` (if subscribed) → client re-reads. The loop is request/response for most operations, with notifications for server-initiated updates. The server does not get to "call" the client's tools — the protocol is asymmetric. The client drives; the server responds. (There is one exception: the server can request the client's `roots` — the filesystem locations the client operates in — via `roots/list`. This is the only server-initiated request in the core protocol.) ### Phase 4 — Shutdown On stdio: the client closes stdin, waits for the server process to exit, and if it doesn't exit within a timeout, sends SIGTERM. A well-behaved server handles SIGTERM by: finishing in-flight requests if possible, closing any open resource handles, flushing logs, exiting zero. On HTTP+SSE: the client closes the connection. The server detects the disconnect and cleans up. Both paths are the same operation: **graceful teardown**. The server must assume it can be terminated at any time and must not leave the world in a broken state — half-written files, unclosed transactions, leaked temp directories. This is the same discipline as any production service; MCP just inherits it. --- ## DD-22.3 — Tool Definitions: JSON Schema as the Validation Boundary A tool definition is three things: a name, a description, and an input schema. The input schema is JSON Schema. Example: ```json { "name": "search_tickets", "description": "Search support tickets by keyword and status. Returns up to 50 results.", "inputSchema": { "type": "object", "properties": { "query": { "type": "string", "minLength": 1, "maxLength": 200 }, "status": { "type": "string", "enum": ["open", "closed", "all"] }, "limit": { "type": "integer", "minimum": 1, "maximum": 50, "default": 25 } }, "required": ["query"], "additionalProperties": false } } ``` This schema is load-bearing for three reasons: 1. **The model uses it to generate arguments.** The description and schema are what the model sees when deciding how to call the tool. Ambiguous descriptions and loose schemas produce malformed calls. Tight schemas with `additionalProperties: false` catch the model's tendency to invent extra fields. 2. **The server uses it to validate before execution.** The server must re-validate the arguments against the schema *before* executing, because the client (or a man-in-the-middle, or a buggy model) may send invalid input. Never trust the model to produce conforming arguments. Never trust the client to have validated. The schema is the server's defense. 3. **The `additionalProperties: false` is a security control.** Without it, a model (or an attacker controlling tool output that influences the model) can inject unexpected fields. A `send_email` tool that doesn't forbid extra properties might receive `{ to, subject, body, bcc: "attacker@evil.com" }` if the model is steered to add it. Forbidding additional properties closes this. The schema is not documentation. It is the validation boundary. Treat it as such. --- ## DD-22.4 — A Production MCP Server in TypeScript This is a real MCP server, not pseudocode. It compiles against the `@modelcontextprotocol/sdk` package. It demonstrates: tool registration, schema validation, error handling, health logging, and graceful shutdown. ```typescript // server.ts — a production-shaped MCP server import { Server } from "@modelcontextprotocol/sdk/server/index.js"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; import { CallToolRequestSchema, ListToolsRequestSchema, } from "@modelcontextprotocol/sdk/types.js"; // ---- 1. Tool catalog --------------------------------------------------- // Each tool: name, description, JSON Schema for input. // The schema is the validation boundary — additionalProperties: false always. const TOOLS = [ { name: "search_tickets", description: "Search the support ticket system by keyword. Returns up to 50 results, newest first.", inputSchema: { type: "object" as const, properties: { query: { type: "string", minLength: 1, maxLength: 200, description: "Keyword or phrase to search for." }, status: { type: "string", enum: ["open", "closed", "all"], description: "Filter by ticket status. Default: open." }, limit: { type: "integer", minimum: 1, maximum: 50, default: 25, description: "Max results to return (1-50)." }, }, required: ["query"] as const, additionalProperties: false, }, }, { name: "get_ticket", description: "Fetch a single ticket by its numeric ID, including full thread.", inputSchema: { type: "object" as const, properties: { id: { type: "integer", minimum: 1, description: "Ticket ID." }, }, required: ["id"] as const, additionalProperties: false, }, }, ] as const; // ---- 2. Validation helper --------------------------------------------- // Re-validate at the boundary. Never trust the client or the model. function validate(input: unknown, schema: object): { ok: true; value: any } | { ok: false; error: string } { // In production use ajv. Here we do structural checks for illustration. if (typeof input !== "object" || input === null) { return { ok: false, error: "Input must be a JSON object." }; } const obj = input as Record; const s = schema as any; if (s.additionalProperties === false) { const allowed = new Set(Object.keys(s.properties || {})); for (const key of Object.keys(obj)) { if (!allowed.has(key)) { return { ok: false, error: `Unexpected property "${key}".` }; } } } // (full ajv validation elided for brevity — use ajv in production) return { ok: true, value: obj }; } // ---- 3. The server ----------------------------------------------------- const server = new Server( { name: "ticket-mcp", version: "1.0.0" }, { capabilities: { tools: { listChanged: false } } }, ); // List handler: returns the catalog the client surfaces to the model. server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS, })); // Call handler: the model invoked a tool. Validate, dispatch, respond. server.setRequestHandler(CallToolRequestSchema, async (request) => { const { name, arguments: args } = request.params; const tool = TOOLS.find((t) => t.name === name); if (!tool) { // Unknown tool — return a structured error. The model can recover. return { isError: true, content: [{ type: "text", text: `Unknown tool: "${name}".` }], }; } const validated = validate(args ?? {}, tool.inputSchema); if (!validated.ok) { // Validation failed — the model produced malformed arguments. // Return the error as content so the model can retry with corrected input. return { isError: true, content: [{ type: "text", text: `Invalid arguments: ${validated.error}` }], }; } try { const result = await dispatch(name, validated.value); return { content: [{ type: "text", text: JSON.stringify(result, null, 2) }] }; } catch (err) { // Partial failure: the tool ran but threw. Distinguish from validation error. return { isError: true, content: [{ type: "text", text: `Tool execution failed: ${(err as Error).message}` }], }; } }); // ---- 4. Dispatch — the actual business logic -------------------------- async function dispatch(name: string, args: any): Promise { switch (name) { case "search_tickets": { const { query, status = "open", limit = 25 } = args; // In production: call your ticket system's API here. return { query, status, limit, results: [] /* placeholder */ }; } case "get_ticket": { const { id } = args; return { id, found: false /* placeholder */ }; } default: throw new Error(`No handler for tool "${name}".`); } } // ---- 5. Lifecycle: transport + graceful shutdown ---------------------- const transport = new StdioServerTransport(); await server.connect(transport); // Never write protocol messages to stdout — it corrupts the stream. // Log to stderr only. process.stderr.write(`[ticket-mcp] server started, ${TOOLS.length} tools registered\n`); // Graceful shutdown: finish in-flight, then exit. let shuttingDown = false; const shutdown = (signal: string) => { if (shuttingDown) return; shuttingDown = true; process.stderr.write(`[ticket-mcp] received ${signal}, shutting down\n`); // In production: await server.close() and drain in-flight requests. process.exit(0); }; process.on("SIGTERM", () => shutdown("SIGTERM")); process.on("SIGINT", () => shutdown("SIGINT")); ``` Read this code for five production details: 1. **`additionalProperties: false`** appears in every schema. This is not optional. Without it, the model can inject fields the server does not expect. 2. **Validation happens at the call boundary**, not in the dispatch function. The schema is checked before the business logic runs. A validation failure returns an error to the model, which can retry. 3. **Three error classes are distinguished**: unknown tool (return `isError`), validation failure (return `isError` with a message the model can act on), and execution failure (the tool ran but threw). Each has a different recovery path. The model needs to know *which* kind of failure it got. 4. **Logs go to stderr, never stdout.** On stdio transport, stdout is the protocol stream. A stray `console.log` corrupts the stream and the client sees a parse error. This is the single most common MCP server bug. 5. **SIGTERM triggers graceful shutdown.** The server assumes it can be killed at any time and handles it cleanly. --- ## DD-22.5 — Registry, Discovery, and Version Drift A single MCP server is simple. Twelve MCP servers — one per vendor integration — is a platform problem. The registry pattern addresses it. ### The Registry Problem When the agent starts, it needs to know: which servers exist, how to connect to each, what each offers. Two approaches: **Static configuration** — a config file lists the servers. The agent reads it on startup, spawns/connects each, queries capabilities, builds the union of all tools/resources/prompts. Simple, works, but requires a redeploy to add or update a server. **Dynamic registry** — a registry service (or a directory the agent watches) lists available servers. The agent connects on startup and can be notified of additions/removals at runtime. More complex, but enables the "vendor ships a server, agent picks it up without redeploy" workflow that makes MCP a platform primitive. Most production deployments start with static config and graduate to a registry when the server count exceeds what a config file can manage. The line is usually around 5-8 servers. ### Version Drift The hard problem is not connection — it is **version drift**. Server A is at v1.2, server B is at v2.0, server C is still at v0.9. The capabilities differ. The tool names differ. The schemas differ. MCP's capability negotiation handles this at the session level: each session negotiates against whatever version the server speaks. But the agent *author* still has to deal with the fact that the `search_tickets` tool on server A takes `{ query, status }` and the same-named tool on server B (a different vendor's ticket system) takes `{ keyword, state }`. The production discipline: - **Namespace by server.** Tools from server `acme-crm` are surfaced as `acme-crm__search_tickets`, not `search_tickets`. This prevents collisions and tells the model (and the operator) which server owns the call. - **Pin versions.** In the registry/config, record the server version. When a server upgrades, surface the version in the tool metadata so the model (and the operator) knows what changed. - **Treat schema changes as breaking.** If a tool's input schema changes, that is a new tool from the model's perspective. The old schema may produce calls that fail validation against the new one. Version the tool (`search_tickets_v2`) or accept the breakage and re-test the agent. --- ## DD-22.6 — Resources, Subscriptions, and the Flood Risk Resources are passive data the server exposes. A resource has a URI (`ticket://123`, `config://app.yaml`, `log://tail`). The client can read it, list it, and (if the server supports subscriptions) receive updates when it changes. The subscription capability (`resources/subscribe`) is powerful and dangerous. The client subscribes to a URI; the server sends `notifications/resources/updated` whenever the resource changes. This is the right model for "tail this log" or "watch this ticket." The failure mode is **the flood**. A resource that changes rapidly (a busy log, a noisy event stream) generates a notification per change. If the client subscribes to several such resources, the notification volume can overwhelm the client's processing loop. The client's context fills with resource-update noise; the model sees a wall of diffs instead of the conversation. The production discipline: - **Subscribe narrowly.** Subscribe to the specific URI you care about, not a wildcard that matches everything. - **Debounce on the server.** If a resource can change hundreds of times per second (a live log), the server should coalesce updates and send at most one per second, or per client-defined interval. - **Unsubscribe aggressively.** When the model is done with a resource, unsubscribe. Long-lived subscriptions to volatile resources are a leak. - **Treat subscriptions as a capability the server can rate-limit.** A well-designed server caps concurrent subscriptions per client and returns an error if exceeded. --- ## DD-22.7 — Production Patterns and the C4 Connection This deep-dive is the "build a server" side. The security side — rogue servers, prompt injection via tool descriptions, supply-chain attacks — is covered in Course 2B (the SDD-B07 offensive expansion against MCP) and the broader C2B module. Do not duplicate that material here; this deep-dive assumes the server is trustworthy and focuses on building it well. The production patterns that matter for a server operator: - **Connection pooling (HTTP+SSE).** A remote server shared by many agent sessions should pool backend connections (to the database, the vendor API, the cache). One connection per agent session is a leak. - **Health checks.** A remote server should expose a health endpoint (separate from the MCP protocol) that the agent's orchestrator can poll. If the server is unhealthy, the orchestrator routes around it. The MCP protocol itself has no health primitive — this lives outside the protocol. - **Rate limiting.** The server should rate-limit `tools/call` per client. A runaway model (a loop that calls the same tool 10,000 times) will cost real money if the tool calls a paid API. The server, not the client, is the right place for this limit — the client may not know what the tool costs. - **Idempotency for side-effecting tools.** A tool that sends an email, charges a card, or writes to a database should accept an idempotency key. If the client retries (network timeout, process restart), the server deduplicates. This is standard API discipline; MCP inherits it. - **Structured logging to stderr.** Every tool call should be logged (to stderr) with: timestamp, tool name, truncated arguments, duration, success/failure. These logs are the observability layer when something goes wrong in production. ### The C4 Connection (Module E07) Course 4's E07 module covers MCP-at-scale: the registry as a service, the multi-tenant server, the policy layer that gates which agents can call which servers, the audit log of every tool invocation. This deep-dive builds the single server; E07 builds the fleet. Read this first; E07 builds on the assumption that you understand the server lifecycle, capability negotiation, and the three-primitive model established here. --- ## Anti-Patterns ### Logging to stdout on a stdio transport stdout is the protocol stream. A `console.log("got request")` turns into a malformed JSON-RPC message the client tries to parse, fails, and disconnects. **Always** log to stderr. This is the single most common bug in MCP server code, and it is silent in development (where you might be using the HTTP transport) and fatal in production (where you are using stdio). ### Trusting model-produced arguments without re-validation The model is not a reliable schema validator. It will produce arguments with extra fields, wrong types, missing required fields. The server must re-validate against the JSON Schema on every call. If you skip this because "the model knows the schema," you will get a production incident the first time the model is steered to add a `bcc` field to your `send_email` tool. ### One capability blob for everything Don't advertise every capability the server could theoretically support. Advertise what it *currently* supports. A server that advertises `resources/subscribe` but doesn't implement debouncing will flood the client. A server that advertises `tools.listChanged` but never sends the notification is harmless but misleading. Advertise what works; omit what doesn't. ### No graceful shutdown A server that doesn't handle SIGTERM leaves in-flight requests half-finished, temp files on disk, database transactions open. The orchestrator kills the server on a timeout and the world is in a broken state. Handle SIGTERM from day one. ### Treating tool descriptions as documentation The tool description is an instruction to the model. It determines when the model calls the tool and how it formats arguments. A vague description ("searches tickets") produces worse calls than a precise one ("searches support tickets by keyword, returns up to 50 results newest-first, use status to filter"). Write descriptions like prompts, not like docstrings. --- ## Key Terms | Term | Definition | | --- | --- | | MCP | Model Context Protocol — the JSON-RPC 2.0 wire protocol for tool/resource/prompt exchange between agent clients and tool servers. | | Tool | A model-invoked function. Has name, description, JSON Schema input. The primitive that replaces hardcoded tool registries. | | Resource | Server-exposed passive data, addressed by URI. The client reads; the model does not call. | | Prompt | A server-authored, parameterized message template the user invokes by name. | | Capability negotiation | The initialize handshake where client and server exchange what features they support. Determines the valid message set for the session. | | stdio transport | Local transport: the agent spawns the server as a subprocess, communicates over stdin/stdout. | | HTTP+SSE transport | Remote transport: the agent connects over HTTP, receives server-pushed updates via Server-Sent Events. | | JSON-RPC 2.0 | The message envelope. Request (has id, expects reply), Response (carries result or error), Notification (no id, no reply). | | `additionalProperties: false` | JSON Schema directive forbidding extra fields. A security control — prevents the model from injecting unexpected arguments. | | Registry | A service or directory listing available MCP servers. Enables dynamic discovery without agent redeploy. | | Version drift | The condition where connected servers speak different versions, with different tool sets and schemas. Namespacing and pinning address it. | --- ## Lab Exercise See [07-lab-spec.md](07-lab-spec.md) — build a TypeScript MCP server for a ticket system with two tools, schema validation, graceful shutdown, and a test harness that exercises the lifecycle. --- ## References 1. Model Context Protocol Specification — Anthropic, 2024-2025. The canonical protocol document. 2. JSON-RPC 2.0 Specification — jsonrpc.org. The message envelope substrate. 3. Language Server Protocol (LSP) — Microsoft. The predecessor protocol that established the pattern MCP follows. 4. `@modelcontextprotocol/sdk` — the TypeScript SDK. The reference implementation used in this deep-dive's code. 5. DD-01 (Pi) — the hardcoded four-tool harness this deep-dive externalizes. 6. DD-11 (OpenAI Agents SDK) — the 7-provider sandbox abstraction; complementary to MCP's tool-externalization. 7. Course 4 Module E07 — MCP-at-scale: registry as service, multi-tenancy, policy layer, audit. 8. Course 2B SDD-B07 — the offensive expansion against MCP: rogue servers, prompt injection via tool descriptions, supply-chain attacks. (Read this deep-dive first; SDD-B07 is the security counterpart.) --- *Pi's 4 tools are hand-coded. An MCP server externalizes tool definitions into a discoverable, versioned, independently-deployable service. This is the difference between a personal harness and a platform.*