Designing MCP tools for agents: Lessons from building Datadog's MCP server

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations.

Context Efficiency Matters

• Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB.
• CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record.
• YAML can reduce token usage for nested data by around 20% compared with JSON.
• Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size.
• Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget.
• Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached.
• Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future.

Let Agents Query Data

• Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors.

• Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers.

• SQL lets agents aggregate and filter data directly:

  SELECT service, COUNT(*) AS error_count
  FROM logs
  WHERE status = 'error'
  GROUP BY service
  ORDER BY error_count DESC
  LIMIT 10
  

• Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data.

• SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper.

• Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient.

Tools Are Not Free

• Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions.
• Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count.
• Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs.
• Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context.
• Layering introduces additional tool calls and therefore increases latency.
• Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time.

The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.