Consul at Datadog

Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery.

Consul Server Capacity and CPU

Consul servers use Raft consensus to elect a leader and coordinate the cluster.

• Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds.
• Frequent leadership transitions usually indicate insufficient CPU capacity.
• Datadog’s approximate sizing guidance:
  • m3.large: about 300 agent nodes
  • c3.xlarge: about 500 agent nodes
  • c3.2xlarge: about 800 agent nodes
• If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily.
• Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal.

Auditable Configuration Changes

Consul’s key-value store is useful for distributing configuration throughout a cluster.

• Configuration can be retrieved through HTTP or delivered through Consul watches.
• Direct edits without an audit trail make it difficult to determine who changed a value and when.
• git2consul distributes configuration from a Git repository, providing version control and accountability.
• Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day.

Access Control with ACLs

Consul ACLs prevent unauthorized processes from modifying or deleting key-value data.

• Tokens should be limited to the data and operations each process requires.
• Scoped permissions reduce the impact of accidental changes.
• ACLs provide an important safety boundary between services and configuration areas.

Watches Instead of Excessive Polling

Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached.

• Watches notify clients when key-value data changes.
• This reduces unnecessary polling and distributes updates efficiently.
• Watches can sometimes trigger unexpectedly or too frequently.
• Tools such as sifter can help protect systems from excessive watch activity.

Using dnsmasq for Service Discovery

Applications using Consul’s DNS interface can reduce load by placing dnsmasq between clients and Consul.

• Use short DNS TTLs; Datadog commonly uses 10 seconds.
• Query dnsmasq rather than Consul directly so repeated answers can be cached locally.
• At very high request volumes, cache Consul services in an additional hosts file loaded by dnsmasq.
• This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul.
• goshe can collect dnsmasq statistics for monitoring.

Monitoring Cluster Health

Monitoring is essential for operating Consul reliably.

• consul.consul.leader.reconcile.count should remain stable and indicate that a leader exists.
• consul.serf.events.consul_new_leader shows leadership transitions; frequent events suggest instability.
• consul.raft.leader.lastContact measures how recently nodes contacted the leader.
• consul.consul.dns.domain_query.count reveals how many DNS requests are reaching Consul directly.
• Also monitor CPU and network usage on Consul servers.

Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and dnsmasq absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.