Consul at Datadog
Consul has become a critical part of Datadog’s production infrastructure for distributing configuration and discovering services. After 18 months of use, the main lesson is that Consul requires careful capacity planning, controlled access, efficient query patterns, and continuous monitoring. The recommendations aim to keep clusters stable while supporting frequent configuration updates and high-volume service discovery. ## Consul Server Capacity and CPU Consul servers use Raft consensus to elect a leader and coordinate the cluster. - Followers trigger a leadership transition if they cannot hear from the leader for 500 milliseconds. - Frequent leadership transitions usually indicate insufficient CPU capacity. - Datadog’s approximate sizing guidance: - `m3.large`: about 300 agent nodes - `c3.xlarge`: about 500 agent nodes - `c3.2xlarge`: about 800 agent nodes - If transitions occur hourly or more often, increase server CPU capacity until they happen no more than daily. - Standard monitoring may miss brief 500-millisecond CPU spikes, so reducing CPU pressure is important even when dashboards look normal. ## Auditable Configuration Changes Consul’s key-value store is useful for distributing configuration throughout a cluster. - Configuration can be retrieved through HTTP or delivered through Consul watches. - Direct edits without an audit trail make it difficult to determine who changed a value and when. - `git2consul` distributes configuration from a Git repository, providing version control and accountability. - Datadog uses it for cluster-wide configuration updates roughly every 60 seconds, dozens of times per day. ## Access Control with ACLs Consul ACLs prevent unauthorized processes from modifying or deleting key-value data. - Tokens should be limited to the data and operations each process requires. - Scoped permissions reduce the impact of accidental changes. - ACLs provide an important safety boundary between services and configuration areas. ## Watches Instead of Excessive Polling Consul can handle substantial traffic, but it should not be queried hundreds of thousands of times per second like Redis or Memcached. - Watches notify clients when key-value data changes. - This reduces unnecessary polling and distributes updates efficiently. - Watches can sometimes trigger unexpectedly or too frequently. - Tools such as `sifter` can help protect systems from excessive watch activity. ## Using dnsmasq for Service Discovery Applications using Consul’s DNS interface can reduce load by placing `dnsmasq` between clients and Consul. - Use short DNS TTLs; Datadog commonly uses 10 seconds. - Query `dnsmasq` rather than Consul directly so repeated answers can be cached locally. - At very high request volumes, cache Consul services in an additional hosts file loaded by `dnsmasq`. - This setup served over 100,000 DNS requests per second while sending only about 400 requests per second directly to Consul. - `goshe` can collect `dnsmasq` statistics for monitoring. ## Monitoring Cluster Health Monitoring is essential for operating Consul reliably. - `consul.consul.leader.reconcile.count` should remain stable and indicate that a leader exists. - `consul.serf.events.consul_new_leader` shows leadership transitions; frequent events suggest instability. - `consul.raft.leader.lastContact` measures how recently nodes contacted the leader. - `consul.consul.dns.domain_query.count` reveals how many DNS requests are reaching Consul directly. - Also monitor CPU and network usage on Consul servers. Consul works best when server nodes have sufficient CPU, configuration changes are managed through version control, ACLs restrict access, watches replace aggressive polling, and `dnsmasq` absorbs service-discovery traffic. Continuous monitoring of leadership, Raft connectivity, DNS load, CPU, and networking helps identify failures before they affect production.
Read original(opens in new tab)