How we scaled fast, reliable configuration distribution to thousands of workload containers

Datadog’s seemingly simple tenant-configuration CRUD system must propagate updates rapidly and reliably to thousands of containers processing millions of logs per second. Loading configuration on every log is too expensive, while periodic caching introduces stale data and delayed updates. Datadog initially used database-backed caches invalidated through Kafka, but growing scale exposed reliability and resilience problems tied to repeated workload access to the central database.

The Challenge of Propagating Context Data

• Datadog calls tenant-specific settings—such as log parsing rules, Sensitive Data Scanner settings, and storage quotas—“context data.”
• Configuration changes are expected to take effect almost immediately, including in Live Tail.
• The same context data may be consumed by thousands of containers handling traffic for many tenants.
• Because configuration directly affects customer-data processing, propagation must be both low-latency and highly reliable.
• The system must assume that failures can occur anywhere in a large distributed environment.

Why On-Demand Fetching and Simple Caching Fail

• Fetching configuration from a database for every incoming log would create an impractical read load.
  • Large tenants can generate hundreds of thousands of logs per second.
  • Each processing instance could require thousands of database reads per second.
  • Multiplying this across many instances would require extensive, highly performant database replicas.
• Caching configuration in each workload container reduces reads but does not eliminate the scaling problem.
  • Many workload instances still cache data for a high number of tenants.
  • Increasing the cache interval reduces database load but delays configuration updates.
  • With periodic invalidation, the average propagation delay is roughly half the cache interval.

Context Loading v1: Database-Backed Caches and Kafka

Datadog’s first successful architecture kept tenant configuration in a central durable database while allowing workload containers to cache entries indefinitely.

• A user changes a log-processing configuration.
• The central context database stores the update.
• Kafka publishes an invalidation message after the database write.
• Every workload container receives the notification.
• Each container reloads the affected tenant’s configuration from the database.
• This minimized routine database reads while preserving low-latency updates.

Why the Initial Architecture Needed Reconsideration

• The design required every workload instance to reach the central context database whenever a configuration changed.
• As Datadog added more workloads and containers, update-related database traffic grew substantially.
• Internal game days and production incidents showed that problems affecting the context database could spread to downstream processing workloads.
• Database failures could prevent configuration updates from propagating and potentially make it impossible for new workload containers to initialize their context.
• These reliability concerns demonstrated that Kafka-based invalidation alone did not sufficiently isolate workload processing from context-database failures.

Datadog’s experience shows that configuration propagation at large scale requires more than a durable database and cache invalidation. The system must also reduce dependency on the central database during updates and startup, while continuing to provide near-immediate, reliable propagation.