Introducing Kafka-Kit: Tools for scaling Kafka

Datadog operates Kafka at extreme scale, ingesting trillions of data points daily and requiring petabytes of NVMe storage. To manage frequent data movement caused by scaling, recovery, and capacity changes, the company built Kafka-Kit, a set of operational tools that improve partition placement and replication control. Its primary tools, topicmappr and autothrottle, automate safer and more predictable Kafka operations.

Kafka-Kit

• Kafka-Kit addresses two major operational areas:
  • Data placement across brokers
  • Replication auto-throttling
• Its main tools are:
  • topicmappr, for generating partition-to-broker mappings
  • autothrottle, for automatically controlling replication bandwidth

Partition Placement with topicmappr

topicmappr replaces Kafka’s kafka-reassign-partitions.sh --generate functionality while adding operational safeguards and placement controls.

• Produces deterministic output: identical inputs generate the same partition map.
• Supports minimal-movement broker replacement:
  • Failed brokers can be replaced without unnecessarily moving healthy partitions.
  • Partitions with complete in-sync replicas are normally left untouched.
• Provides rack-aware placement using Kafka’s broker.rack metadata and ZooKeeper.
• Supports placement based on:
  • Partition count
  • Storage size, enabling bin-packing and storage rebalancing
• Allows replication factors to be increased or decreased while topics are running.
• Generates clear summaries of:
  • Brokers being removed or added
  • Partition-level changes
  • Broker distribution before and after reassignment
  • Warnings and resulting partition-map files

The tool is written in Go and can run from any system with access to Kafka’s ZooKeeper cluster. It requires topic names and broker IDs, then verifies that the brokers are live, sufficiently numerous, and properly distributed across configured localities.

Replacing Failed Brokers

For a failed broker, topicmappr can rebuild affected topics while limiting movement to the necessary partitions.

• Existing replicas are preserved whenever possible.
• Replacement brokers fill the gaps left by failed brokers.
• The generated report makes the proposed changes visible before execution.
• The example replaces broker 1002 with brokers 1003 and 1004, showing the updated replica assignments and broker totals.

Placement Strategies

topicmappr offers multiple strategies for deciding where replicas should live, including count and tunable storage placement.

Count Placement Strategy

• The default strategy.
• Balances leadership and the number of partitions held by each broker.
• Works well when traffic is expected to be distributed evenly across partitions.
• Does not require metrics data, allowing maps to be generated quickly.
• Also attempts to maximize the number of distinct broker-to-broker replica relationships.
• This avoids concentrating a broker’s partitions with the same small subset of peers, improving distribution across the cluster and its racks.

Storage-Aware Placement

• The storage strategy uses partition size when assigning replicas.
• It supports storage bin-packing and rebalancing, which is important when brokers have uneven disk utilization.
• This is particularly useful for Datadog’s large Kafka clusters, where storage capacity—not just partition count—can determine when data must be moved.

Datadog’s approach demonstrates that Kafka’s flexible primitives can be extended with purpose-built tooling. For large deployments, deterministic assignments, rack awareness, minimal movement, and storage-based balancing can make scaling and failure recovery substantially safer and more predictable.