Introducing Kafka-Kit: Tools for scaling Kafka
Datadog operates Kafka at extreme scale, ingesting trillions of data points daily and requiring petabytes of NVMe storage. To manage frequent data movement caused by scaling, recovery, and capacity changes, the company built Kafka-Kit, a set of operational tools that improve partition placement and replication control. Its primary tools, `topicmappr` and `autothrottle`, automate safer and more predictable Kafka operations. ## Kafka-Kit - Kafka-Kit addresses two major operational areas: - Data placement across brokers - Replication auto-throttling - Its main tools are: - `topicmappr`, for generating partition-to-broker mappings - `autothrottle`, for automatically controlling replication bandwidth ## Partition Placement with `topicmappr` `topicmappr` replaces Kafka’s `kafka-reassign-partitions.sh --generate` functionality while adding operational safeguards and placement controls. - Produces deterministic output: identical inputs generate the same partition map. - Supports minimal-movement broker replacement: - Failed brokers can be replaced without unnecessarily moving healthy partitions. - Partitions with complete in-sync replicas are normally left untouched. - Provides rack-aware placement using Kafka’s `broker.rack` metadata and ZooKeeper. - Supports placement based on: - Partition count - Storage size, enabling bin-packing and storage rebalancing - Allows replication factors to be increased or decreased while topics are running. - Generates clear summaries of: - Brokers being removed or added - Partition-level changes - Broker distribution before and after reassignment - Warnings and resulting partition-map files The tool is written in Go and can run from any system with access to Kafka’s ZooKeeper cluster. It requires topic names and broker IDs, then verifies that the brokers are live, sufficiently numerous, and properly distributed across configured localities. ## Replacing Failed Brokers For a failed broker, `topicmappr` can rebuild affected topics while limiting movement to the necessary partitions. - Existing replicas are preserved whenever possible. - Replacement brokers fill the gaps left by failed brokers. - The generated report makes the proposed changes visible before execution. - The example replaces broker `1002` with brokers `1003` and `1004`, showing the updated replica assignments and broker totals. ## Placement Strategies `topicmappr` offers multiple strategies for deciding where replicas should live, including `count` and tunable `storage` placement. ### Count Placement Strategy - The default strategy. - Balances leadership and the number of partitions held by each broker. - Works well when traffic is expected to be distributed evenly across partitions. - Does not require metrics data, allowing maps to be generated quickly. - Also attempts to maximize the number of distinct broker-to-broker replica relationships. - This avoids concentrating a broker’s partitions with the same small subset of peers, improving distribution across the cluster and its racks. ## Storage-Aware Placement - The storage strategy uses partition size when assigning replicas. - It supports storage bin-packing and rebalancing, which is important when brokers have uneven disk utilization. - This is particularly useful for Datadog’s large Kafka clusters, where storage capacity—not just partition count—can determine when data must be moved. Datadog’s approach demonstrates that Kafka’s flexible primitives can be extended with purpose-built tooling. For large deployments, deterministic assignments, rack awareness, minimal movement, and storage-based balancing can make scaling and failure recovery substantially safer and more predictable.
Read original(opens in new tab)