Evolving Cloudflare’s Threat Intelligence Platform: actionable, scalable, and ETL-less
Cloudflare’s Threat Intelligence Platform (TIP) is designed to turn massive volumes of security telemetry into actionable intelligence without relying on traditional ETL pipelines. Its sharded, SQLite-backed architecture uses Durable Objects and edge-based GraphQL to provide near-real-time analysis across millions of events. By combining automated telemetry with analyst investigations, the platform aims to help security teams understand threats and block them proactively. ## Motivation for Building the Platform - Cloudflare began developing the TIP after launching Cloudforce One in 2022 and discovering that existing tools could not adequately track adversary infrastructure. - The platform models the full threat lifecycle, connecting: - Threat actors to malware - Cases to indicators - Events to broader campaigns - It is designed for: - Multiple datasets and tenants - Group-based and tenant-to-tenant sharing - Extensibility and edge-scale performance - Visual analysis and automated response - Cloudflare Workers allow the platform to evolve with the runtime and support features such as Smart Placement, higher CPU limits, and Hyperdrive. ## Beyond the SIEM - The TIP complements rather than replaces a SIEM: - SIEMs focus on real-time log aggregation and alerting. - The TIP provides long-term retention, specialized threat schemas, and historical context. - Analysts can enrich alerts with: - Indicator history - Known threat-actor associations - Campaign relationships - Risk scores and intelligence context - Findings from analysts feed new indicators of compromise back into the platform. - This feedback loop keeps intelligence current and helps organizations move from reactive investigation to proactive defense. ## Sharded Storage Without ETL Bottlenecks - Cloudflare distributes Threat Events across many logical shards instead of using one centralized database. - Each shard is a Durable Object with a private SQLite database, providing transactional consistency and avoiding a single database bottleneck. - Cloudflare Queues handle asynchronous ingestion, helping absorb high-volume attack spikes. - R2 stores data for long-term retention, while SQLite maintains a hot index for fast access. - Because data is available directly in the platform’s operational store, complex ETL pipelines and synchronization delays are avoided. ## Parallel Queries at the Edge - GraphQL runs in the same Worker-based system that powers the Threat Events platform, keeping data live from ingestion through querying. - Queries are fanned out to relevant Durable Objects in parallel rather than executed against one large table. - The platform first verifies permissions and excludes shards that cannot contain matching events, such as shards outside the requested date range. - Results from multiple shards are aggregated with `Promise.all`, enabling low-latency searches across global datasets. - Smart Placement positions query Workers near the Durable Objects they access, reducing tail latency. Cloudflare’s approach combines edge-native storage, parallel execution, and analyst-driven enrichment to make threat intelligence both scalable and actionable. The practical goal is a unified system that explains not only what is malicious, but also why it matters and how to automatically prevent it.
Read original(opens in new tab)