How we built Cloudflare's data platform and an AI agent on top of it (opens in new tab)
Cloudflare built Town Lake to unify data scattered across production databases, analytics systems, streams, and object storage behind one governed SQL interface. The platform combines Trino, Iceberg on R2, DataHub, and custom access-control and PII-detection services to make data fresher, more discoverable, and safer to use. Skipper extends Town Lake with a natural-language AI interface intended to provide fast, accurate, and auditable answers without requiring users to write SQL. ## The Data Sprawl Problem - Cloudflare processes over a billion events per second across a network spanning more than 330 cities and 120 countries. - Relevant data was distributed across: - Postgres - ClickHouse - BigQuery - Kafka - Google Cloud Storage and R2 - Numerous pipelines and production databases - Users needed separate credentials, query languages, retention expectations, and system knowledge for each source. - Sampled analytics data worked for dashboards but was unsuitable for billing, usage calculations, and security investigations. - External vendors created cost and dependency concerns. - Important data was difficult to discover because table locations, schemas, joins, and customer-ID mappings depended on tribal knowledge. - Data infrastructure had historically been treated as a back-office service rather than core company infrastructure. ## Goals for the New Platform Cloudflare wanted a single place where authorized employees could answer questions about customers, traffic, billing, security events, and support activity. - Support both: - Fresh, accurate, unsampled data for billing and investigations - Fast, downsampled data for dashboards and exploration - Provide built-in governance: - Automatic PII detection - Sensitive tables locked down by default - Auditable access - Time-limited permission grants - Build the system using Cloudflare’s own products, including R2, Workers, Access, and Workflows. - Eventually let employees ask questions in plain English rather than requiring SQL knowledge. - That natural-language interface became Skipper. ## Town Lake’s Lakehouse Architecture Town Lake is a lakehouse: a query engine combines data from object storage and operational systems while a metadata layer makes the data behave like a unified database. - **Trino** serves as the query engine. - A single query can join Postgres, ClickHouse, and Iceberg tables stored on R2. - Trino pushes filters into source systems and combines results without requiring intermediate materialization. - **R2 Data Catalog and Apache Iceberg** store warm and cold data. - Iceberg provides schema evolution, time travel, partition evolution, and compaction. - Data can be rolled from per-minute to hourly and eventually daily granularity as it ages. - Older data becomes cheaper to store while remaining queryable. - Parquet files on R2 cost less than retaining equivalent data in an OLAP database. - **DataHub** provides the metadata catalog. - It stores table and column descriptions, owners, lineage, and glossary terms. - Users can discover what a table contains, which teams maintain it, and how it relates to upstream and downstream data. ## Access Control and Privacy - **Lifeguard** manages access policies. - Rules are stored in D1. - User and group memberships are retrieved dynamically from Cloudflare’s internal access-management system. - Lifeguard produces JSON policies that Trino reads over HTTP. - It also supplies access information to Skipper and the Gateway, allowing users to be blocked before queries execute. - **Skimmer** continuously scans tables for PII. - It samples rows from columns across the data platform. - Workers AI classifies whether columns contain personally identifiable information. Cloudflare’s overall approach is to combine unified querying, durable low-cost storage, rich metadata, and policy enforcement so data can be broadly useful without sacrificing accuracy or governance.