Event Processing | Techlist.io

datadogNov 18, 20253 min readCurated summary

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute

File integrity monitoring must provide more than proof that a file changed: security teams need to know how, why, and by whom it changed. Datadog found that filesystem scans, inotify, and auditd could not provide sufficient context, reliability, or scalability. An eBPF-based approach delivered kernel-level visibility into processes and containers, but required extensive filtering and edge processing to handle more than 10 billion events per minute. ## Why Traditional Monitoring Falls Short - Periodic scans can miss changes that are made and reverted between scans. - Scans show that a file changed, but not the process, container, or mechanism responsible. - `inotify` lacks the system-level context needed to correlate file events with processes and containers. - `auditd` offers richer information but can impose significant performance overhead and struggle under heavy load. ## eBPF for Context-Rich File Monitoring - eBPF observes file activity directly in the Linux kernel in real time. - Events can include: - The modified file - The process that triggered the change - The container in which the process ran - Additional security-relevant metadata - This context makes events more useful for investigations than simple “file changed” notifications. ## Scaling at the Agent and Backend - Datadog observed more than 10 billion file-related events per minute across its infrastructure. - Each serialized event was approximately 5 KB, making unrestricted transmission infeasible—potentially several terabytes per second. - Sending every event would also overload Agents through excessive CPU, memory, serialization, and network usage. - Agent-side rules filter events locally, discarding noise before transmission. - This reduced the stream to roughly one million events per minute while preserving detection coverage. ## Filtering Events in the Kernel - A basic architecture loads eBPF programs into the Agent, observes system activity, writes events to a ring buffer, and evaluates them in user space. - Sensitive workloads can generate up to 5,000 relevant syscalls per second. - Initial implementations risked ring-buffer backlogs and dropped events, creating security blind spots. - Datadog moved as much evaluation as possible into eBPF programs to reduce the number of events reaching user space. - The Agent could then perform a deeper second-stage evaluation before forwarding events to the backend. ## Two-Stage Evaluation: Approvers and Discarders - eBPF’s safety constraints limit computation, especially on older Linux kernels. - The system therefore separates evaluation into: - **In-kernel filtering:** Lightweight decisions that quickly approve or discard events. - **User-space evaluation:** More complex analysis using richer context, correlations, and logic unsuitable for the kernel. - This design balances kernel safety and performance with the need for detailed security detection. Datadog’s approach shows that scalable FIM requires combining eBPF’s deep visibility with aggressive filtering at the edge and in the kernel. The practical recommendation is to keep expensive analysis in user space while rejecting irrelevant events as early as possible.

Read original

Event Processing Database Design Linux Ebpf+4

datadogJan 29, 20253 min readCurated summary

Husky: Efficient compaction at Datadog scale

Husky is a distributed event store built on object storage for observability workloads reaching trillions of events per day. Because data is written continuously, rarely updated, and queried both recently and historically, its storage layer must minimize object-store fetches while still supporting high query parallelism. The central design challenge is choosing a compaction and layout strategy that keeps fragments manageable without sacrificing query speed. ## Husky’s Query Execution Model - Ingested events are grouped into files called **fragments** and stored in systems such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. - Metadata for each fragment is stored separately in FoundationDB. - For each query: - Metadata is scanned to identify relevant fragments. - Fragments are distributed among query workers. - Workers scan their assigned data. - Results are merged. - Query cost depends mainly on: - The number of fragments fetched from object storage. - The number of events scanned within those fragments. - Husky therefore focuses on: - Reducing the total number of files through efficient compaction. - Organizing data so queries scan as few irrelevant events as possible. ## The Compaction “Goldilocks” Problem - Compaction combines many small fragments into a larger fragment containing the same data. - FoundationDB transactions atomically replace the old fragments with the compacted one, ensuring queries see a consistent state either before or after compaction. - Ingestion writers buffer events per tenant to avoid producing extremely small files, but they flush periodically to keep newly ingested data queryable quickly. - These flushed fragments may contain only a few thousand events, making queries inefficient when they must fetch thousands of objects and metadata records. ## Balancing Fragment Size Husky must find a fragment size that balances several competing concerns: - **Object storage and metadata overhead** - Fewer, larger fragments reduce the number of fetches and metadata entries. - **Compaction cost** - Larger or more aggressively reorganized fragments require more CPU and more object-storage GET and PUT operations. - **Query parallelism** - Smaller fragments allow more workers to operate concurrently. - Larger fragments reduce distribution overhead but can limit parallelism for large analytical queries. - **Scan efficiency** - Query workers use vectorized execution, which is most effective when scanning sufficiently large batches of rows. - **Data locality and compression** - Compaction can place events with similar timestamps or tags near one another. - This improves compression and allows queries to skip irrelevant data, but requires additional processing and query-pattern analysis. Fragments that are too small create excessive fetch and scheduling overhead. Fragments that are too large reduce parallelism and can make broad queries slower. The goal is a “just right” size suited to typical query patterns. ## Storage Layout and Query Selectivity - Husky organizes events along both: - The time dimension. - Spatial dimensions such as tags. - Keeping commonly queried events close together reduces the amount of data that must be scanned. - Similar data also compresses more effectively. - Achieving this layout increases compaction work, creating a tradeoff between lower query cost and lower maintenance cost. ## Scalable Compaction - Husky’s storage system depends on compaction being efficient enough to run continuously at very large scale. - The design must account not only for the final fragment size, but also for the CPU and object-storage costs required to produce it. - Atomic metadata updates ensure that compaction can occur without exposing partial results or inconsistent table states. Husky’s approach treats compaction as a core part of query performance rather than simple file maintenance. A practical design must tune fragment sizes, merge frequency, and data layout together to minimize total system cost while preserving fast access to both recent events and large historical datasets.

Read original

Event Processing Database Design Query Optimization Foundationdb+4