Datadog | Techlist.io

datadogMar 4, 20262 min readCurated summary

Designing MCP tools for agents: Lessons from building Datadog's MCP server | Datadog

Datadog is presented as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The provided content, however, consists almost entirely of Datadog’s website navigation rather than the blog post itself, so it does not include Gartner’s evaluation criteria, Datadog’s strengths, or any supporting analysis. ## Gartner Recognition - The page headline announces Datadog’s “Leader” position in the Gartner Magic Quadrant for Observability Platforms. - A link is provided to a Gartner-related resource page. - No ranking details, competitor comparisons, or Gartner commentary are included in the supplied text. ## Datadog’s Product Coverage The navigation indicates that Datadog offers a broad observability and operations platform spanning: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, data-quality, job, log, and sensitive-data monitoring. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection. - **Software delivery and service management:** CI visibility, testing, developer portals, incident response, SLOs, workflows, and case management. - **AI:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and an MCP server. ## Limitations of the Provided Content - The actual article body is absent. - The text does not explain why Gartner recognized Datadog as a Leader. - It provides no technical findings, customer examples, methodology, or conclusions beyond the headline. The supplied excerpt supports only the conclusion that Datadog announced Gartner recognition and positions itself as a comprehensive observability platform. A substantive summary would require the full article text.

Read original

AI Agents Kubernetes Datadog Observability+3

datadogFeb 18, 20261 min readCurated summary

How we reduced the size of our Agent Go binaries by up to 77% | Datadog

The supplied text does not include the tech blog post itself. It contains Datadog navigation links and a promotional banner announcing its recognition as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms, but no article body or technical sections. ## Available content - Datadog promotes observability products covering: - Infrastructure and Kubernetes monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - Software delivery and CI visibility - Service management - AI-powered investigation and monitoring - The page links to an engineering article at: - `/blog/engineering/agent-go-binaries/` - No technical explanation, examples, conclusions, or section content from that article is included. Please provide the blog post’s full text or relevant excerpt for a substantive summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogFeb 18, 20263 min readCurated summary

How we reduced the size of our Agent Go binaries by up to 77%

The Datadog Agent’s Linux artifact grew from 428 MiB in version 7.16.0 to 1.22 GiB in 7.60.0, creating problems for serverless, IoT, and containerized environments. Rather than remove features, Datadog reduced Go binary sizes by up to 77% between versions 7.60.0 and 7.68.0. The effort combined dependency analysis, targeted code refactoring, and renewed use of Go linker optimizations. ## Why the Agent Became So Large - The Agent supports many operating systems, architectures, distributions, and deployment environments. - Its codebase contains hundreds of dependencies, including cloud SDKs, container runtimes, and security tools. - Build tags and dependency injection determine which features are included in each binary. - The compressed Linux amd64 Debian package grew from 126 MiB to 265 MiB. - Its uncompressed size increased from 428 MiB to 1,248 MiB—a 192% increase over five years. - Go binaries represented a substantial portion of that growth and became the primary optimization target. ## How Go Selects Dependencies - Go compiles required packages individually before the linker combines them into a binary. - Files are included only when they: - Are not test files ending in `_test.go` - Match the current operating system, architecture, and build tags - Satisfy other constraints such as CGO settings, compiler version, or architecture features - Starting from the main package, Go transitively includes imported packages and the runtime required by every Go binary. - Unnecessary dependencies can be excluded by: - Adding a build tag to the file that imports them - Moving dependency-using symbols into a separate package imported only by relevant binaries ## Analyzing Imports and Dependencies - `go list` reveals all packages used for a specific OS, architecture, and set of build tags. - `goda` generates dependency graphs, including indirect imports. - `goda` can also show only the paths leading to a particular target package using its `reach` function. - These tools account for `GOOS`, `GOARCH`, and build constraints, making them useful for examining platform-specific builds. ## Why Package Lists Are Not Enough - A package’s presence does not directly indicate its binary size impact. - The linker removes symbols that are not reachable from the program’s entry points. - The same package can therefore contribute different amounts of code depending on how it is used. - Importing a package can still have significant side effects: - `init` functions execute. - Global variables are initialized. - These behaviors may force otherwise unnecessary symbols to remain in the binary. - Certain uses of reflection can also limit linker optimizations. - Datadog used `go-size-analyzer` to measure the contribution of individual dependencies more accurately than import graphs alone. ## Overall Optimization Strategy - Datadog systematically audited dependencies rather than removing product capabilities. - The work focused on restructuring imports, isolating optional functionality, and restoring linker optimizations that had been disabled or undermined over time. - The resulting improvements brought artifact sizes close to levels from roughly five years earlier. - Some compiler and linker behaviors uncovered during the effort led to improvements benefiting other large Go projects, including Kubernetes. The practical lesson is to treat binary size as an ongoing dependency and architecture concern: analyze actual symbol reachability, isolate optional features behind build constraints or packages, and verify each build variant independently.

Read original

Kubernetes Docker & Containers Go Dependency Injection+4

datadogJan 7, 20261 min readCurated summary

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog

The provided text does not include the blog post’s article body. It contains Datadog’s navigation menu and a link to an engineering post titled around “eBPF workload protection lessons,” so there is not enough source material to accurately summarize its technical arguments or conclusions. ## Available information - The page is hosted by Datadog’s engineering blog. - The linked topic concerns workload protection built with eBPF. - Datadog’s broader product areas include infrastructure monitoring, application performance monitoring, security, logs, and AI. - The excerpt itself does not describe: - The eBPF implementation - Design challenges or trade-offs - Performance considerations - Security detection methods - Lessons learned or recommendations Please provide the article text or a fuller extract for a substantive summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogJan 7, 20263 min readCurated summary

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

Read original

Database Design Observability Ebpf Linux Kernel+4

datadogNov 18, 20252 min readCurated summary

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute | Datadog

Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a provider of broad, integrated monitoring across infrastructure, applications, logs, security, digital experiences, software delivery, and AI. The supplied content does not include Gartner’s evaluation details or the blog post’s supporting arguments. ## Recognition and Positioning - Datadog highlights its designation as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The announcement emphasizes Datadog’s unified observability platform rather than a single monitoring product. ## Breadth of the Platform - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications and data:** APM, database monitoring, continuous profiling, data-stream monitoring, and job monitoring. - **Logs and observability operations:** log management, sensitive-data scanning, audit trails, and observability pipelines. - **Security:** cloud security, SIEM, workload protection, code security, vulnerability management, and application/API protection. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery and service management:** CI visibility, testing, feature flags, incident response, SLOs, workflow automation, and case management. - **AI capabilities:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and investigation tools. Overall, the available material presents Datadog’s Gartner Leader recognition and extensive product coverage, but it does not provide enough article text to summarize the specific reasoning behind the designation.

Read original

Kubernetes Datadog Observability Ebpf+2

datadogNov 18, 20253 min readCurated summary

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute

File integrity monitoring must provide more than proof that a file changed: security teams need to know how, why, and by whom it changed. Datadog found that filesystem scans, inotify, and auditd could not provide sufficient context, reliability, or scalability. An eBPF-based approach delivered kernel-level visibility into processes and containers, but required extensive filtering and edge processing to handle more than 10 billion events per minute. ## Why Traditional Monitoring Falls Short - Periodic scans can miss changes that are made and reverted between scans. - Scans show that a file changed, but not the process, container, or mechanism responsible. - `inotify` lacks the system-level context needed to correlate file events with processes and containers. - `auditd` offers richer information but can impose significant performance overhead and struggle under heavy load. ## eBPF for Context-Rich File Monitoring - eBPF observes file activity directly in the Linux kernel in real time. - Events can include: - The modified file - The process that triggered the change - The container in which the process ran - Additional security-relevant metadata - This context makes events more useful for investigations than simple “file changed” notifications. ## Scaling at the Agent and Backend - Datadog observed more than 10 billion file-related events per minute across its infrastructure. - Each serialized event was approximately 5 KB, making unrestricted transmission infeasible—potentially several terabytes per second. - Sending every event would also overload Agents through excessive CPU, memory, serialization, and network usage. - Agent-side rules filter events locally, discarding noise before transmission. - This reduced the stream to roughly one million events per minute while preserving detection coverage. ## Filtering Events in the Kernel - A basic architecture loads eBPF programs into the Agent, observes system activity, writes events to a ring buffer, and evaluates them in user space. - Sensitive workloads can generate up to 5,000 relevant syscalls per second. - Initial implementations risked ring-buffer backlogs and dropped events, creating security blind spots. - Datadog moved as much evaluation as possible into eBPF programs to reduce the number of events reaching user space. - The Agent could then perform a deeper second-stage evaluation before forwarding events to the backend. ## Two-Stage Evaluation: Approvers and Discarders - eBPF’s safety constraints limit computation, especially on older Linux kernels. - The system therefore separates evaluation into: - **In-kernel filtering:** Lightweight decisions that quickly approve or discard events. - **User-space evaluation:** More complex analysis using richer context, correlations, and logic unsuitable for the kernel. - This design balances kernel safety and performance with the need for detailed security detection. Datadog’s approach shows that scalable FIM requires combining eBPF’s deep visibility with aggressive filtering at the edge and in the kernel. The practical recommendation is to keep expensive analysis in user space while rejecting irrelevant events as early as possible.

Read original

Database Design Linux Ebpf Security Monitoring+4

datadogNov 4, 20253 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

Read original

Database Design PostgreSQL Apache Kafka Elasticsearch+4

datadogNov 4, 20251 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog

The supplied content does not include the blog post’s article text. It contains Datadog’s navigation menu and a promotional link to its Gartner recognition, while the URL suggests the post concerns CDC replication and search. ## Available Information - Datadog was named a **Leader in the 2026 Gartner Magic Quadrant for Observability Platforms**. - The page promotes Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and digital experience - Software delivery and service management - AI-powered observability - The referenced article URL is `engineering/cdc-replication-search`, indicating a likely focus on **change data capture (CDC), data replication, and search systems**. ## Missing Article Details - No sections, technical explanations, architecture diagrams, implementation details, or conclusions from the blog post are present in the supplied text. - A reliable summary of the CDC replication approach cannot be produced without the article body. Please provide the full post text or its main sections for a complete summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+2

datadogOct 21, 20252 min readCurated summary

Detecting malicious pull requests at scale with LLMs | Datadog

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

datadogOct 21, 20253 min readCurated summary

Detecting malicious pull requests at scale with LLMs

Datadog built an LLM-powered security reviewer, BewAIre, to examine every pull request for malicious intent as AI-assisted development increases code volume and reviewer fatigue. Unlike traditional static analysis, the system evaluates the design and likely purpose of changes, not just known syntactic patterns. In testing, it achieved over 99.3% accuracy with a 0.03% false-positive rate and now operates in production. ## Why Conventional Review Is Struggling - Datadog processes nearly 10,000 pull requests per week across internal and external repositories, with volume growing as coding assistants become commonplace. - More changes increase the attack surface and make subtle attacks harder to notice. - Reviewers, linters, and scanners can be overwhelmed by verbose or repetitive AI-generated code. - Static analysis is effective at identifying known bad patterns but generally lacks understanding of developer intent. - Legitimate security-sensitive changes, such as permission-model updates, can resemble malicious behavior. ## The tj-actions/changed-files Attack - An attacker used a compromised personal access token belonging to `tj-actions-bot` to modify `index.js`. - The injected Python code extracted secrets and other sensitive information from a GitHub Actions runner’s process memory. - Base64 encoding concealed both the malicious code reference and output written to build logs. - The commit was made to look like a legitimate `renovate[bot]` update. - Version tags were redirected to the malicious commit, allowing the compromise to spread quickly. - Even protected branches, two-person reviews, and other standard controls could miss an attack disguised as a routine dependency or maintenance change. ## BewAIre’s LLM-Based Review Pipeline - **Ingestion:** Collects pull requests merged into the default branch across Datadog repositories. - **Preprocessing:** Normalizes each pull request, extracts its diff, and adds metadata such as the author and repository type. - **Inference:** Sends the code changes and contextual information to an LLM to reason about intent and classify the change as benign or malicious. - **Security signals:** Converts each verdict into a Datadog security signal that appears in internal dashboards and can trigger alerts or pages for security engineers. - Each finding includes a written explanation of why the change may be malicious. ## Improving Accuracy and Limiting Noise - Datadog used prompt engineering and data tuning to improve the model’s ability to distinguish malicious behavior from legitimate changes. - Suppression rules were added for known-safe patterns. - Testing on a curated, balanced dataset of hundreds of malicious and benign pull requests produced: - More than 99.3% overall accuracy. - More than 93.5% balanced accuracy. - A 0.03% false-positive rate. - The team also designed the system to address model degradation caused by context limits and continuously evaluated it against real-world attacks, including the tj-actions and Nx incidents. ## Production Deployment - BewAIre reviews Datadog pull requests in real time and is already deployed across the company’s repositories. - The system is available in Preview to Static Code Analysis customers. - Its purpose is to add scalable, intent-focused detection without imposing stricter manual review requirements that could slow development. Datadog’s approach suggests that LLMs can complement—not replace—static analysis and human investigation by providing continuous, intent-aware security review at large scale.

Read original

Large Language Models Prompt Engineering Code Review Git+4

datadogOct 15, 2025Original article

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | Datadog (opens in new tab)

Following a major 2023 incident that caused a near-total platform outage despite partial infrastructure availability, Datadog shifted its engineering philosophy from "never-fail" architectures to a model of graceful degradation. The company identified that prioritizing absolute data correctness during systemic stress created "square-wave" failures, where the entire platform appeared down if even a portion of data was missing. By moving toward a "fail better" mindset, Datadog now focuses on maintaining core functionality and data persistence even when underlying infrastructure is compromised. ## Limitations of the Never-Fail Approach * Classical root-cause analysis focused on a legacy, unsupervised global update mechanism that disconnected 50–60% of production Kubernetes nodes. * While the "precipitating event" was easily identified and disabled, the engineering team realized that fixing the trigger did not address the systemic fragility that caused a binary (up/down) failure pattern. * Prioritizing absolute accuracy meant that systems would wait for all data tags to process before displaying results; under stress, this caused the UI to show no data at all rather than "almost correct" data. * Sequential queuing, aggressive retry logic, and node-specific processing requirements exacerbated the bottleneck, preventing real-time recovery. ## Prioritizing Graceful Degradation * The incident prompted a shift away from relying solely on redundancy to prevent outages, acknowledging that some level of failure is eventually inevitable at scale. * Engineering priorities were redefined to ensure that data is never lost (even if delayed) and that real-time data is processed before stale backlogs. * The platform now aims to serve partial-but-accurate results to customers during an incident, providing visibility rather than a complete blackout. * Implementation is handled as a company-wide program where individual product teams adapt these principles to their specific architectural needs. ## Strengthening Data Persistence at Intake * Analysis revealed that data was lost during the outage because it was stored in memory or on local disks before being replicated to persistent stores. * The original design favored low-latency responses by acknowledging receipt of data before it was fully replicated, making that data unrecoverable if the node failed. * Downstream failures caused intake nodes to overflow their local buffers, leading to data loss even on nodes that remained online. * New architectural changes focus on implementing disk-based persistence at the very beginning of the processing pipeline to ensure data survives node restarts and downstream congestion. To build truly resilient systems, engineering teams must move beyond trying to prevent every possible failure trigger. Instead, focus on designing services that can survive partial infrastructure loss by prioritizing data persistence and allowing for degraded states that still provide value to the end user.

Kubernetes System Design Graceful Degradation Incident Analysis+4

datadogOct 15, 20253 min readCurated summary

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

Read original

Database Design Kubernetes Data Loss Prevention Graceful Degradation+4

datadogOct 1, 20251 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, but no substantive discussion of the linked “Husky Query Architecture” article. ## Available Content ### Datadog’s Observability Platform - Datadog promotes products covering: - Infrastructure and container monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI and agent observability - The navigation emphasizes Datadog’s broad, integrated platform approach. ### Gartner Recognition - The page links to Datadog’s announcement that it was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The supplied text does not include the evaluation criteria, cited strengths, limitations, or Gartner’s comparative analysis. No reliable summary of the Husky query architecture can be produced without the article’s body text.

Read original

Kubernetes Observability Application Performance Monitoring Log Management+4

datadogOct 1, 20253 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events

Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.

Read original

Database Design Data Architecture Foundationdb Husky+3

Filter by tag