Datadog’s page announces that the company was named a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms. However, the provided content contains only the page header, navigation links, and product categories—not the blog post itself—so its technical argument and supporting details cannot be reliably summarized.
## Available Content
- Announcement:
- Datadog was named a Leader in the Gartner® Magic Quadrant™ for Observability Platforms.
- The page links to a Gartner-related resource.
- Product areas listed:
- Infrastructure and application monitoring
- Logs, databases, and data observability
- Security and cloud security
- Digital experience monitoring
- CI/CD and software delivery
- Service management
- AI and observability tools
- The URL references a “noise suppression library,” but no corresponding article text was included.
Please provide the full blog post content for a substantive summary.
Datadog’s Log Forwarding system resembles a package delivery network: it must move large volumes of data efficiently and reliably to many unpredictable destinations. Kafka provides ordered transport, but its FIFO behavior creates difficult tradeoffs when endpoints are slow or unavailable. The central challenge is preserving delivery guarantees without losing logs, creating duplicates, blocking unrelated destinations, or overwhelming customer infrastructure.
## What Log Forwarding Does
- Datadog forwards processed, enriched logs as schemaless JSON records.
- Destinations can include:
- Elasticsearch
- Splunk
- Generic HTTP endpoints accepting JSON `POST` requests
- The system must support thousands of tenants and external endpoints with widely varying reliability and performance.
## Kafka as the Distribution Network
- Logs move through Datadog on Kafka topics, analogous to packages traveling on conveyor belts.
- Each Kafka partition provides strict FIFO ordering:
- Records are read in the order they were written.
- Kafka offsets must be committed in that same order.
- Logs for different destinations are spread across multiple partitions, so records for a single destination may need to be regrouped during delivery.
- Assigning a dedicated Kafka partition to every destination would be simple conceptually but infeasible at scale.
## Reliability Challenges
- External endpoints may be:
- Temporarily unavailable
- Slow or unstable
- Unreachable for hours or days
- The system must avoid:
- Losing customer logs
- Sending duplicate logs
- Delaying all destinations because one endpoint is unhealthy
- Excessive resource usage
- Overwhelming or effectively DDoSing a customer endpoint
- Sending one HTTP request per log would be inefficient, so logs should be buffered and delivered in batches, much like packages going to the same address.
## Kafka Ordering and Blocked Progress
- Waiting for each forwarding request to succeed before reading more Kafka data protects against data loss but can halt progress.
- Continuing to read and acknowledge Kafka records before successful delivery risks losing logs.
- Because offsets must be committed in order, one unavailable destination can block later records in the same partition—even if those records belong to healthy destinations.
- This makes coordination between Kafka consumption, retries, batching, and concurrent delivery especially complex in a multi-tenant system.
## Lessons from Log Archives
- Datadog had prior experience with similar delivery problems in its Log Archives feature.
- Archiving was easier because:
- Cloud object storage endpoints are generally more reliable.
- Archiving has lower latency requirements.
- Those lessons helped the team anticipate reliability and ordering pitfalls in Log Forwarding.
## Dedicated Kafka Topics per Destination
- A possible solution would be to assign one or more Kafka partitions to each destination.
- This would isolate destinations so that one slow endpoint could not block others.
- However, the approach would require an impractically large number of Kafka topics or partitions as the number of customers and destinations grows.
Datadog’s “Reliable Log Delivery” post explains how log-collection systems can avoid losing data when networks, destinations, or agents fail. Its central recommendation is to combine acknowledgments, buffering, retries, and controlled backpressure to provide at-least-once delivery without allowing outages to overwhelm the collector.
## Why Reliable Delivery Matters
- Logs are often needed during incidents, precisely when infrastructure and networks may be unstable.
- Temporary destination failures can cause data loss if collectors only keep logs in memory.
- Retrying without limits can create duplicate logs, unbounded memory usage, or cascading failures.
## Buffering and Persistence
- Collectors should buffer logs while downstream services are unavailable.
- In-memory buffers provide speed but cannot survive process crashes or host restarts.
- Disk-backed queues improve durability by preserving unsent logs across transient failures.
- Storage limits are necessary so a prolonged outage does not fill the host’s disk.
## Acknowledgments and Retries
- A log should be removed from the queue only after the destination confirms successful receipt.
- Failed or unacknowledged deliveries are retried, allowing temporary network and service failures to recover automatically.
- At-least-once delivery is the practical reliability target, meaning duplicates may occur and downstream systems should handle them safely.
- Retry policies should use delays and backoff rather than continuously retrying at full speed.
## Backpressure and Operational Trade-offs
- When downstream systems slow down, collectors must apply backpressure instead of accepting unlimited data.
- Backpressure can limit memory consumption and protect the rest of the host.
- Teams must define what happens when buffers reach capacity, such as dropping the oldest data, rejecting new logs, or prioritizing important streams.
- Reliability also requires monitoring queue size, delivery latency, retry rates, and dropped records.
A dependable logging pipeline is not built from retries alone. It requires durable buffering, explicit delivery acknowledgments, bounded resources, and clear failure behavior; organizations should choose retention and overflow policies according to the operational value of their logs.
The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link whose URL suggests an article about scaling configuration delivery to containers, but no article text or technical sections are available to summarize.
Please provide the post’s body or a readable URL extract, and I can summarize it in the requested format.
The provided text does not contain the blog post itself. It mainly includes Datadog’s navigation menu and a promotional link announcing its Gartner Magic Quadrant recognition, so the article’s argument and technical conclusions cannot be reliably summarized.
## Content Present in the Extract
- A promotional banner links to Datadog’s recognition as a **Leader in the Gartner Magic Quadrant for Observability Platforms**.
- The page navigation lists Datadog offerings across:
- Infrastructure and application monitoring
- Logs, databases, and data observability
- Security
- Digital experience monitoring
- Software delivery
- Service management
- AI capabilities
- The URL suggests the intended article is **“Unwinding a Shared Database”**, but its body text is missing.
## Practical Conclusion
Please provide the article’s actual text or a complete page extract for a meaningful technical summary.
Datadog’s seemingly simple tenant-configuration CRUD system must propagate updates rapidly and reliably to thousands of containers processing millions of logs per second. Loading configuration on every log is too expensive, while periodic caching introduces stale data and delayed updates. Datadog initially used database-backed caches invalidated through Kafka, but growing scale exposed reliability and resilience problems tied to repeated workload access to the central database.
## The Challenge of Propagating Context Data
- Datadog calls tenant-specific settings—such as log parsing rules, Sensitive Data Scanner settings, and storage quotas—“context data.”
- Configuration changes are expected to take effect almost immediately, including in Live Tail.
- The same context data may be consumed by thousands of containers handling traffic for many tenants.
- Because configuration directly affects customer-data processing, propagation must be both low-latency and highly reliable.
- The system must assume that failures can occur anywhere in a large distributed environment.
## Why On-Demand Fetching and Simple Caching Fail
- Fetching configuration from a database for every incoming log would create an impractical read load.
- Large tenants can generate hundreds of thousands of logs per second.
- Each processing instance could require thousands of database reads per second.
- Multiplying this across many instances would require extensive, highly performant database replicas.
- Caching configuration in each workload container reduces reads but does not eliminate the scaling problem.
- Many workload instances still cache data for a high number of tenants.
- Increasing the cache interval reduces database load but delays configuration updates.
- With periodic invalidation, the average propagation delay is roughly half the cache interval.
## Context Loading v1: Database-Backed Caches and Kafka
Datadog’s first successful architecture kept tenant configuration in a central durable database while allowing workload containers to cache entries indefinitely.
- A user changes a log-processing configuration.
- The central context database stores the update.
- Kafka publishes an invalidation message after the database write.
- Every workload container receives the notification.
- Each container reloads the affected tenant’s configuration from the database.
- This minimized routine database reads while preserving low-latency updates.
## Why the Initial Architecture Needed Reconsideration
- The design required every workload instance to reach the central context database whenever a configuration changed.
- As Datadog added more workloads and containers, update-related database traffic grew substantially.
- Internal game days and production incidents showed that problems affecting the context database could spread to downstream processing workloads.
- Database failures could prevent configuration updates from propagating and potentially make it impossible for new workload containers to initialize their context.
- These reliability concerns demonstrated that Kafka-based invalidation alone did not sufficiently isolate workload processing from context-database failures.
Datadog’s experience shows that configuration propagation at large scale requires more than a durable database and cache invalidation. The system must also reduce dependency on the central database during updates and startup, while continuing to provide near-immediate, reliable propagation.
The supplied content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Detecting Faulty Deployments.” As a result, there is not enough information to accurately summarize the article’s arguments, implementation details, or conclusions.
## Available context
- The linked article appears to concern identifying deployments that introduce faults or regressions.
- Datadog’s platform covers related capabilities such as:
- Application Performance Monitoring
- Metrics and infrastructure monitoring
- Logs and error tracking
- CI Visibility and software delivery monitoring
- Service-level objectives and incident response
- The page also promotes Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms.
## Missing information
- The article’s detection methodology
- Metrics, queries, or deployment signals used
- Alerting, rollback, or remediation procedures
- Technical examples and conclusions
Please provide the article text or a page extract containing the post body for an accurate summary.
Datadog rewrote its AWS Lambda extension from Go into Rust to overcome the performance limits of adapting its large, host-oriented Datadog Agent to Lambda’s constrained environment. The redesign reduced cold-start latency by 82%, memory usage by 40%, and binary size from 55 MB to 7 MB. The project succeeded by narrowing the problem, enforcing performance budgets from the beginning, and designing specifically for Lambda’s execution model.
## Why the Original Extension Needed to Change
- The Lambda extension runs as a sidecar process, collecting logs, metrics, traces, profiles, and process data asynchronously.
- It was originally based on the Datadog Agent, which is designed for hosts, containers, and clusters.
- The Agent’s fairness, buffering, caching, and high-throughput features introduced unnecessary overhead in Lambda.
- Optimization attempts included:
- Removing dependencies with build tags
- Compressing binaries with UPX
- Eliminating unnecessary `init` methods
- Exploring Go plugins for lazy loading
- These changes could not reduce additional cold-start latency below roughly 450–500 milliseconds.
## Why a Rewrite—and Why Rust
- Rewrites are risky because they can lose undocumented invariants, reproduce subtle bugs, and create the burden of supporting two systems.
- The team concluded that Lambda represented a fundamentally different scale and workload from the general-purpose Datadog Agent.
- Rust was well suited because:
- Memory safety reduces the risk of crashes and data races.
- Extension crashes also terminate the Lambda function and trigger another cold start.
- Rust produces small binaries with limited runtime overhead.
- Lambda targets a narrow platform set: Amazon Linux on x86 and Arm.
- Compile-time concurrency guarantees support reliable multithreaded code.
- A hackathon prototype demonstrated enough potential to begin the full rewrite, named Project Bottlecap.
## Project Bottlecap’s Design Constraints
- The extension had to minimize interference with the function handler, especially because many Lambda functions serve latency-sensitive APIs.
- Telemetry work should occur after the handler returns whenever possible.
- The team also minimized post-runtime duration—the CPU time added after normal function execution.
- Performance was monitored from the start:
- Dashboards and alerts tracked cold-start overhead.
- Every pull request was benchmarked.
- Regressions were investigated before merging.
- The team accepted targeted tradeoffs for speed, including manually implementing AWS API calls and request signing instead of using SDKs that added too much overhead.
- The design emphasized optionality because Lambda workloads range from small API functions to large asynchronous batch jobs.
- Planned flush strategies included:
- Flushing at the end of an invocation for infrequently called or CPU-constrained functions
- Periodic or in-invocation flushing for workloads needing different latency and resource tradeoffs
The practical lesson is that software optimized for large, long-running systems may be fundamentally unsuitable for serverless runtimes. When optimization reaches a hard performance floor, a focused rewrite—constrained by the target environment and measured continuously—can deliver major gains.
The provided text does not contain the blog post’s main article body. It mainly includes Datadog’s navigation links and a promotional announcement that Gartner named Datadog a Leader in the 2026 Magic Quadrant for Observability Platforms. The linked page URL suggests the intended post concerns Datadog’s AWS Lambda extension and Rust, but no technical details are included.
## Datadog’s Observability Platform Recognition
- Datadog promotes its recognition as a Gartner Magic Quadrant Leader.
- The surrounding navigation highlights products for:
- Infrastructure and cloud monitoring
- Application performance monitoring
- Logs and data observability
- Security
- Digital experience monitoring
- Software delivery
- Service management
- AI and platform capabilities
## Referenced Lambda and Rust Article
- The navigation links to an engineering post titled around the Datadog Lambda Extension and Rust.
- However, the supplied content contains no explanation of:
- Why Rust was selected
- The extension’s architecture
- Performance or resource improvements
- Deployment and compatibility considerations
The article body is needed for a substantive technical summary.
The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link, including a URL suggesting an article about building a Kafka streaming platform with custom abstractions, but no article text or technical sections to summarize.
Please provide the full blog post content for an accurate summary.
The provided content does not include the blog post itself. It contains Datadog’s navigation links and a reference to an article at `/blog/engineering/husky-storage-compaction/`, but no substantive text about Husky or storage compaction.
## Available Information
- The article appears to be an engineering post about **Husky storage compaction**.
- The surrounding site navigation lists Datadog products across infrastructure, applications, data, logs, security, and AI.
- No technical details, arguments, implementation choices, or conclusions from the article are present.
Please provide the article’s body text or a complete excerpt for an accurate summary.
The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link titled “Formal Modeling and Simulation,” but no article text or technical explanation to summarize.
## Available Information
- Datadog promotes its observability platform across:
- Infrastructure and application monitoring
- Logs, databases, and data pipelines
- Security
- Digital experience monitoring
- Software delivery and service management
- AI-powered observability
- The page also advertises Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms.
- The linked engineering article appears to concern formal modeling and simulation, but its subject, methods, and conclusions are not included.
## Conclusion
Please provide the blog post’s full text or the relevant article content to receive an accurate technical summary.
Lengthy CI pipelines and flaky tests often hinder developer productivity by causing unnecessary wait times and costly infrastructure usage. To address this, Datadog developed a Ruby test impact analysis library that dynamically maps tests to specific source files, allowing the CI runner to skip tests unrelated to the latest code changes. By moving beyond standard coverage tools and utilizing low-level Ruby VM interpreter events, this solution significantly reduces testing time while maintaining high performance and correctness.
## The Strategy of Test Impact Analysis
* Lengthy CI pipelines (often exceeding 20 minutes) increase the likelihood of intermittent "flaky" failures that are unrelated to current code changes.
* While parallelization can reduce time, it increases cloud computing costs and does not mitigate the flakiness of irrelevant tests.
* Test impact analysis generates a dynamic map between each test and the source files executed during its run; if a commit doesn't touch those files, the test is safely skipped.
* Success depends on three pillars: correctness (never skipping a necessary test), performance (low overhead), and seamlessness (no required code changes for the user).
## Limitations of Standard Coverage Tools
* Ruby’s built-in `Coverage` module (enhanced in version 3.1 with `resume`/`suspend` methods) proved incompatible with existing total code coverage tools like `simplecov`.
* Initial prototypes using the `Coverage` module showed a performance overhead of 300%, making the test suite four times slower.
* The `TracePoint` API was also evaluated as an alternative to spy on code execution via the `line` event, but it still produced a significant median overhead of 200% to 400%.
* Benchmarks were conducted using the `rubocop` test suite—a "hard mode" scenario with 20,000+ tests—to ensure the tool could handle high-sensitivity environments.
## Implementing a Custom C Extension
* To bypass the limitations of high-level APIs, developers utilized Ruby’s C extension capabilities to hook directly into the Virtual Machine.
* The library uses `rb_add_event_hook2` and `rb_thread_add_event_hook` to subscribe to the `RUBY_EVENT_LINE` event at the interpreter level.
* The implementation involves a C-based `dd_cov_start` function that triggers when a test begins and a `dd_cov_stop` function to collect the results.
* During execution, the tool uses `rb_sourcefile()` to identify the current file and stores it in a Ruby hash only if the file is located within the project’s root directory.
For engineering teams struggling with bloated CI pipelines, adopting test impact analysis is a highly effective way to optimize resources. By utilizing tools like Datadog’s Intelligent Test Runner, which leverages low-level VM events for minimal overhead, teams can cut their testing time in half without sacrificing the reliability of their master branch.
Datadog built a Ruby test impact analysis library to reduce CI time and avoid rerunning unrelated, flaky tests. The approach maps each test to the source files it executes, then runs only tests affected by a commit. Existing Ruby coverage APIs were too slow or incompatible with standard coverage tools, so Datadog developed a faster solution using Ruby VM interpreter events.
## The CI Problem
- Large test suites often take 20 minutes or more and may fail because of unrelated flaky tests.
- Parallel execution reduces runtime but increases cloud costs and does not eliminate flakiness.
- Selective testing can reduce:
- Pipeline duration
- Cloud resource usage
- Exposure to unrelated flaky tests
- Test impact analysis determines which source files each test executes and compares them with files changed in the latest Git commit.
## Requirements for Test Impact Analysis
Datadog’s library needed to provide:
- **Correctness:** Never skip a test that could detect a regression.
- **Performance:** Add minimal overhead because impact data must be collected on every commit and branch.
- **Seamlessness:** Require no user code changes and avoid changing test behavior or interfering with existing tooling.
## Limitations of Ruby Coverage APIs
- Ruby’s built-in `Coverage` module can collect per-test coverage using `resume` and `suspend`, introduced in Ruby 3.1.
- A prototype using Coverage had two major problems:
- It conflicted with tools such as SimpleCov that collect total code coverage.
- It added up to 300% overhead, making the test suite roughly four times slower.
- Datadog then tried Ruby’s `TracePoint` API, subscribing to the `line` VM event.
- TracePoint avoided interference with SimpleCov and provided the required data, but still introduced roughly 200% overhead, reaching 400% in some cases.
## A Custom Coverage Tool Using Ruby VM Events
- Datadog examined Ruby’s internals, including:
- `coverage.c`
- `rb_coverage_resume`
- `rb_resume_coverages`
- `rb_add_event_hook2`
- Ruby’s C extension API supports registering callbacks for `RUBY_EVENT_LINE`, allowing a custom native implementation.
- The proof of concept:
- Registers a line-event hook for the current thread.
- Records the source file for executed lines.
- Ignores files outside the project root.
- Removes the hook when collection stops.
- Returns and resets the collected coverage data for the next test.
- Implementing collection closer to the VM was intended to preserve correctness while substantially reducing the overhead of per-test impact tracking.
Datadog’s experience shows that selective testing is a promising way to make CI faster and more reliable, but practical test impact analysis requires a low-level implementation. Standard coverage and tracing APIs provide useful functionality but can impose unacceptable performance costs, making a purpose-built native VM-event collector a better fit.
Bits AI uses large language models to generate first drafts of incident postmortems from Datadog Incident Management metadata and Slack discussions. The goal is to reduce documentation effort without replacing the human-led reflection that makes postmortems valuable. Datadog found that production-quality LLM features require extensive prompt experimentation, safeguards against hallucinations, careful model selection, and strong privacy and feedback mechanisms.
## Why LLMs Are Difficult to Use for Postmortems
- Postmortems must remain factual, consistent, and useful, unlike more creative LLM applications.
- LLMs can ignore instructions, produce inconsistent formatting, repeat information, or hallucinate plausible but false details.
- More than 100 hours were spent refining section structures and instructions so they worked across varied incidents.
- Building these systems requires a combination of software engineering, product management, data science, and technical writing.
## Balancing Structured and Unstructured Incident Data
- Structured Datadog data includes incident metadata, alerts, graphs, and manually entered fields such as customer impact.
- Slack discussions provide current, detailed, and unstructured context from the incident response process.
- Manually maintained fields are often more reliable but can become outdated, while LLMs can continually reassess live discussions.
- Combining both sources helped the model understand the full incident and prioritize information appropriately.
## Cost, Speed, and Quality Trade-offs
- Datadog compared models including GPT-3.5 and GPT-4.
- Results varied by as much as 50 times in cost and speed.
- GPT-4 generally produced more accurate output but was substantially slower and more expensive.
- Model choice therefore depended on the required balance between quality, latency, and operating cost.
## Preserving Human Control and Privacy
- AI-generated drafts were intended to support authors, not become final documents automatically.
- Keeping engineers involved preserves the discovery and learning process that postmortems are meant to encourage.
- Because incidents may contain sensitive information or secrets, Datadog scrubbed data before sending it to an LLM.
- The system was designed to ensure authors remained responsible for reviewing and improving the draft.
## Experimentation and Reliability Safeguards
- A custom API extracted and structured incident data from Datadog, Slack, and other sources.
- The framework enabled rapid testing of different datasets, models, input configurations, and output token limits.
- Datadog refined prompts and instructions to guide the model toward useful recommendations while retaining author agency.
- Lowering model temperature and tuning other parameters reduced irrelevant or fabricated content.
- A related feature generated concise incident summaries for responders joining Slack channels, providing a faster testing environment for LLM-based incident analysis.
## Evaluating Generated Drafts
- Datadog used both qualitative and quantitative evaluation methods.
- Authors of earlier postmortems compared their human-written documents with AI-generated drafts for the same incidents.
- Evaluation included criteria such as factual accuracy and conciseness, with the provided article ending before the full evaluation approach was described.
- Continuous feedback was used to refine prompts, models, and the overall drafting workflow.
Overall, reliable AI-assisted postmortems require more than selecting a capable model. The strongest approach combines carefully prepared incident data, privacy controls, model and prompt experimentation, systematic evaluation, and mandatory human review.
