techlist.io
KREN

log-management

57 posts

datadog

How we built a real-world evaluation platform for autonomous SRE agents at scale (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog navigation links and a page title announcing that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no substantive discussion of the evaluation platform or its conclusions. ## Available Information - Datadog’s page promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI capabilities, including Bits AI Agents and Bits Investigation - The referenced URL path suggests the intended article may concern Datadog’s “Bits AI eval platform,” but the article text is not included. ## Conclusion Please provide the full blog post content for a meaningful section-by-section summary.

log-managementaik8sobservability+4
datadog

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The surrounding product catalog presents Datadog as a broad platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, service management, and AI. However, the provided content does not include the blog post’s detailed analysis or Gartner’s specific evaluation criteria. ## Gartner Recognition - Datadog highlights its position as a **Leader** in the **Gartner Magic Quadrant for Observability Platforms 2026**. - The announcement links to a Gartner resource but provides no further details about the ranking, strengths, or limitations. ## Broad Observability Platform - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud cost, storage, GPU monitoring, and Cloudcraft. - **Applications and data:** Application Performance Monitoring, service monitoring, profiling, dynamic instrumentation, database monitoring, data streams, data quality, and jobs monitoring. - **Logs and security:** Log management, sensitive-data scanning, audit trails, observability pipelines, cloud security, SIEM, code security, vulnerability management, and workload protection. - **Digital experience:** Browser and mobile RUM, product analytics, session replay, synthetic monitoring, mobile testing, error tracking, and experiments. - **Software delivery and service management:** CI visibility, test optimization, continuous testing, feature flags, code coverage, event management, SLOs, incident response, workflow automation, and service catalogs. - **AI capabilities:** Agent observability, GPU monitoring, AI integrations, AI agents, investigation tools, security analysis, MCP Server, and agent-building features. Datadog’s positioning is based on consolidating telemetry, security, developer, operations, and AI capabilities into one observability platform. To assess the Gartner recognition fully, readers would need the linked report or the complete article, which is not included here.

log-managementk8sdatadogobservability+2
datadog

Designing MCP tools for agents: Lessons from building Datadog's MCP server | Datadog (opens in new tab)

Datadog is presented as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The provided content, however, consists almost entirely of Datadog’s website navigation rather than the blog post itself, so it does not include Gartner’s evaluation criteria, Datadog’s strengths, or any supporting analysis. ## Gartner Recognition - The page headline announces Datadog’s “Leader” position in the Gartner Magic Quadrant for Observability Platforms. - A link is provided to a Gartner-related resource page. - No ranking details, competitor comparisons, or Gartner commentary are included in the supplied text. ## Datadog’s Product Coverage The navigation indicates that Datadog offers a broad observability and operations platform spanning: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, data-quality, job, log, and sensitive-data monitoring. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection. - **Software delivery and service management:** CI visibility, testing, developer portals, incident response, SLOs, workflows, and case management. - **AI:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and an MCP server. ## Limitations of the Provided Content - The actual article body is absent. - The text does not explain why Gartner recognized Datadog as a Leader. - It provides no technical findings, customer examples, methodology, or conclusions beyond the headline. The supplied excerpt supports only the conclusion that Datadog announced Gartner recognition and positions itself as a comprehensive observability platform. A substantive summary would require the full article text.

log-managementk8sai-agentdatadog+3
datadog

How we reduced the size of our Agent Go binaries by up to 77% | Datadog (opens in new tab)

The supplied text does not include the tech blog post itself. It contains Datadog navigation links and a promotional banner announcing its recognition as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms, but no article body or technical sections. ## Available content - Datadog promotes observability products covering: - Infrastructure and Kubernetes monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - Software delivery and CI visibility - Service management - AI-powered investigation and monitoring - The page links to an engineering article at: - `/blog/engineering/agent-go-binaries/` - No technical explanation, examples, conclusions, or section content from that article is included. Please provide the blog post’s full text or relevant excerpt for a substantive summary.

log-managementk8sdatadogobservability+3
datadog

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog (opens in new tab)

The provided text does not include the blog post’s article body. It contains Datadog’s navigation menu and a link to an engineering post titled around “eBPF workload protection lessons,” so there is not enough source material to accurately summarize its technical arguments or conclusions. ## Available information - The page is hosted by Datadog’s engineering blog. - The linked topic concerns workload protection built with eBPF. - Datadog’s broader product areas include infrastructure monitoring, application performance monitoring, security, logs, and AI. - The excerpt itself does not describe: - The eBPF implementation - Design challenges or trade-offs - Performance considerations - Security detection methods - Lessons learned or recommendations Please provide the article text or a fuller extract for a substantive summary.

log-managementk8sdatadogobservability+3
aws

Amazon CloudWatch introduces unified data management and analytics for operations, security, and compliance (opens in new tab)

Amazon CloudWatch has evolved into a unified platform for managing operational, security, and compliance log data, significantly reducing the need for redundant data stores and complex ETL pipelines. By standardizing ingestion through industry-standard formats like OCSF and OpenTelemetry, the service enables seamless cross-source analytics while lowering operational overhead and storage costs. This update allows organizations to move away from fragmented data silos toward a centralized, Iceberg-compatible architecture for deeper technical and business insights. **Data Ingestion and Schema Normalization** * Automatically collects AWS-vended logs across accounts and regions via AWS Organizations, including CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs. * Includes pre-built connectors for a wide range of third-party sources, such as endpoint security (CrowdStrike, SentinelOne), identity providers (Okta, Entra ID), and network security (Zscaler, Palo Alto Networks). * Utilizes managed Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry (OTel) conversion to ensure data consistency across disparate sources. * Provides built-in processors, such as Grok for custom parsing and field-level operations, to transform and manipulate strings during the ingestion phase. **Unified Architecture and Cost Optimization** * Consolidates log management into a single service with built-in governance, eliminating the need to store and maintain duplicate copies of data across different tools. * Introduces Apache Iceberg-compatible access via Amazon S3 Tables, allowing data to be queried in place by external tools. * Removes the requirement for complex ETL pipelines by providing a unified data store that is accessible to Amazon Athena, Amazon SageMaker Unified Studio, and other Iceberg-compatible analytics engines. **Advanced Analytics and Discovery Tools** * Supports multiple query interfaces, allowing users to interact with logs using natural language, SQL, LogsQL, or PPL (Piped Processing Language). * The new "Facets" interface enables intuitive filtering by application, account, region, and log type, featuring intelligent parameter inference for cross-account queries. * Enables the correlation of operational logs with business data from third-party tools like ServiceNow CMDB or GitHub to provide a more comprehensive view of organizational health. Organizations should leverage these unified management features to consolidate their security and operational monitoring into a single source of truth. By adopting OCSF normalization and the new S3 Tables integration, teams can reduce the technical debt associated with managing multiple log silos while improving their ability to run cross-functional analytics.

log-managementamazon-s3apache-icebergdata-analytics+4
datadog

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog (opens in new tab)

The supplied content does not include the blog post’s article text. It contains Datadog’s navigation menu and a promotional link to its Gartner recognition, while the URL suggests the post concerns CDC replication and search. ## Available Information - Datadog was named a **Leader in the 2026 Gartner Magic Quadrant for Observability Platforms**. - The page promotes Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and digital experience - Software delivery and service management - AI-powered observability - The referenced article URL is `engineering/cdc-replication-search`, indicating a likely focus on **change data capture (CDC), data replication, and search systems**. ## Missing Article Details - No sections, technical explanations, architecture diagrams, implementation details, or conclusions from the blog post are present in the supplied text. - A reliable summary of the CDC replication approach cannot be produced without the article body. Please provide the full post text or its main sections for a complete summary.

log-managementk8sdatadogobservability+2
datadog

Detecting malicious pull requests at scale with LLMs | Datadog (opens in new tab)

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

log-managementk8sdatadogobservability+4
datadog

Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, but no substantive discussion of the linked “Husky Query Architecture” article. ## Available Content ### Datadog’s Observability Platform - Datadog promotes products covering: - Infrastructure and container monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI and agent observability - The navigation emphasizes Datadog’s broad, integrated platform approach. ### Gartner Recognition - The page links to Datadog’s announcement that it was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The supplied text does not include the evaluation criteria, cited strengths, limitations, or Gartner’s comparative analysis. No reliable summary of the Husky query architecture can be produced without the article’s body text.

log-managementk8sobservabilityapplication-performance-monitoring+4
datadog

From hand-tuned Go to self-optimizing code: Building BitsEvolve | Datadog (opens in new tab)

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional link announcing its 2026 Gartner Magic Quadrant recognition. As a result, there is not enough article content to produce a reliable technical summary. ### Available Information - Datadog is promoted as a “Leader” in the Gartner Magic Quadrant for Observability Platforms. - The page links to Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Incident and service management - AI and automation - The referenced blog URL appears to be titled **“Self-Optimizing System,”** but its article text is not included. Please provide the blog post’s main content or a complete page extract for an accurate summary.

log-managementk8sdatadogobservability+3
datadog

Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, so there is insufficient technical material to summarize the article. ### Content Included - A link to Datadog’s Gartner announcement. - Navigation categories covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article may concern scaling process or pipeline efficiency, but its body is not present. Please provide the full blog post text for a substantive summary.

log-managementk8sdatadogobservability+4
datadog

How we built a real-time, client-side noise suppression library without server dependencies | Datadog (opens in new tab)

Datadog’s page announces that the company was named a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms. However, the provided content contains only the page header, navigation links, and product categories—not the blog post itself—so its technical argument and supporting details cannot be reliably summarized. ## Available Content - Announcement: - Datadog was named a Leader in the Gartner® Magic Quadrant™ for Observability Platforms. - The page links to a Gartner-related resource. - Product areas listed: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and cloud security - Digital experience monitoring - CI/CD and software delivery - Service management - AI and observability tools - The URL references a “noise suppression library,” but no corresponding article text was included. Please provide the full blog post content for a substantive summary.

log-managementk8sdatadogobservability+3
datadog

How we built reliable log delivery to thousands of unpredictable endpoints | Datadog (opens in new tab)

Datadog’s “Reliable Log Delivery” post explains how log-collection systems can avoid losing data when networks, destinations, or agents fail. Its central recommendation is to combine acknowledgments, buffering, retries, and controlled backpressure to provide at-least-once delivery without allowing outages to overwhelm the collector. ## Why Reliable Delivery Matters - Logs are often needed during incidents, precisely when infrastructure and networks may be unstable. - Temporary destination failures can cause data loss if collectors only keep logs in memory. - Retrying without limits can create duplicate logs, unbounded memory usage, or cascading failures. ## Buffering and Persistence - Collectors should buffer logs while downstream services are unavailable. - In-memory buffers provide speed but cannot survive process crashes or host restarts. - Disk-backed queues improve durability by preserving unsent logs across transient failures. - Storage limits are necessary so a prolonged outage does not fill the host’s disk. ## Acknowledgments and Retries - A log should be removed from the queue only after the destination confirms successful receipt. - Failed or unacknowledged deliveries are retried, allowing temporary network and service failures to recover automatically. - At-least-once delivery is the practical reliability target, meaning duplicates may occur and downstream systems should handle them safely. - Retry policies should use delays and backoff rather than continuously retrying at full speed. ## Backpressure and Operational Trade-offs - When downstream systems slow down, collectors must apply backpressure instead of accepting unlimited data. - Backpressure can limit memory consumption and protect the rest of the host. - Teams must define what happens when buffers reach capacity, such as dropping the oldest data, rejecting new logs, or prioritizing important streams. - Reliability also requires monitoring queue size, delivery latency, retry rates, and dropped records. A dependable logging pipeline is not built from retries alone. It requires durable buffering, explicit delivery acknowledgments, bounded resources, and clear failure behavior; organizations should choose retention and overflow policies according to the operational value of their logs.

log-managementk8sdatadogobservability+4
datadog

How we scaled fast, reliable configuration distribution to thousands of workload containers | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link whose URL suggests an article about scaling configuration delivery to containers, but no article text or technical sections are available to summarize. Please provide the post’s body or a readable URL extract, and I can summarize it in the requested format.

log-managementk8sdatadogobservability+3
datadog

How we scaled fast, reliable configuration distribution to thousands of workload containers (opens in new tab)

Datadog’s seemingly simple tenant-configuration CRUD system must propagate updates rapidly and reliably to thousands of containers processing millions of logs per second. Loading configuration on every log is too expensive, while periodic caching introduces stale data and delayed updates. Datadog initially used database-backed caches invalidated through Kafka, but growing scale exposed reliability and resilience problems tied to repeated workload access to the central database. ## The Challenge of Propagating Context Data - Datadog calls tenant-specific settings—such as log parsing rules, Sensitive Data Scanner settings, and storage quotas—“context data.” - Configuration changes are expected to take effect almost immediately, including in Live Tail. - The same context data may be consumed by thousands of containers handling traffic for many tenants. - Because configuration directly affects customer-data processing, propagation must be both low-latency and highly reliable. - The system must assume that failures can occur anywhere in a large distributed environment. ## Why On-Demand Fetching and Simple Caching Fail - Fetching configuration from a database for every incoming log would create an impractical read load. - Large tenants can generate hundreds of thousands of logs per second. - Each processing instance could require thousands of database reads per second. - Multiplying this across many instances would require extensive, highly performant database replicas. - Caching configuration in each workload container reduces reads but does not eliminate the scaling problem. - Many workload instances still cache data for a high number of tenants. - Increasing the cache interval reduces database load but delays configuration updates. - With periodic invalidation, the average propagation delay is roughly half the cache interval. ## Context Loading v1: Database-Backed Caches and Kafka Datadog’s first successful architecture kept tenant configuration in a central durable database while allowing workload containers to cache entries indefinitely. - A user changes a log-processing configuration. - The central context database stores the update. - Kafka publishes an invalidation message after the database write. - Every workload container receives the notification. - Each container reloads the affected tenant’s configuration from the database. - This minimized routine database reads while preserving low-latency updates. ## Why the Initial Architecture Needed Reconsideration - The design required every workload instance to reach the central context database whenever a configuration changed. - As Datadog added more workloads and containers, update-related database traffic grew substantially. - Internal game days and production incidents showed that problems affecting the context database could spread to downstream processing workloads. - Database failures could prevent configuration updates from propagating and potentially make it impossible for new workload containers to initialize their context. - These reliability concerns demonstrated that Kafka-based invalidation alone did not sufficiently isolate workload processing from context-database failures. Datadog’s experience shows that configuration propagation at large scale requires more than a durable database and cache invalidation. The system must also reduce dependency on the central database during updates and startup, while continuing to provide near-immediate, reliable propagation.

log-managementdatabase-designdatadogobservability+4