Datadog | Techlist.io

datadogNov 1, 20182 min readCurated summary

Improving trust with Datadog Log Management

Datadog handles hundreds of thousands of emails daily and uses Amazon SES for critical messages such as password resets. Because SES and CloudWatch did not provide sufficiently accessible, support-friendly event data, Datadog built a serverless pipeline that forwards SES events to Datadog Log Management. This provides low-maintenance delivery infrastructure, searchable email metrics, and monitoring for failures. ## Exporting Amazon SES Events - SES configuration sets define which email events to capture: - Send - Reject - Bounce - Complaint - Delivery - Open - Click - Events are published to an Amazon SNS topic. - SNS invokes an AWS Lambda function for every event. - The Lambda forwards the event to Datadog Logs using the Datadog API key. - Terraform provisions the SNS topic, SES configuration set, event destination, IAM role, and Lambda function. - The example uses Python 2.7 and stores the API key as a Lambda environment variable; production systems should encrypt the key. - This architecture avoids maintaining a custom email service while preserving visibility into email processing. ## Making SES Events Searchable in Datadog - SES events arrive in Datadog as JSON. - Datadog’s existing AWS integration pipeline processes the logs automatically. - Important fields can be converted into facets directly from a log entry. - Datadog uses fields such as the email event type and subject to quickly search for specific password reset activity. ## Monitoring and Operational Benefits - Support teams can verify whether a recipient received or interacted with a password reset email. - The entire delivery and logging pipeline is serverless and requires minimal maintenance. - Monitors can be configured on the logs to alert normal escalation channels when any part of the pipeline fails. - The solution combines the reliability of Amazon SES with Datadog’s observability and search capabilities. Overall, routing SES events through SNS and Lambda into Datadog Log Management is a practical way to create a trusted, observable password-reset email system without operating a separate mail infrastructure.

Read original

Database Design Datadog Log Management AWS+4

datadogNov 1, 20181 min readCurated summary

Improving trust with Datadog Log Management | Datadog

The provided content does not include the blog post’s main article text. It contains Datadog navigation links and a promotional banner announcing Datadog as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, plus a link suggesting the article concerns improving trust with Datadog Log Management. ## Datadog’s Observability Platform - Datadog promotes a broad observability platform covering: - Infrastructure and cloud monitoring - Application performance monitoring - Logs and sensitive-data protection - Security monitoring - Digital experience monitoring - Software delivery and service management - AI-powered investigation and automation - The banner highlights Datadog’s recognition as a Gartner Magic Quadrant Leader. ## Log Management and Trust - The linked article appears to focus on improving trust through Datadog Log Management. - The supplied text does not provide details about the specific problems, technologies, or recommendations discussed in the post. A complete summary requires the article’s actual body text rather than the surrounding website navigation.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogAug 13, 20182 min readCurated summary

Introducing Kafka-Kit: Tools for scaling Kafka | Datadog

Datadog’s “Kafka Kit” is a collection of operational tools designed to make Apache Kafka easier to scale and manage. The post argues that Kafka’s built-in administrative mechanisms become difficult to use safely as clusters grow, particularly when rebalancing partitions or adding and removing brokers. Kafka Kit automates these workflows while emphasizing balanced assignments, controlled changes, and operational visibility. ## Why Kafka Scaling Becomes Difficult - Growing Kafka clusters require frequent partition movement and broker rebalancing. - Native Kafka reassignment workflows can involve large, complex JSON configurations. - Poorly planned changes can create: - Uneven storage and traffic distribution - Excessive network and disk I/O - Overloaded brokers - Extended recovery times - Operational changes must account for replication, leadership, broker capacity, and rack or availability-zone placement. ## Kafka Kit’s Approach - Kafka Kit provides reusable tooling for common Kafka administration tasks. - The tools generate and apply partition assignments instead of requiring operators to construct them manually. - Assignments can be optimized for more even distribution of: - Partitions - Replicas - Leaders - Storage and traffic - The tooling is intended to support both routine balancing and larger cluster changes, such as adding or decommissioning brokers. ## Safer Partition Reassignment - Reassignments can be performed incrementally rather than moving all partitions at once. - Changes can be throttled to limit their effect on production workloads. - Operators can inspect proposed assignments before applying them. - Controlled movement reduces the risk of saturating Kafka brokers, disks, or network links. - The approach makes long-running migrations easier to monitor and interrupt if necessary. ## Operating Kafka at Scale - Datadog built the tools from its experience running Kafka as a critical part of its data infrastructure. - At large scale, Kafka administration needs to be repeatable and automatable rather than dependent on manual intervention. - Separating planning from execution allows teams to validate capacity and placement before changing the cluster. - Standardized tooling also helps reduce the chance of configuration errors during high-risk maintenance operations. Kafka Kit is most useful for teams operating Kafka clusters large enough that manual partition management is unreliable or disruptive. Automating assignment generation, throttling, validation, and broker lifecycle changes can make scaling more predictable and safer.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

datadogAug 13, 20183 min readCurated summary

Introducing Kafka-Kit: Tools for scaling Kafka

Datadog operates Kafka at extreme scale, ingesting trillions of data points daily and requiring petabytes of NVMe storage. To manage frequent data movement caused by scaling, recovery, and capacity changes, the company built Kafka-Kit, a set of operational tools that improve partition placement and replication control. Its primary tools, `topicmappr` and `autothrottle`, automate safer and more predictable Kafka operations. ## Kafka-Kit - Kafka-Kit addresses two major operational areas: - Data placement across brokers - Replication auto-throttling - Its main tools are: - `topicmappr`, for generating partition-to-broker mappings - `autothrottle`, for automatically controlling replication bandwidth ## Partition Placement with `topicmappr` `topicmappr` replaces Kafka’s `kafka-reassign-partitions.sh --generate` functionality while adding operational safeguards and placement controls. - Produces deterministic output: identical inputs generate the same partition map. - Supports minimal-movement broker replacement: - Failed brokers can be replaced without unnecessarily moving healthy partitions. - Partitions with complete in-sync replicas are normally left untouched. - Provides rack-aware placement using Kafka’s `broker.rack` metadata and ZooKeeper. - Supports placement based on: - Partition count - Storage size, enabling bin-packing and storage rebalancing - Allows replication factors to be increased or decreased while topics are running. - Generates clear summaries of: - Brokers being removed or added - Partition-level changes - Broker distribution before and after reassignment - Warnings and resulting partition-map files The tool is written in Go and can run from any system with access to Kafka’s ZooKeeper cluster. It requires topic names and broker IDs, then verifies that the brokers are live, sufficiently numerous, and properly distributed across configured localities. ## Replacing Failed Brokers For a failed broker, `topicmappr` can rebuild affected topics while limiting movement to the necessary partitions. - Existing replicas are preserved whenever possible. - Replacement brokers fill the gaps left by failed brokers. - The generated report makes the proposed changes visible before execution. - The example replaces broker `1002` with brokers `1003` and `1004`, showing the updated replica assignments and broker totals. ## Placement Strategies `topicmappr` offers multiple strategies for deciding where replicas should live, including `count` and tunable `storage` placement. ### Count Placement Strategy - The default strategy. - Balances leadership and the number of partitions held by each broker. - Works well when traffic is expected to be distributed evenly across partitions. - Does not require metrics data, allowing maps to be generated quickly. - Also attempts to maximize the number of distinct broker-to-broker replica relationships. - This avoids concentrating a broker’s partitions with the same small subset of peers, improving distribution across the cluster and its racks. ## Storage-Aware Placement - The storage strategy uses partition size when assigning replicas. - It supports storage bin-packing and rebalancing, which is important when brokers have uneven disk utilization. - This is particularly useful for Datadog’s large Kafka clusters, where storage capacity—not just partition count—can determine when data must be moved. Datadog’s approach demonstrates that Kafka’s flexible primitives can be extended with purpose-built tooling. For large deployments, deterministic assignments, rack awareness, minimal movement, and storage-based balancing can make scaling and failure recovery substantially safer and more predictable.

Read original

Database Design Apache Kafka Zookeeper Kafka Kit+4

datadogApr 24, 20183 min readCurated summary

Using Datadog APM to improve the performance of Homebrew

Andrew Robert McBurney describes using Datadog APM to diagnose and optimize Homebrew’s slow `brew linkage` command. Instrumentation identified `LinkageChecker#check_dylibs` as the main bottleneck, and replacing repeated dynamic-library processing with persistent caching reduced execution time from 11.5 seconds to 182 milliseconds for 106 packages. A later implementation used Ruby’s built-in PStore instead of SQLite3 to avoid an additional gem dependency. ## Finding the Bottleneck with APM - Homebrew is widely used at Datadog, so improving its performance provides broad benefits. - The `brew linkage` command checks the library links of installed formulas and can identify when a reinstall is needed. - The target was to scan roughly 50 packages, including large packages such as Boost, in under five seconds. - The author instrumented Homebrew with Datadog’s Ruby `ddtrace` gem. - Flame graphs showed that most execution time was spent in `LinkageChecker#check_dylibs`. ## Why Multithreading Was Not Effective - The author tested Ruby threads as a way to process libraries concurrently. - Ruby’s Global Interpreter Lock limited the achievable parallelism. - Threading failed to meet the required performance target, so a different approach was needed. ## SQLite3-Based Caching - The expensive library-processing results were stored in an on-disk SQLite database. - A `linkage` table recorded: - Formula names and library paths - Linkage categories such as `system_dylibs`, `broken_dylibs`, `undeclared_deps`, and `brewed_dylibs` - Optional labels for selected linkage types - A uniqueness constraint on `(name, path, type, label)` prevented duplicate cache entries. - Homebrew could insert and retrieve linkage data using SQL queries. ## Performance Improvements - Without caching, processing 106 packages took 11.5 seconds. - Boost alone required about 1.01 seconds for dynamic-library checks. - With caching enabled: - The full command completed in 182 milliseconds. - Boost’s check took approximately 1.38 milliseconds. - The cached implementation significantly exceeded the original five-second performance requirement. ## Moving to PStore - After submitting the SQLite3 implementation for review, Homebrew maintainers recommended Ruby’s PStore. - PStore provides file-based persistence built around Ruby’s `Hash` data structure. - Its main advantage is avoiding a third-party SQLite3 gem dependency while preserving the benefits of caching. The central lesson is that profiling should guide optimization: rather than adding ineffective threading, the author located the true bottleneck and achieved a dramatic speedup through persistent caching. For similar command-line performance problems, instrument the complete execution path first, then choose the simplest cache or storage mechanism that satisfies both speed and dependency constraints.

Read original

Datadog Application Performance Monitoring Performance Optimization Caching+4

datadogApr 24, 20181 min readCurated summary

Using Datadog APM to improve the performance of Homebrew | Datadog

Datadog announces that Gartner named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The provided content is primarily the website’s navigation menu and does not include the blog article’s substantive discussion, methodology, or supporting evidence. ## Gartner Recognition - Datadog highlights its position as a Leader in Gartner’s Magic Quadrant for Observability Platforms. - The linked resource appears to support or explain the recognition. ## Datadog’s Observability Portfolio The navigation indicates that Datadog’s platform spans: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, profiling, dynamic instrumentation, and service monitoring. - **Logs and data:** log management, database monitoring, data-stream monitoring, and observability pipelines. - **Security:** cloud security, SIEM, vulnerability management, workload protection, and code security. - **Digital experience:** real-user monitoring, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery and service management:** CI visibility, testing, incident response, SLOs, workflow automation, and internal developer portals. - **AI capabilities:** agent observability, AI integrations, investigation agents, and GPU monitoring. The supplied excerpt does not provide enough article text to summarize Datadog’s specific strengths, Gartner’s evaluation criteria, or the company’s evidence for being named a Leader.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

datadogApr 16, 20181 min readCurated summary

Cgo and Python | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link to an engineering article titled “CGO and Python,” but no article text from which to produce a reliable summary. Please provide the post’s body or a readable URL extract, and I can summarize it in the requested format.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogApr 16, 20182 min readCurated summary

Cgo and Python

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

Read original

Database Design Python Go Datadog Agent+4

datadogMar 23, 20182 min readCurated summary

What product designers can learn from explanatory journalism

Explanatory journalism offers product designers a useful model: prioritize important context over the latest information, prevent context from being lost, and explain decisions clearly to different audiences. The author argues that thorough research and documentation make design communication more concise and persuasive. A well-maintained “papertrail” becomes the foundation for better decisions and explanations. ## Favor the Important Over the New - Constantly updating feeds encourage people to value recent information over significant information. - Designers face the same bias when reacting to the latest: - Support ticket - Meeting summary - Customer call - Feedback should be collected in one place and labeled by source so it can be weighted appropriately. - Tracking how many customers or teams request something helps identify common needs and prioritize them. ## Prevent Context Collapse - Sharing tools make it easy to distribute mockups and updates, but recipients may lack the background needed to evaluate them properly. - Product decisions often draw on many sources: - Customer support conversations - Product managers’ customer meetings - Sales requirements - Design research interviews - Executive goals - Designers should assemble this context over time and preserve it in one accessible location. - During reviews, the designer must explain not only the proposed solution but also how it responds to—or intentionally excludes—the concerns of each audience. ## Use the Available Space - Digital publishing and podcasts are not constrained by the same space limitations as traditional media, making background and context easier to provide. - The author describes design work as two phases: - **Expansion:** Gather research, precedent, data, requirements, and evidence. - **Contraction:** Distill that material into concise explanations and decisions. - A thorough, organized research record makes short summaries, presentations, and conversations more accurate and effective. - Links from concise explanations to the underlying research preserve context without overwhelming the main narrative. ## Build and Maintain a Design Papertrail - Documentation benefits the designer even when nobody else reads every detail. - A papertrail should be: - Created during the research process - Kept up to date - Revisited regularly - Used as the canonical source for design decisions - The work focuses primarily on product usability, workflows, and interaction design, while product managers may focus more on pricing, prioritization, timelines, and business considerations. The practical recommendation is to document broadly, organize the evidence, and explain selectively. Designers who maintain strong context will make better decisions and communicate them more effectively.

Read original

User Experience Product Design Interaction Design Social Media+4

datadogMar 23, 2018Original article

What product designers can learn from explanatory journalism | Datadog (opens in new tab)

Product designers can significantly improve their impact by adopting the techniques of explanatory journalism, which prioritizes deep context over the constant noise of new information. By shifting the focus from simply presenting features to explaining the "why" and "how" behind them, designers can better navigate the complex needs of various stakeholders. This approach fosters more rigorous decision-making and ensures that product solutions are grounded in a comprehensive understanding of the problem space. ### Prioritizing Impact Over Recency * Designers often face a "newness bias" where the latest support ticket or customer call carries disproportionate weight compared to long-term goals. * To counteract this, designers should aggregate feedback from diverse sources—such as high-value customers and recurring requests—to identify and prioritize what is truly important rather than what is merely recent. * Effective prioritization requires a centralized system to track the frequency and source of feedback, allowing for a more objective weighting of product requirements. ### Mitigating Context Collapse * In a large organization, "context collapse" occurs when information is shared across different teams (Sales, Support, Research, Executives) without accounting for their unique perspectives or goals. * A designer's role involves assembling disparate pieces of data—including interview notes, sales requirements, and executive goals—into a single, cohesive narrative. * Beyond just presenting work, designers must frame their solutions specifically for each audience, explaining how the design addresses their specific context or why certain requests were triaged out. ### Leveraging the Unlimited Design Papertrail * The design process should cycle through "expansion," where research and data are gathered without space constraints, and "contraction," where that information is distilled into actionable insights. * Developing a thorough "papertrail" of documentation helps the designer master the subject matter, making their eventual summaries more concise and authoritative. * This documentation should include organized interview notes—categorized by job role and company size—and competitive research to serve as a permanent "canon" for all design decisions. To produce more effective work, designers should embrace the role of an "explainer" by meticulously documenting their research and expansion phases. Building a robust, updated papertrail not only clarifies the designer's own thinking but also provides the necessary evidence to defend usability and interaction design choices in a fast-moving product environment.

Database Design Product Design Usability Testing Design Research+4

datadogSep 20, 20173 min readCurated summary

Secure (and usable) multi-AWS account IAM setup

Managing multiple AWS accounts increases billing, compliance, and operational complexity, but it provides valuable security boundaries between workloads. The post proposes a centralized IAM design in which users exist in only one account, receive minimal default permissions, and temporarily assume narrowly scoped roles in target accounts. MFA and short-lived credentials further limit the impact of compromised credentials, although the approach is constrained to roughly ten accounts by IAM group limitations. ## Why Use Multiple AWS Accounts? - Accounts isolate workloads at several levels: - **Network:** VPCs are separate unless explicitly peered. - **API access:** A compromised account cannot access others without delegated roles. - **Compute and billing:** Per-account spending limits, billing alerts, and CloudTrail monitoring can contain or reveal abuse such as cryptocurrency mining. - The main challenge is gaining these security benefits without creating unmanageable administrative overhead. - The proposed pattern is better suited to production environments than development setups where every developer may have a separate account. - Current IAM group limitations restrict the design to approximately ten accounts. ## Centralized IAM User Management - Each IAM user should exist in only one account. - Centralizing users makes onboarding, offboarding, password policies, and auditing easier. - Other accounts can be monitored for unexpected IAM user creation, which should generally never occur. - Users receive only the permissions needed to manage their own authentication settings. ## Why Use IAM Instead of SSO? - SSO would centralize authentication through an existing identity provider. - However, the post argues that SSO can limit the ability to apply MFA protection precisely to privileged operations. - Since MFA is central to the proposed privilege-escalation model, standalone IAM users are retained despite their administrative drawbacks. ## Minimal Default Permissions By default, users should be able to: - View their own IAM record. - Change their console password. - Create, update, or delete their own API credentials. - Create their own MFA device. They should have no meaningful access to other AWS APIs. This limits the blast radius of stolen usernames, passwords, or API keys: without the user’s MFA device, an attacker can generally do little more than inspect the compromised user’s record. ## Temporary Privilege Through Role Assumption - Users perform privileged work by calling `sts:AssumeRole` in the target account. - Assuming a role provides temporary credentials and a session token. - Credentials have a short lifetime, normally no more than one hour by default. - This resembles the Unix `sudo` model: users begin with restricted permissions and explicitly elevate privileges when necessary. ## MFA-Protected Privilege Escalation A role assumption should succeed only when both conditions are met: - The user has an active MFA device and can authenticate with it. - The user belongs to a group authorized to assume the requested role. MFA does not eliminate compromise risk, but it makes stolen API credentials substantially less useful and limits the damage they can cause without the corresponding MFA device. ## Privileges Grouped by Operational Topic - Administrative capabilities are divided into roles based on responsibility rather than assigning broad account-wide permissions. - Example roles include: - A **VPC role** for network configuration. - An **EC2 role** for instance management. - An **S3 role** for storage administration. - Users can then be mapped to groups and roles according to their job responsibilities, making permissions easier to understand and maintain. ## Practical Recommendation Use multiple AWS accounts as deliberate security boundaries, but centralize IAM users, keep their baseline permissions minimal, require MFA for role assumption, and grant temporary, topic-specific roles. This provides strong isolation while keeping privilege management manageable, particularly for production environments with a limited number of accounts.

Read original

AWS Api Security Iam Mfa+4

datadogSep 20, 20173 min readCurated summary

Secure (and usable) multi-AWS account IAM setup | Datadog

The post presents a defense-in-depth approach to securing AWS accounts with IAM. Its central recommendation is to minimize long-lived credentials and broad permissions by combining strong authentication, role-based access, least privilege, and continuous auditing. Secure account governance is treated as an ongoing operational process rather than a one-time configuration. ## Protect the Root User - Use the root user only for tasks that cannot be performed through IAM. - Enable multi-factor authentication (MFA), preferably with a hardware security key. - Avoid creating root access keys. - Store root credentials securely and monitor for any root-user activity. ## Use Federated, Role-Based Access - Prefer AWS IAM Identity Center or an external identity provider for human access. - Grant users access through groups and roles instead of individual permissions. - Use short-lived role credentials rather than permanent IAM user access keys. - Require separate roles for administrative, development, production, and read-only work. ## Apply Least Privilege - Start with narrowly scoped permissions and expand them only when necessary. - Restrict actions by resource, account, region, and relevant condition keys. - Avoid wildcard permissions such as `Action: "*"` and `Resource: "*"`. - Use IAM Access Analyzer and CloudTrail activity to identify unused or excessive permissions. - Add permission boundaries or organization-level Service Control Policies when teams need guardrails around delegated administration. ## Secure Workloads and Automation - Assign IAM roles directly to EC2 instances, Lambda functions, containers, and other workloads. - Do not embed access keys in source code, configuration files, or deployment artifacts. - Store unavoidable secrets in services such as AWS Secrets Manager or Systems Manager Parameter Store. - Rotate and revoke credentials promptly when they are exposed or no longer required. ## Monitor and Audit IAM - Enable CloudTrail across accounts and regions, with logs protected from modification. - Alert on suspicious activity, including root-user use, policy changes, disabled logging, and unusual access-key behavior. - Regularly review users, groups, roles, policies, and unused credentials. - Use AWS Config, Security Hub, or equivalent controls to check compliance with account-security requirements. ## Centralize Governance - Manage multiple AWS accounts through AWS Organizations. - Keep production and sensitive workloads isolated from development accounts. - Apply Service Control Policies to prevent high-risk actions, even for administrators. - Establish a controlled emergency or “break-glass” access process with strong monitoring. The practical recommendation is to combine MFA, centralized identity, temporary role credentials, narrowly scoped permissions, and continuous auditing. No individual IAM setting is sufficient on its own; security comes from layering preventive controls with detection and response.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+2

datadogSep 6, 20171 min readCurated summary

Robust statistical distances for machine learning | Datadog

The supplied text does not include the blog post itself; it is largely Datadog’s navigation menu. The only identifiable article is **“Robust Statistical Distances for Machine Learning,”** so a detailed, source-grounded summary is not possible without the article body. ## Article Focus - The post appears to address statistical distances used to compare probability distributions in machine-learning systems. - Its focus is likely making these comparisons more **robust to outliers, noisy observations, and distribution shifts**. - Such distances can support tasks including anomaly detection, model monitoring, data-drift detection, and evaluating generated data. ## Why Robustness Matters - Conventional distance measures may be disproportionately influenced by extreme values. - Outliers can make two otherwise similar datasets appear substantially different. - A robust distance should distinguish meaningful distribution changes from isolated or corrupted observations. ## Practical Implication The article’s central recommendation is presumably to choose statistical-distance methods based not only on mathematical properties, but also on their resistance to noise and outliers. Please provide the actual article text for a complete, section-by-section summary with the specific techniques and conclusions.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

datadogSep 6, 20173 min readCurated summary

Robust statistical distances for machine learning

Statistical distances provide quantitative ways to measure how similar or different data distributions are, complementing visual tools such as histograms and Q-Q plots. The post compares the Kolmogorov-Smirnov, Earth Mover’s, and Cramér-von Mises distances, showing that each responds differently to local changes, long tails, and shifts in distribution. No single metric is universally best; the appropriate choice depends on which differences matter most. ## Visual Inspection and Q-Q Plots - Histograms offer a quick comparison of: - Minimum and maximum values - Center or average - Spread and overall shape - Q-Q plots sort both datasets and plot corresponding values against one another. - Points close to the first bisector indicate that the datasets likely come from similarly shaped distributions. - Visual methods are useful heuristics but do not provide a precise quantitative distance. ## Kolmogorov-Smirnov Distance - The KS distance compares empirical cumulative distribution functions (CDFs). - It is the largest absolute difference between the two CDFs at any point. - It is a true metric: - It is nonnegative. - It is zero only for identical distributions. - It is symmetric. - It satisfies the triangle inequality. - The distance is bounded between 0 and 1. - This makes it convenient for determining whether distributions are similar, but less useful for measuring how far apart very different distributions are. - For normal distributions with equal standard deviations and increasingly separated means, KS quickly levels off rather than growing proportionally. ## Earth Mover’s Distance - Earth Mover’s Distance (EMD), or the first Wasserstein distance, measures the minimum work required to transform one distribution into another. - “Work” is the amount of mass moved multiplied by the distance it travels. - EMD is equivalent to the area between the two empirical CDFs. - It is particularly useful for distributions with long or significant tails because distant mass contributes substantially to the result. - Unlike KS, EMD is unbounded and can grow with the physical separation between distributions. ## Cramér-von Mises Distance - The Cramér-von Mises (CM) distance sums the squared differences between empirical CDFs and takes the square root. - Its relationship to EMD resembles the relationship between L2 and L1 norms. - CM is less sensitive to isolated local changes than KS but more sensitive than EMD. - For normal distributions with separated means: - EMD grows linearly with the separation. - KS rapidly reaches a plateau. - CM grows approximately like the square root of the separation. ## Sensitivity to Distribution Changes - KS focuses on the maximum CDF difference, making it highly sensitive to local deformations. - EMD averages absolute CDF differences, so localized changes may have little effect. - CM provides a compromise, detecting local changes without reacting as strongly as KS. - In examples involving shifted or localized probability mass: - KS can increase dramatically from a small local deformation. - EMD may barely change. - CM usually shows a moderate increase. - Conversely, changes in distant high-percentile regions may have little effect on KS while producing much larger changes in EMD and CM. Choose the distance according to the type of distributional difference you need to detect: KS for maximum local deviation, EMD for overall displacement and long tails, and CM for a balanced sensitivity to both global and local changes.

Read original

Data Visualization Anomaly Detection Statistical Distances Kolmogorov Smirnov+3

datadogAug 23, 20171 min readCurated summary

Scaling support with Vagrant and Terraform | Datadog

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional banner announcing its Gartner recognition, while the linked page suggests an article about scaling support with Vagrant and Terraform. ### Visible Content - Datadog announces that it was named a **Leader in the Gartner Magic Quadrant for Observability Platforms**. - The page navigation lists products across: - Infrastructure and application monitoring - Logs, security, and digital experience - CI/CD, service management, and AI - The URL references an engineering post titled **“Scaling Support With Vagrant and Terraform,”** but no article text is present. A meaningful technical summary requires the article body or a complete extract of the post.

Read original

Kubernetes Datadog Observability Infrastructure Monitoring+2

Filter by tag