datadogMay 12, 2022

How Datadog's IT team automated account inactivity and SaaS spend management (opens in new tab)

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

pythondatabase-designpostgresqlmicroservices+3

datadogOct 15, 2021

Our journey taking Kubernetes state metrics to the next level | Datadog (opens in new tab)

Datadog’s container observability team significantly improved the performance of kube-state-metrics (KSM) by contributing core architectural enhancements to the upstream open-source project. Faced with scalability bottlenecks where metrics collection for large clusters took tens of seconds and generated massive data payloads, they revamped the underlying library to achieve a 15x improvement in processing duration. These contributions allowed for high-granularity monitoring at scale, ensuring that the Datadog Agent can efficiently handle millions of metrics across thousands of Kubernetes nodes. ### Challenges with KSM Scalability * KSM uses the informer pattern to expose cluster-level metadata via the Openmetrics format, but the volume of data grows exponentially with cluster size. * In high-scale environments, a single node generates approximately nine metrics, while a single pod can generate up to 40 metrics. * In clusters with thousands of nodes and tens of thousands of pods, the `/metrics` endpoint produced payloads weighing tens of megabytes. * The time required to crawl these metrics often exceeded 15 seconds, forcing administrators to reduce check frequency and sacrifice real-time data granularity. ### Limitations of Legacy Implementations * KSM v1 relied on a monolithic loop that instantiated a Builder to track resources via stores, but it lacked efficient hooks for metric generation. * The original Python-based Datadog Agent check struggled with the "data dump" approach of KSM, where all metrics were processed at once during query time. * To manage the load, Datadog was forced to split KSM into multiple deployments based on resource types (e.g., separate deployments for pods, nodes, and secondary resources like services or deployments). * This fragmentation made the infrastructure more complex to manage and did not solve the fundamental issue of inefficient metric serialization. ### Architectural Improvements in KSM v2.0 * Datadog collaborated with the upstream community during the development of KSM v2.0 to introduce a more extensible design. * The team focused on improving the Builder and metric generation hooks to prevent the system from dumping the entire dataset at query time. * By moving away from the restrictive v1 library structure, they enabled more efficient reconciliation of metric names and metadata joins. * The resulting 15x performance gain allows the Datadog Agent to reconcile labels and tags—such as joining deployment labels to specific metrics—without the significant latency overhead previously experienced. Contributing back to the open-source community proved more effective than maintaining internal forks for scaling Kubernetes infrastructure. Organizations running high-density clusters should prioritize upgrading to KSM v2.0 and optimizing their agent configurations to leverage these architectural improvements for better observability performance.

pythondatabase-designk8sdatadog+3

datadogOct 15, 2021

Our journey taking Kubernetes state metrics to the next level (opens in new tab)

Datadog contributed major scalability improvements to kube-state-metrics (KSM), after discovering that its metric collection process struggled with very large Kubernetes clusters. Millions of metrics could require tens of megabytes and tens of seconds to process every 15 seconds, forcing Datadog to reduce collection frequency. Their redesign improved collection duration by 15x and enabled more granular monitoring at scale. ## Datadog’s Kubernetes Observability Role - The Datadog Containers team monitors Kubernetes infrastructure and ensures reliable collection of: - Logs - Traces - Custom metrics - Profiles - Security signals - KSM is central to Datadog products such as Kubernetes metrics integration and Orchestrator Explorer. ## How Kubernetes State Metrics Works - KSM uses Kubernetes informers to watch objects registered with the API server. - Enabled collectors monitor resources such as pods, nodes, deployments, and services. - It generates lifecycle and metadata metrics in text-based OpenMetrics format. - Users can restrict monitored resources through the `resources` flag. - The Datadog Agent’s KSM check: - Runs every 15 seconds. - Discovers KSM containers. - Crawls their `/metrics` endpoint. - Reconciles metric metadata and applies configured label joins. - Label joins allow metadata from one metric, such as a deployment label, to become a tag on other metrics for the same object. ## Scaling Challenges - Datadog found that KSM needed to be split across multiple deployments beyond a few hundred nodes and thousands of pods. - Their deployments divided collectors by resource type: - Pods - Nodes - Services, deployments, jobs, persistent volumes, and other resources - Metric volume varied substantially: - Endpoints, jobs, and deployments produced roughly five metrics per object. - Nodes produced around nine metrics each. - Pods produced around 40 metrics each. - Large clusters with thousands of nodes and tens of thousands of pods could generate millions of metrics per scrape. - Crawling the metrics endpoint could take tens of seconds and transfer tens of megabytes. - Datadog had to reduce check frequency, sacrificing metric granularity and user experience. ## KSM’s Original Architecture - KSM v1 relied on a central loop that created a Builder and managed resource stores. - Each store used informers to track a particular Kubernetes resource. - For example, an HPA store maintained the list-and-watch logic for HorizontalPodAutoscalers. - The Builder generated metrics from the tracked resources. - Datadog identified two limitations: - Too much data was emitted and processed at query time. - The Builder did not provide a suitable extension point for custom metric-generation logic. ## Contributing the Redesign Upstream - As the KSM community prepared version 2.0 in early 2020, Datadog saw an opportunity to address its scalability and extensibility problems in the upstream project. - Rather than maintaining a private solution, the team contributed its findings and improvements to the open-source community. - The resulting work reportedly reduced metric collection duration by 15x, making high-scale, more frequent collection practical. Datadog’s experience shows that upstream open-source collaboration can solve internal infrastructure bottlenecks while improving the project for the broader Kubernetes community.

pythondatabase-designk8sdatadog+4

datadogFeb 22, 2021

How we minimized the overhead of Kubernetes in our job system (opens in new tab)

Kubernetes can improve machine management and scalability, but its scheduling and runtime overhead can significantly reduce job throughput if configured poorly. Datadog found its Kubernetes-based job system used more CPU and completed jobs 40–50% more slowly than the previous VM-based system. By designing a controlled experiment, choosing better metrics, and tuning pod resource requests, the team recovered performance to roughly VM parity while investigating the overhead of running one parent process per pod. ## Designing a Comparable Experiment - The initial comparison was difficult because the Kubernetes and VM deployments differed in: - Number of nodes - Number of worker-parent clusters - Workload and enqueue rate - A controlled experiment was created using: - Identical `c5.2xlarge` machines - The same kernel version, `3.13.0-141` - Both systems repeatedly running a simple Python job - Each Kubernetes pod contained one parent process and its worker processes, making pod count equivalent to parent-process count per node. - The older kernel did not include CPU mitigations, avoiding that variable in the comparison. ## Choosing Useful Performance Metrics ### Measuring node effort - Load average initially appeared useful for measuring machine utilization. - Kubernetes background processes—such as cluster polling and pod-state checks—artificially increased load average. - Load average counts runnable processes rather than the amount of CPU time they actually consume. - The team therefore used CPU idle time instead: - It measures unused CPU capacity. - It reflects actual CPU work rather than the number of active processes. ### Measuring system performance - The job system optimized for throughput rather than latency. - Throughput was measured by the number of jobs completed within 30 seconds. - Latency remained useful for detecting queueing problems, but throughput was the primary success metric. ## Tuning Kubernetes Resource Requests - The main performance gains came from improving pod scheduling. - The target was six pods per `c5.2xlarge` node. - Initially, each pod requested: - One full CPU core - More memory than necessary - Since the node had eight cores and approximately 1.5 GiB of memory consumed by Kubernetes and system services, only four pods could be scheduled. - Requests were reduced to: - `100m` CPU, or 100 millicores - `500 MB` memory - CPU tuning generally enabled six pods per node, although some nodes still scheduled only five. - Further memory reduction was needed because system daemons consumed enough memory to prevent six pods from fitting on some nodes. - Resource requests affect scheduling minimums, while limits constrain containers after they start. - These request changes did not slow jobs because the pods still received sufficient resources to operate. ## One Parent Process per Pod - The team considered placing multiple parent processes in each pod to reduce potential pod overhead. - One parent plus its workers was a natural application unit and simplified orchestration. - The decision depended on how much overhead each pod introduced: - High overhead would favor fewer, larger pods. - Low overhead would favor one parent per pod for simpler management. - Using `pstree`, the team identified six job-system instances per node and traced their process trees through components such as: - `containerd-shim` - `tini` - The application process - They estimated that each pod included overhead associated with three containers, particularly `containerd-shim`. - CPU overhead was then investigated using `perf sched`. The practical lesson is to compare equivalent workloads, measure actual CPU consumption rather than relying blindly on load average, and tune Kubernetes requests for the desired packing density. Resource requests should be large enough for reliable operation but not so large that they unnecessarily prevent pods from being scheduled together.

pythondatabase-designk8sresource-management+4

datadogOct 7, 2020

How we wrote a Python profiler (opens in new tab)

Datadog built a Python continuous profiler because Python lacked Java-style, always-on production profiling tools. The post argues that deterministic profilers such as `cProfile` impose too much overhead for continuous use, while statistical profiling can provide representative performance data with minimal disruption. Datadog’s profiler addresses this through modular collectors, recording, scheduling, and data export. ## Profiling Versus Tracing - **Profiling** measures resource consumption such as CPU time and memory allocation to reveal performance problems. - **Tracing** records individual operations—such as SQL queries or HTTP requests—within a request timeline. - Tracing explains request latency, but profiling provides deeper insight into code-level execution and operating-system resource usage. ## Limitations of Deterministic Python Profilers - Python’s `cProfile`, available since CPython 2.5, records every function call and the time spent in each call. - It can provide a complete execution flow, but its usefulness depends heavily on code structure: - A program built around a few large functions produces little actionable detail. - A program containing thousands of functions can incur two- or three-times runtime overhead. - This overhead makes deterministic profiling unsuitable for always-on production environments. ## Why Profile in Production? - Optimizing without profiling is essentially guessing; real workloads often differ from development environments. - Production systems vary from developer machines in hardware, concurrency, input data, and workload behavior. - Continuous profiling captures how an application actually consumes resources under authentic conditions. - These requirements lead to statistical rather than deterministic profiling. ## Statistical Profiling in Python - Statistical profilers sample program activity periodically instead of recording every function call. - Individual short-lived calls may be missed, but repeated sampling over hours produces a reliable picture of resource consumption. - Lower overhead allows the application to run closer to its normal, unprofiled behavior. - Datadog evaluated numerous open-source Python profilers but found limitations involving platform support, collected data, or presentation-focused designs. - The team therefore developed its own statistical profiler, incorporating ideas from the tools it studied. ## Datadog Python Profiler Design The profiler was designed around three constraints: - Keep runtime overhead as low as possible. - Make deployment simple. - Support common operating systems and environments. Its architecture, inspired by the JDK Flight Recorder, consists of: - **Collectors:** Gather data such as CPU usage and memory allocation. - **Recorder:** Stores events produced by collectors. - **Exporter:** Sends profiling data outside the application. - **Scheduler:** Invokes components at appropriate intervals, such as exporting data every 60 seconds. - **Profiler:** Provides the high-level interface used by applications. ## Stack Collection - The stack collector is the primary built-in collector. - It wakes 100 times per second and captures the execution stack of every Python thread. - For each thread, it gathers information including: - The currently executing function - CPU time consumed - Exceptions being handled - The collector monitors the time required to inspect the application so it can control and limit its own CPU overhead. A statistical profiler with low overhead is the appropriate foundation for continuous production profiling, giving teams evidence about real application behavior without substantially changing that behavior.

pythondatadogjavaprofiling+4

datadogJun 3, 2019

Secure publication of Datadog Agent integrations with TUF and in-toto (opens in new tab)

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised. ## Independent Integration Publishing - Agent integrations were traditionally bundled into full Agent releases. - This delayed important integration updates and prevented users from trying new integrations immediately. - Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself. ## End-to-End Verification with in-toto - TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure. - in-toto defines the software supply chain as a fixed sequence of signed steps. - Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work. - The integration pipeline includes: - Developers signing Python and YAML source files. - CI/CD packaging the source into Python wheels without modifying existing wheels. - A signing step applying TUF signatures to the wheels. - The Datadog Agent verifying that the downloaded wheel matches the developer-signed source. ## Secure Distribution with TUF - in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys. - TUF supplies signed, compromise-resilient metadata for: - The root of trust for wheels and supply-chain metadata. - The in-toto-defined workflow. - Public keys used to verify the workflow. - TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata. - Offline trust bootstrapping and protected developer keys are essential to the overall security model. ## Hardware-Protected Developer Signing - Developers use Yubikeys to generate and store GPG signing keys. - Private keys cannot be exported from the device, assuming correct firmware. - Signing requires both a secret PIN and physical interaction with the Yubikey. - A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk. ## Transparent Verification for Users - The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations. - Users need no workflow changes under normal conditions. - If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure. Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.

pythondatabase-designdatadogyaml+4

datadogApr 16, 2018

Cgo and Python (opens in new tab)

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

pythondatabase-designgodatadog-agent+4

datadogJun 15, 2017

Hackathon project: Viewing Datadog metrics in Minecraft (opens in new tab)

Datadog engineers used a two-day hackathon to display real-time Datadog metrics inside Minecraft. They connected Minecraft’s Python API with Datadog’s metrics API, then built configurable, live-updating graphs and monitor indicators in the game world. The project demonstrated that even an unconventional visualization environment can be practical to prototype with familiar tools. ## Controlling Minecraft with Python - The team used Raspberry Juice and a Minecraft Pi Edition server to expose Minecraft controls. - They ran the setup on laptops for better performance and faster development. - The `py3minepi` library enabled Python code to create, remove, and query blocks. - Creating a block required only a connection to the server and a call such as `mc.setBlock(...)`. ## Retrieving Datadog Metrics - The Datadog Python library provided access to the Metrics API. - The prototype authenticated with an API key and application key. - It queried recent data, such as average system CPU idle time over the previous five minutes. - The Minecraft and Datadog components were then combined so metric values could be rendered as blocks and structures. - Monitor status indicators changed between green and red depending on whether an alert was active. ## YAML-Based Dashboard Configuration - The team moved dashboard definitions out of Python code into YAML files. - Configuration specified: - Graph position, size, and orientation - Visual properties such as colors, transparency, and borders - Datadog queries and time ranges - Monitor IDs and display locations - This allowed complete dashboards containing multiple graphs and monitor indicators to be updated in real time. ## Handling Minecraft’s Persistence - Minecraft blocks remain in the world after being created, while metric graphs change constantly. - Early experiments left behind random cubes that made the world difficult to navigate. - The team implemented “vacuum” functions to remove everything generated by the visualization code before redrawing it. ## Rendering and Performance Challenges - Without browser technologies such as JavaScript and CSS, graphs had to be reduced to rows of data and represented with Minecraft blocks. - Large graphs could overwhelm the data pipeline. - Caching was added to reduce bandwidth usage and avoid repeatedly requesting the same data. - The performance concerns mirrored Datadog’s everyday engineering work, where caching and efficient data handling are essential. ## Hackathon Experience - The first four hours focused on configuring the environment and connecting the systems. - The remaining time was spent experimenting with building, viewing, destroying, and rebuilding metric displays. - The project’s main value was creative exploration rather than production monitoring. The prototype shows how quickly APIs can be combined to create unusual monitoring interfaces. While Minecraft is not intended to replace conventional dashboards, the project is a playful demonstration of real-time data visualization and rapid experimentation.

pythondatabase-designdatadogapi+3

datadogJun 8, 2017

Protobuf parsing in Python | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s site navigation and a link whose URL suggests the article concerns Protobuf parsing in Python, but no technical claims, explanations, or conclusions are available to summarize. ## Available Information - The linked article appears to be: - **“Protobuf parsing in Python”** - Located on Datadog’s engineering blog. - The rest of the content is primarily Datadog product navigation, covering: - Infrastructure and application monitoring - Logs, security, digital experience, CI, and AI products - A separate banner promotes Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms. Please provide the article’s body text or a complete page extract for a substantive summary.

pythonserializationprotobufparsing

datadogJun 8, 2017

Protobuf parsing in Python (opens in new tab)

Protocol Buffers provides a compact, efficient binary format for structured data, making it suitable for APIs and inter-machine communication. The post introduces Protobuf through a Python metrics example and explains how to serialize and deserialize messages. It also shows how to stream multiple messages by prefixing each with its length, since Protobuf messages are not inherently self-delimiting. ## Protocol Buffers Basics - A `.proto` file defines the structure of a message. - The example `Metric` message contains: - A name - A type - A floating-point value - Repeated string tags - The `protoc` compiler generates language-specific code, such as Python’s `metric_pb2.py`. - Python can serialize a message with `SerializeToString()` and restore it with `ParseFromString()`. ## Streaming Multiple Messages - A single Protobuf message can be parsed directly, but consecutive messages need delimiters. - Protobuf does not automatically indicate where one message ends and the next begins. - The recommended approach is to prepend each serialized message with its byte length. - The length is encoded as a Varint, which uses fewer bytes for smaller integers. - This mirrors Java’s `writeDelimitedTo` and `parseDelimitedFrom` behavior and is also how the kube-state-metrics API chains messages. ## Varints and Python Implementation - Python’s Protobuf library does not provide public convenience methods for delimited messages. - The implementation uses internal helpers: - `_VarintBytes` to encode message lengths - `_DecodeVarint32` to read them - Serialization writes the length followed by the message bytes. - Deserialization reads the length, extracts the corresponding byte range, and parses it as a `Metric`. - The example loads the entire stream into memory, though a production implementation could process data incrementally. For APIs that exchange sequences of structured records, length-prefixed Protobuf messages offer an efficient and interoperable alternative to plain-text formats. Teams should account for message framing explicitly and use generated code plus appropriate streaming logic when handling multiple messages.

pythonk8sdatadogmetrics+4

datadogJan 23, 2017

The trouble with mounting (opens in new tab)

Datadog found that some agents stopped reporting all metrics because they became stuck in an unkillable state during disk checks. The root cause was `os.statvfs`, whose glibc implementation can hang while inspecting NFS mounts configured with hard-mount behavior. Since agents run in unpredictable customer environments, Datadog isolated the call in a separate thread and allowed the main process to continue after a timeout. ## Detecting the Hang - Customers reported gaps across every metric, indicating that the agent—not an individual check—had stopped functioning. - Logs showed the agent sometimes hung without producing an error. - A watchdog failed to terminate it because the process was stuck in an unkillable system call. - Developer-mode timing data identified `os.statvfs` as the consistently slow operation. ## How NFS Causes Unkillable Processes - `os.statvfs` calls the Linux `statvfs` function through CPython and glibc. - `statvfs` can hang when examining a remote directory mounted through NFS. - NFS hard mounts retry indefinitely and do not time out system calls. - Soft mounts eventually return an error, while the `intr` option allows interruption of the calling process. - Hard mounts may be appropriate when reads and writes must eventually succeed, but they are risky with unreliable NFS connections because they are the default in many configurations. ## The `/proc/mounts` Complication - Glibc’s `statvfs` implementation checks each directory listed in `/proc/mounts` until it finds the requested mount. - Consequently, a disconnected NFS mount can block `statvfs` even when the agent is checking a different filesystem. - This made changing NFS mount options impractical as a universal fix because Datadog cannot control customers’ system configurations. ## Datadog’s Workaround - The agent now runs `statvfs` on a separate thread. - If the call exceeds a timeout, the main agent thread continues operating. - This approach avoids total metric loss across heterogeneous environments. - The trade-off is a modest increase in memory usage on systems with hard-mounted NFS volumes. The practical lesson is to treat filesystem statistics as potentially blocking operations, especially in environments with NFS. Isolating such calls behind timeouts provides more reliable monitoring than assuming system calls will always return promptly.

pythondatabase-designdatadogmetrics+3

datadogOct 27, 2016

Cheering on coworkers: Building culture with Datadog dashboards (opens in new tab)

Christian’s colleagues built a Datadog dashboard to remotely track his progress in a six-day, 850 km ultramarathon. They scraped live race data from the event website, converted it into Datadog metrics, and visualized his distance, ranking, and elapsed time alongside video and other dashboard elements. At publication, Christian was leading by more than 47 km with 44 hours remaining. ## Extracting Race Data - The event website regularly published runners’ statistics and race progress in plain HTML. - A Python crawler using `Requests` retrieved the webpage. - `BeautifulSoup` parsed the HTML to extract: - Current ranking - Total distance run - Elapsed time - Other race information ## Sending Metrics to Datadog - The team used the Datadog Python client and StatsD to emit metrics through the Datadog Agent. - For each runner, the script sent gauge metrics for: - `runner.distance` - `runner.ranking` - `runner.elapsed_time` - Metrics were tagged with each runner’s name, enabling individual tracking and comparisons. ## Building the Dashboard - The collected metrics were combined into a Datadog dashboard. - The dashboard included: - Live race statistics - A live video feed - Animated GIFs for entertainment - Visualizations of meaningful progress metrics - Screens displaying the dashboard were placed in the company’s New York and Paris offices so colleagues could follow and encourage Christian throughout the race. The project demonstrates how a lightweight web scraper, StatsD metrics, and a monitoring dashboard can turn publicly available data into a live, engaging team experience.

pythondatadogdashboard-designbeautifulsoup+2

datadogOct 27, 2016

Cheering on coworkers: Building culture with Datadog dashboards | Datadog (opens in new tab)

Datadog engineers developed a real-time tracking dashboard to monitor a colleague’s progress during an 850km, six-day ultra-marathon challenge. By scraping public race statistics and piping the data into their monitoring platform, the team created a centralized visualization tool to provide remote support and office-wide engagement. ### Data Extraction and Parsing The team needed to harvest race data that was only available as plain HTML on the event’s official website. * A crawler was built using the Python `Requests` library to automate the retrieval of the webpage's source code. * The team utilized `BeautifulSoup` to parse the HTML and isolate specific data points, such as the runner's current ranking and total distance covered. ### Ingesting Metrics with StatsD Once the data was structured, it was converted into telemetry using the Datadog agent and the `statsd` Python library. * The script utilized `dog.gauge` to emit three primary metrics: `runner.distance`, `runner.ranking`, and `runner.elapsed_time`. * Each metric was assigned a "name" tag corresponding to the runner, allowing the team to filter data and compare participants within the Datadog interface. * The data was updated periodically to ensure the dashboard reflected the most current race standings. ### Dashboard Visualization and Results The final phase involved synthesizing the metrics into a high-visibility dashboard displayed in the company’s New York and Paris offices. * The dashboard combined technical performance graphs with multimedia elements, including live video feeds and GIFs, to create an interactive cheering station. * The system successfully tracked the athlete's 47km lead in real-time, providing the team with immediate updates on his physical progress and elapsed time over the 144-hour event. This project demonstrates how standard observability tools can be repurposed for creative "life-graphing" applications. By combining simple web scraping with metric ingestion, engineers can quickly build custom monitoring solutions for any public data source.

pythondatadogmetricsbeautifulsoup+3

datadogAug 23, 2016

Restroom hacks (opens in new tab)

Datadog built an office bathroom-availability monitor to reduce contention without compromising privacy or existing door functionality. Raspberry Pi 2 devices, GPIO-connected sensors, and simple Unix tools provided a low-maintenance way to report whether bathrooms were occupied. The project showed that the hardest parts were adapting to varied real-world hardware, mounting sensors cleanly, and dealing with unreliable Wi-Fi—not writing software. ## Project Goals - Avoid intrusive monitoring: - No cameras or sensors that could feel invasive. - Provide reliable occupancy information with minimal false positives and negatives. - Use door-lock status where possible as the occupancy signal. - Avoid interfering with existing locks and doors. - Keep devices secure, professional-looking, easy to maintain, and remotely updateable. - Treat the project as a fun hardware experiment. ## Adapting to Different Bathrooms - Bathrooms differed significantly in: - Lock styles, including push-button handles and rotary stall locks. - Number of rooms or stalls. - Availability and location of power outlets. - Wi-Fi quality, especially near concrete walls and older electrical equipment. - These variations required different sensor designs rather than one universal installation. ## Raspberry Pi and Sensor Hardware - Raspberry Pi 2 Model Bs served as the project’s controllers because they: - Ran Linux. - Supported Wi-Fi and SSH administration. - Were compact enough to conceal. - The team used several sensor types: - Magnetic reed switches for detecting door position. - Pin switches for detecting sliding stall-lock positions. - Photoresistors were purchased as a possible way to detect darkness but were not needed in the MVP. - For push-button locks, reed switches detected whether the door was open or closed. Although this could theoretically misreport a closed but unoccupied bathroom, it worked reliably in practice. - Stall-lock sensors were hidden inside hollow metal panels. Automotive-style pin switches were mounted using simple carved wooden blocks that contacted the sliding lock without obstructing it. - Wiring was concealed in wiremolding, with Raspberry Pis placed inside outlet boxes where possible. ## GPIO and Unix-Based Monitoring - Raspberry Pi GPIO pins were accessed through files in `/sys/class/gpio/`. - A Python script read sensor values and translated them into bathroom availability. - Configuration handled differences between normally open and normally closed sensors. - The service was exposed through `tcpserver` and managed with `daemontools`. - A basic command-line client could query status with Netcat, for example: ```sh nc 11.bathrooms.datadog-internal.com 50 ``` ## Making Availability Easy to Use - Employees could check status from the command line. - Some added bathroom availability to TextBar. - Datadog dashboards displayed bathroom status throughout the New York office. - The implementation required very little code; most effort went into sensor selection, physical installation, and network troubleshooting. The project demonstrates that inexpensive, hackable hardware combined with simple Linux tools can solve a practical office problem. For similar systems, prioritize non-intrusive sensors, flexible installation designs, and secure remote management; the resulting software can remain remarkably small.

pythonopen-sourcenetwork-securitymonitoring+4