discord·Feb 18, 2026·3 min readCurated summary

Osprey: Open Sourcing our Rule Engine

Discord is open-sourcing Osprey, a rule engine designed to help platforms detect and respond to emerging safety threats in real time. Built with ROOST and internet.dev, it processes platform events, evaluates configurable rules, and produces actionable verdicts with minimal engineering effort. Osprey emphasizes scale, rapid rule deployment, transparency, extensibility, and continuous improvement. ## Goals for a Modern Rule Engine Osprey was designed around several requirements: - Process thousands of events per second in real time. - Let teams create and deploy expressive rules within minutes. - Return clear verdicts indicating whether activity is safe, suspicious, or malicious. - Explain how rules were executed and expose errors for investigation and debugging. - Support feedback loops that improve future detection rules. - Remain extensible enough to address new attack patterns. ## Osprey’s Processing Model Osprey accepts platform events called **Actions** through either: - Synchronous gRPC requests. - Asynchronous message queues. The engine evaluates these actions using rules written in SML, a Python-based rule language. Rules can use Python UDFs, Features, and Effects, while synchronous requests can return Verdict effects directly to callers. Outputs are sent to Apache Druid, which powers investigation and analysis tools. ## Actions Actions are JSON-like events submitted to Osprey. - Each action type has a unique name and schema. - Callers can customize the payload with relevant platform data. - Example data includes login attempts, user IDs, usernames, email addresses, and IP addresses. - Rules extract and evaluate values from these action payloads. ## Rules and SML Rules are the central mechanism for detecting suspicious behavior. - SML uses a Python-inspired syntax intended to be accessible to less-technical rule authors. - Rules can reference other rules and extracted data. - Static validation enforces consistent rule-writing practices. - Validation can be extended with Python, from naming conventions to more complex domain-specific checks. - Example rules identify a known spammer by email and apply a `spammer` label to the associated user entity. ## User-Defined Functions UDFs are regular Python functions that extend Osprey’s rule language and standard library. - Built-in capabilities such as `Rule`, `WhenRules`, and `JsonData` are implemented as UDFs. - Teams can add their own UDFs when integrating Osprey into other products. - UDFs can retrieve information from external services, including machine-learning models. - They can be configured for asynchronous execution and access external-service providers through the execution context. - A sample UDF obtains a link-spam score from an external prediction service. ## Features and Entities Features are globally named variables produced during Osprey executions. - Features are exported to Apache Druid for later querying and investigation. - Prefixing a variable name with `_` keeps it local instead of exporting it. - Examples include `UserId` and `UserEmail`, extracted from JSON action data. - Entities are a specialized type of Feature representing persistent objects such as users, servers, or email addresses. - Entities can receive effects such as labels, classifications, and signals. - Entity types determine which effects are valid through static validation. - The Osprey interface provides dedicated Entity Views for examining an entity’s history. ## Effects Effects are outcomes triggered when rules evaluate as true. - They are validated and processed in aggregate after execution. - Effects can modify or annotate entities with labels, classifications, or signals. - Verdict effects can be returned synchronously to inform the requesting service of a safety determination. Osprey’s open-source release gives platforms a reusable foundation for real-time trust and safety enforcement. Teams interested in adopting it can explore the repository at [github.com/roostorg/osprey](https://github.com/roostorg/osprey).

Read original(opens in new tab)

PythonGrpcApache DruidTrust And Safety+3

meta·Dec 22, 2025Original article

Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption (opens in new tab)

The 2025 Typed Python Survey highlights that type hinting has transitioned from an optional feature to a core development standard, with 86% of respondents reporting frequent usage. While mid-career developers show the highest enthusiasm for typing, the ecosystem faces ongoing friction from tooling fragmentation and the complexity of advanced type logic. Overall, the community is pushing for a more robust system that mirrors the expressive power of TypeScript while maintaining Python’s hallmark flexibility. ## Respondent Demographics and Adoption Trends * The survey analyzed responses from 1,241 developers, the majority of whom are highly experienced, with nearly half reporting over a decade of Python expertise. * Adoption is highest among developers with 5–10 years of experience (93%), whereas junior developers (83%) and those with over 10 years of experience (80%) show slightly lower usage rates. * The lower adoption among seniors is attributed to the management of legacy codebases and long-standing habits formed before type hints were introduced to the language. ## Primary Drivers for Typing Adoption * **Incremental Integration:** Developers value the "gradual typing" approach, which allows them to add types to existing projects at their own pace without breaking the codebase. * **Improved Tooling and IDE Support:** Typing significantly enhances developer experience by enabling more accurate autocomplete, jump-to-definition, and inline documentation in IDEs. * **Bug Prevention and Readability:** Type hints act as living documentation that helps catch subtle bugs during refactoring and makes complex codebases easier for teams to reason about. * **Library Compatibility:** Features like Protocols and Generics are highly appreciated, particularly for their synergy with modern libraries like Pydantic and FastAPI that utilize type annotations at runtime. ## Technical Pain Points and Ecosystem Friction * **Third-Party Integration:** A major hurdle is the inconsistent quality or total absence of type stubs in massive libraries like NumPy, Pandas, and Django. * **Tooling Fragmentation:** Developers expressed frustration over inconsistencies between major type checkers like Mypy and Pyright, as well as the slow performance of Mypy in large projects. * **Conceptual Complexity:** Advanced features such as variance (co/contravariance), decorators, and complex nested Generics remain difficult for many developers to implement correctly. * **Runtime Limitations:** Because Python does not enforce types at the interpreter level, some developers find it difficult to justify the verbosity of typing when it offers no native runtime guarantees. ## Most Requested Type System Enhancements * **TypeScript Parity:** There is a strong demand for features found in TypeScript, specifically Intersection types (using the `&` operator), Mapped types, and Conditional types. * **Utility Types:** Developers are looking for built-in utilities like `Pick`, `Omit`, and `keyof` to handle dictionary shapes more effectively. * **Improved Structural Typing:** While `TypedDict` exists, respondents want more flexible, anonymous structural typing to handle complex data structures without excessive boilerplate. * **Performance and Enforcement:** There is a recurring request for an official, high-performance built-in type checker and optional runtime enforcement to bridge the gap between static analysis and execution. As the Python type system continues to mature, developers should prioritize incremental adoption in shared libraries and internal APIs to maximize the benefits of static analysis. While waiting for more advanced features like intersection types, focusing on tooling consistency—such as aligning team standards around a specific type checker—can mitigate much of the friction identified in the 2025 survey.

PythonDeveloper ToolsStatic TypingType Hints+4

slack·Nov 6, 2025·3 min readCurated summary

Build better software to build software better

Slack’s backend build pipeline for Quip and Slack Canvas once took 60 minutes, delaying feedback and slowing delivery. The team improved build performance by applying familiar software-engineering techniques—caching, parallelization, precise interfaces, and careful decomposition—using Bazel. The central argument is that build systems should be designed like high-performance programs: do less work, distribute unavoidable work, and define work units rigorously. ## Modeling Builds as Dependency Graphs - Applications can be represented as directed acyclic graphs of source files, intermediate artifacts, and deployable outputs. - A backend artifact depends on Python files, while a frontend artifact depends on TypeScript files. - Changing a Python file should rebuild the backend but not unrelated frontend components. - Clearly defined graph nodes allow build systems to optimize work rather than rebuilding everything. ## Caching and Hermetic Work - Caching avoids repeating expensive operations by storing outputs for known inputs. - The article uses a cached recursive `factorial()` function as an analogy: - The input is the cache key. - The return value is the cached artifact. - Effective caching requires work to be: - **Hermetic:** dependent only on explicitly provided inputs. - **Idempotent:** producing the same output for the same inputs. - Cache hit rate matters: poorly defined work units produce more cache misses. ## Granular Cache Units - Caching an entire `process_images(images, transforms)` operation is inefficient because changing one image invalidates the result for every image. - A more granular design caches `process_image(image, transform)` independently. - The higher-level operation can then reuse cached results and process only new image-transform combinations. - Smaller, well-defined units generally improve cache reuse and reduce rebuild time. ## Parallelizing Independent Work - Image processing can also be distributed across CPU threads using `ThreadPoolExecutor`. - Parallel work requires: - Completely specified inputs and outputs. - The ability to transfer data across thread, process, or network boundaries. - Handling completion and failure in any order. - APIs must document ordering guarantees; the threaded example returns images in completion order rather than input order. - Work-unit granularity affects scalability: - Too few large tasks limit available parallelism. - Too many tiny tasks may introduce coordination overhead. - The appropriate balance depends on the workload. ## Applying These Principles to Bazel - Bazel represents builds as directed acyclic graphs made of targets. - Each target defines: - Its input or dependency files. - Its output files. - The commands that transform inputs into outputs. - This structure provides the foundation for caching and parallel execution, just as explicit function inputs and outputs enable those optimizations in application code. The practical recommendation is to design build steps as small, hermetic, idempotent, and independently executable units. Combined with Bazel’s dependency graph, this lets teams avoid unnecessary work, maximize cache hits, and run independent tasks concurrently—turning slow build pipelines into faster sources of developer feedback.

Read original(opens in new tab)

PythonTypeScriptCachingBazel+4

google·Nov 5, 2025Original article

DS-STAR: A state-of-the-art versatile data science agent (opens in new tab)

DS-STAR is an advanced autonomous data science agent developed to handle the complexity and heterogeneity of real-world data tasks, ranging from statistical analysis to visualization. By integrating a specialized file analysis module with an iterative planning and verification loop, the system can interpret unstructured data and refine its reasoning steps dynamically based on execution feedback. This architecture allows DS-STAR to achieve state-of-the-art performance on major industry benchmarks, effectively bridging the gap between natural language queries and executable, verified code. ## Comprehensive Data File Analysis The framework addresses a major limitation of current agents—the over-reliance on structured CSV files—by implementing a dedicated analysis stage for diverse data formats. * The system automatically scans a directory to extract context from heterogeneous formats, including JSON, unstructured text, and markdown files. * A Python-based analysis script generates a textual summary of the data structure and content, which serves as the foundational context for the planning phase. * This module ensures the agent can navigate complex, multi-file environments where critical information is often spread across non-relational sources. ## Iterative Planning and Verification Architecture DS-STAR utilizes a sophisticated loop involving four specialized roles to mimic the workflow of a human expert conducting sequential analysis. * **Planner and Coder:** A Planner agent establishes high-level objectives, which a Coder agent سپس translates into executable Python scripts. * **LLM-based Verification:** A Verifier agent acts as a judge, assessing whether the generated code and its output are sufficient to solve the problem or if the reasoning is flawed. * **Dynamic Routing:** If the Verifier identifies gaps, a Router agent guides the refinement process by adding new steps or correcting errors, allowing the cycle to repeat for up to 10 rounds. * **Intermediate Review:** The agent reviews intermediate results before proceeding to the next step, similar to how data scientists use interactive environments like Google Colab. ## Benchmarking and State-of-the-Art Performance The effectiveness of the DS-STAR framework was validated through rigorous testing against existing agents like AutoGen and DA-Agent. * The agent secured the top rank on the public DABStep leaderboard, raising accuracy from 41.0% to 45.2% compared to previous best-performing models. * Performance gains were consistent across other benchmarks, including KramaBench (39.8% to 44.7%) and DA-Code (37.0% to 38.5%). * DS-STAR showed a significant advantage in "hard" tasks—those requiring the synthesis of information from multiple, varied data sources—demonstrating its superior versatility in complex environments. By automating the time-intensive tasks of data wrangling and verification, DS-STAR provides a robust template for the next generation of AI assistants. Organizations looking to scale their data science capabilities should consider adopting iterative agentic workflows that prioritize multi-format data understanding and self-correcting execution loops.

PythonAILarge Language ModelsMachine Learning+4

google·Jul 31, 2025Original article

MLE-STAR: A state-of-the-art machine learning engineering agent (opens in new tab)

MLE-STAR is a state-of-the-art machine learning engineering agent designed to automate complex ML tasks by treating them as iterative code optimization challenges. Unlike previous agents that rely solely on an LLM’s internal knowledge, MLE-STAR integrates external web searches and targeted ablation studies to pinpoint and refine specific pipeline components. This approach allows the agent to achieve high-performance results, evidenced by its ability to win medals in 63% of Kaggle competitions within the MLE-Bench-Lite benchmark. ## External Knowledge and Targeted Ablation The core of MLE-STAR’s effectiveness lies in its ability to move beyond generic machine learning libraries by incorporating external research and specific performance testing. * The agent uses web search to retrieve task-specific, state-of-the-art models and approaches rather than defaulting to familiar libraries like scikit-learn. * Instead of modifying an entire script at once, the system conducts an ablation study to evaluate the impact of individual pipeline components, such as feature engineering or model selection. * By identifying which code blocks have the most significant impact on performance, the agent can focus its reasoning and optimization efforts where they are most needed. ## Iterative Refinement and Intelligent Ensembling Once the critical components are identified, MLE-STAR employs a specialized refinement process to maximize the effectiveness of the generated solution. * Targeted code blocks undergo iterative refinement based on LLM-suggested plans that incorporate feedback from prior experimental failures and successes. * The agent features a unique ensembling strategy where it proposes multiple candidate solutions and then designs its own method to merge them. * Rather than using simple validation-score voting, the agent iteratively improves the ensemble strategy itself, treating the combination of models as a distinct optimization task. ## Robustness and Safety Verification To ensure the generated code is both functional and reliable for real-world deployment, MLE-STAR incorporates three specialized diagnostic modules. * **Debugging Agent:** Automatically analyzes tracebacks and execution errors in Python scripts to provide iterative corrections. * **Data Leakage Checker:** Reviews the solution script prior to execution to ensure the model does not improperly access test dataset information during the training phase. * **Data Usage Checker:** Analyzes whether the script is utilizing all available data sources, preventing the agent from overlooking complex data formats in favor of simpler files like CSVs. By combining external grounding with a granular, component-based optimization strategy, MLE-STAR represents a significant shift in automated machine learning. For organizations looking to scale their ML workflows, such an agent suggests a future where the role of the engineer shifts from manual coding to high-level supervision of autonomous agents that can navigate the vast landscape of research and data engineering.

PythonAILarge Language ModelsAI Agents+5

line·May 23, 2025Original article

Implementing a RAG-based (opens in new tab)

To address the operational burden of handling repetitive user inquiries for the AWX automation platform, LY Corporation developed a support bot utilizing Retrieval-Augmented Generation (RAG). By combining internal documentation with historical Slack thread data, the system provides automated, context-aware answers that significantly reduce manual SRE intervention. This approach enhances service reliability by ensuring users receive immediate assistance while allowing engineers to focus on high-priority development tasks. ### Technical Infrastructure and Stack * **Slack Integration**: The bot is built using the **Bolt for Python** framework to handle real-time interactions within the company’s communication channels. * **LLM Orchestration**: **LangChain** is used to manage the RAG pipeline; the developers suggest transitioning to LangGraph for teams requiring more complex multi-agent workflows. * **Embedding Model**: The **paraphrase-multilingual-mpnet-base-v2** (SBERT) model was selected to support multi-language inquiries from LY Corporation’s global workforce. * **Vector Database**: **OpenSearch** serves as the vector store, chosen for its availability as an internal PaaS and its efficiency in handling high-dimensional data. * **Large Language Model**: The system utilizes **OpenAI (ChatGPT) Enterprise**, which ensures business data privacy by preventing the model from training on internal inputs. ### Enhancing LLM Accuracy through RAG and Vector Search * **Overcoming LLM Limits**: Traditional LLMs suffer from "hallucinations," lack of up-to-date info, and opaque sourcing; RAG fixes this by providing the model with specific, trusted context during the prompt phase. * **Embedding and Vectorization**: Textual data from wikis and chats are converted into high-dimensional vectors, where semantically similar phrases (e.g., "Buy" and "Purchase") are stored in close proximity. * **k-NN Retrieval**: When a user asks a question, the bot uses **k-Nearest Neighbors (k-NN)** algorithms to retrieve the top *k* most relevant snippets of information from the vector database. * **Contextual Generation**: Rather than relying on its internal training data, the LLM generates a response based specifically on the retrieved snippets, leading to higher accuracy and domain-specific relevance. ### AWX Support Bot Workflow and Data Sources * **Multi-Source Indexing**: The bot references two main data streams: the official internal AWX guide wiki and historical Slack inquiry threads where previous solutions were discussed. * **Automated First Response**: The workflow begins when a user submits a query via a Slack workflow; the bot immediately processes the request and provides an initial AI-generated answer. * **Human-in-the-Loop Validation**: After receiving an answer, users can click "Issue Resolved" to close the ticket or "Call AWX Admin" if the AI's response was insufficient. * **Efficiency Gains**: This tiered approach filters out "RTFM" (Read The F***ing Manual) style questions, ensuring that human administrators only spend time on unique or complex technical issues. Implementing a RAG-based support bot is a highly effective strategy for SRE teams looking to scale their internal support without increasing headcount. For the best results, organizations should focus on maintaining clean internal documentation and selecting embedding models that reflect the linguistic diversity of their specific workforce.

PythonAIRAGVector Db+5

discord·Feb 27, 2025·3 min readCurated summary

Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries. ## Why Traditional A/B Testing Was Difficult - Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile. - The feature inherently involves networks: one user sends a message and another receives it. - Network effects violate the assumption that treatment and control users behave independently, known as SUTVA. - Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization. - User-level A/B testing risked cross-group interactions. - Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior. ## How Synthetic Controls Work - Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units. - Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from: - 50% Argentina - 30% Uruguay - 20% Chile - The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes. - This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country. - It also produces a result that may be more representative than learning only how users in one specific country respond. ## Benefits and Evaluation - Synthetic controls can account for both observable and unobservable differences between regions. - They require: - Outcome data for the treated unit before and after treatment - Data from multiple untreated control units over the same periods - An analytical library, such as `Synth` in R or `SyntheticControlMethods` in Python - Discord evaluates the fit using Mean Squared Prediction Error (MSPE). - A close pre-treatment fit indicates that the synthetic control is a credible counterfactual. - A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region. - Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention. Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.

Read original(opens in new tab)

PythonAb TestingCausal InferenceStatistical Analysis+3

figma·Apr 24, 2024·3 min readCurated summary

Speeding Up C++ Build Times | Figma Blog

Figma cut C++ build times roughly in half by addressing unnecessary header inclusion rather than relying solely on faster hardware or caching. The team found that compiled bytes were growing much faster than the codebase itself, making transitive header dependencies the main culprit. They combined automated include analysis with CI-based measurement to prevent both unused includes and costly dependency regressions. ## Why Build Times Were Getting Worse - In 2023, Figma’s codebase grew by about 10%, but build times increased by 50%. - C++ builds were a major productivity problem and a top concern in internal developer surveys. - Faster M1 Max machines, Ccache, and remote caching provided only temporary or insufficient improvements. - The team observed that build times were closely related to the amount of code passed to the compiler after preprocessing. ## How C++ Header Inclusion Affects Builds - The preprocessor expands every `#include` into a single large file before compilation. - Transitive dependencies are included as well: - If file C includes B, and B includes A, C receives the contents of both A and B. - As a result, a small source change can cause the compiler to process a very large amount of unrelated code. ## Removing Unnecessary Includes - Figma suspected that many files included headers they did not use directly or relied on headers only for transitive dependencies. - Removing unnecessary includes from the largest files produced: - A 31% reduction in compiled bytes. - A 25% reduction in cold build time. - These results confirmed that compiled byte volume was strongly correlated with build performance. ## DIWYDU: Automating Include Cleanup - Google’s Include What You Use (IWYU) tool was considered but proved difficult to apply retroactively to Figma’s large codebase. - Figma created a less strict alternative called **Don’t Include What You Don’t Use (DIWYDU)**. - DIWYDU: - Uses Python bindings for `libclang`. - Parses source and header files into Clang Abstract Syntax Trees. - Identifies types, functions, and variables directly used by each file. - Flags headers that are included but provide no directly used symbols. - The tool runs on feature branches to prevent unnecessary includes from accumulating. ## DIWYDU’s Limitations - It analyzes Figma-owned files but excludes Standard Template Library headers. - STL headers may define symbols through private internal includes, making direct dependency analysis difficult. - Python’s `libclang` bindings expose less of Clang’s AST than the compiler’s native C++ APIs, sometimes producing `UNEXPOSED_EXPR` nodes. - A future C++ implementation could provide more accurate AST access. - DIWYDU cannot detect cases where an included header is genuinely required but excessively large. - Such regressions may need forward declarations or header decomposition instead. ## Measuring Dependency Growth with `includes.py` - Figma built `includes.py` to measure the transitive bytes associated with each source file. - The tool is written entirely in Python and typically runs in a few seconds without invoking Clang. - It: - Crawls first-party source, header, and generated files. - Counts file sizes. - Builds a dependency graph. - Estimates the total bytes passed to the compiler for each source file. - Standard library includes are treated as zero bytes because Figma mainly accesses them through internal wrapper directories. - CI uses the measurements to compare pull requests and warn authors when changes significantly increase compiled bytes. Figma’s approach demonstrates that controlling header dependencies can deliver larger and more durable gains than simply adding hardware or cache capacity. Teams working on large C++ codebases should automate unused-include checks, measure transitive dependency size in CI, and use forward declarations or smaller headers when necessary.

Read original(opens in new tab)

PythonCI/CDC++Static Analysis+4

datadog·May 12, 2022·3 min readCurated summary

How Datadog's IT team automated account inactivity and SaaS spend management

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

Read original(opens in new tab)

PythonDatabase DesignPostgreSQLMicroservices+3

datadog·Oct 15, 2021Original article

Our journey taking Kubernetes state metrics to the next level | Datadog (opens in new tab)

Datadog’s container observability team significantly improved the performance of kube-state-metrics (KSM) by contributing core architectural enhancements to the upstream open-source project. Faced with scalability bottlenecks where metrics collection for large clusters took tens of seconds and generated massive data payloads, they revamped the underlying library to achieve a 15x improvement in processing duration. These contributions allowed for high-granularity monitoring at scale, ensuring that the Datadog Agent can efficiently handle millions of metrics across thousands of Kubernetes nodes. ### Challenges with KSM Scalability * KSM uses the informer pattern to expose cluster-level metadata via the Openmetrics format, but the volume of data grows exponentially with cluster size. * In high-scale environments, a single node generates approximately nine metrics, while a single pod can generate up to 40 metrics. * In clusters with thousands of nodes and tens of thousands of pods, the `/metrics` endpoint produced payloads weighing tens of megabytes. * The time required to crawl these metrics often exceeded 15 seconds, forcing administrators to reduce check frequency and sacrifice real-time data granularity. ### Limitations of Legacy Implementations * KSM v1 relied on a monolithic loop that instantiated a Builder to track resources via stores, but it lacked efficient hooks for metric generation. * The original Python-based Datadog Agent check struggled with the "data dump" approach of KSM, where all metrics were processed at once during query time. * To manage the load, Datadog was forced to split KSM into multiple deployments based on resource types (e.g., separate deployments for pods, nodes, and secondary resources like services or deployments). * This fragmentation made the infrastructure more complex to manage and did not solve the fundamental issue of inefficient metric serialization. ### Architectural Improvements in KSM v2.0 * Datadog collaborated with the upstream community during the development of KSM v2.0 to introduce a more extensible design. * The team focused on improving the Builder and metric generation hooks to prevent the system from dumping the entire dataset at query time. * By moving away from the restrictive v1 library structure, they enabled more efficient reconciliation of metric names and metadata joins. * The resulting 15x performance gain allows the Datadog Agent to reconcile labels and tags—such as joining deployment labels to specific metrics—without the significant latency overhead previously experienced. Contributing back to the open-source community proved more effective than maintaining internal forks for scaling Kubernetes infrastructure. Organizations running high-density clusters should prioritize upgrading to KSM v2.0 and optimizing their agent configurations to leverage these architectural improvements for better observability performance.

PythonDatabase DesignKubernetesDatadog+3

datadog·Oct 15, 2021·3 min readCurated summary

Our journey taking Kubernetes state metrics to the next level

Datadog contributed major scalability improvements to kube-state-metrics (KSM), after discovering that its metric collection process struggled with very large Kubernetes clusters. Millions of metrics could require tens of megabytes and tens of seconds to process every 15 seconds, forcing Datadog to reduce collection frequency. Their redesign improved collection duration by 15x and enabled more granular monitoring at scale. ## Datadog’s Kubernetes Observability Role - The Datadog Containers team monitors Kubernetes infrastructure and ensures reliable collection of: - Logs - Traces - Custom metrics - Profiles - Security signals - KSM is central to Datadog products such as Kubernetes metrics integration and Orchestrator Explorer. ## How Kubernetes State Metrics Works - KSM uses Kubernetes informers to watch objects registered with the API server. - Enabled collectors monitor resources such as pods, nodes, deployments, and services. - It generates lifecycle and metadata metrics in text-based OpenMetrics format. - Users can restrict monitored resources through the `resources` flag. - The Datadog Agent’s KSM check: - Runs every 15 seconds. - Discovers KSM containers. - Crawls their `/metrics` endpoint. - Reconciles metric metadata and applies configured label joins. - Label joins allow metadata from one metric, such as a deployment label, to become a tag on other metrics for the same object. ## Scaling Challenges - Datadog found that KSM needed to be split across multiple deployments beyond a few hundred nodes and thousands of pods. - Their deployments divided collectors by resource type: - Pods - Nodes - Services, deployments, jobs, persistent volumes, and other resources - Metric volume varied substantially: - Endpoints, jobs, and deployments produced roughly five metrics per object. - Nodes produced around nine metrics each. - Pods produced around 40 metrics each. - Large clusters with thousands of nodes and tens of thousands of pods could generate millions of metrics per scrape. - Crawling the metrics endpoint could take tens of seconds and transfer tens of megabytes. - Datadog had to reduce check frequency, sacrificing metric granularity and user experience. ## KSM’s Original Architecture - KSM v1 relied on a central loop that created a Builder and managed resource stores. - Each store used informers to track a particular Kubernetes resource. - For example, an HPA store maintained the list-and-watch logic for HorizontalPodAutoscalers. - The Builder generated metrics from the tracked resources. - Datadog identified two limitations: - Too much data was emitted and processed at query time. - The Builder did not provide a suitable extension point for custom metric-generation logic. ## Contributing the Redesign Upstream - As the KSM community prepared version 2.0 in early 2020, Datadog saw an opportunity to address its scalability and extensibility problems in the upstream project. - Rather than maintaining a private solution, the team contributed its findings and improvements to the open-source community. - The resulting work reportedly reduced metric collection duration by 15x, making high-scale, more frequent collection practical. Datadog’s experience shows that upstream open-source collaboration can solve internal infrastructure bottlenecks while improving the project for the broader Kubernetes community.

Read original(opens in new tab)

PythonDatabase DesignKubernetesDatadog+4

datadog·Feb 22, 2021·3 min readCurated summary

How we minimized the overhead of Kubernetes in our job system

Kubernetes can improve machine management and scalability, but its scheduling and runtime overhead can significantly reduce job throughput if configured poorly. Datadog found its Kubernetes-based job system used more CPU and completed jobs 40–50% more slowly than the previous VM-based system. By designing a controlled experiment, choosing better metrics, and tuning pod resource requests, the team recovered performance to roughly VM parity while investigating the overhead of running one parent process per pod. ## Designing a Comparable Experiment - The initial comparison was difficult because the Kubernetes and VM deployments differed in: - Number of nodes - Number of worker-parent clusters - Workload and enqueue rate - A controlled experiment was created using: - Identical `c5.2xlarge` machines - The same kernel version, `3.13.0-141` - Both systems repeatedly running a simple Python job - Each Kubernetes pod contained one parent process and its worker processes, making pod count equivalent to parent-process count per node. - The older kernel did not include CPU mitigations, avoiding that variable in the comparison. ## Choosing Useful Performance Metrics ### Measuring node effort - Load average initially appeared useful for measuring machine utilization. - Kubernetes background processes—such as cluster polling and pod-state checks—artificially increased load average. - Load average counts runnable processes rather than the amount of CPU time they actually consume. - The team therefore used CPU idle time instead: - It measures unused CPU capacity. - It reflects actual CPU work rather than the number of active processes. ### Measuring system performance - The job system optimized for throughput rather than latency. - Throughput was measured by the number of jobs completed within 30 seconds. - Latency remained useful for detecting queueing problems, but throughput was the primary success metric. ## Tuning Kubernetes Resource Requests - The main performance gains came from improving pod scheduling. - The target was six pods per `c5.2xlarge` node. - Initially, each pod requested: - One full CPU core - More memory than necessary - Since the node had eight cores and approximately 1.5 GiB of memory consumed by Kubernetes and system services, only four pods could be scheduled. - Requests were reduced to: - `100m` CPU, or 100 millicores - `500 MB` memory - CPU tuning generally enabled six pods per node, although some nodes still scheduled only five. - Further memory reduction was needed because system daemons consumed enough memory to prevent six pods from fitting on some nodes. - Resource requests affect scheduling minimums, while limits constrain containers after they start. - These request changes did not slow jobs because the pods still received sufficient resources to operate. ## One Parent Process per Pod - The team considered placing multiple parent processes in each pod to reduce potential pod overhead. - One parent plus its workers was a natural application unit and simplified orchestration. - The decision depended on how much overhead each pod introduced: - High overhead would favor fewer, larger pods. - Low overhead would favor one parent per pod for simpler management. - Using `pstree`, the team identified six job-system instances per node and traced their process trees through components such as: - `containerd-shim` - `tini` - The application process - They estimated that each pod included overhead associated with three containers, particularly `containerd-shim`. - CPU overhead was then investigated using `perf sched`. The practical lesson is to compare equivalent workloads, measure actual CPU consumption rather than relying blindly on load average, and tune Kubernetes requests for the desired packing density. Resource requests should be large enough for reliable operation but not so large that they unnecessarily prevent pods from being scheduled together.

Read original(opens in new tab)

PythonDatabase DesignKubernetesResource Management+4

datadog·Oct 7, 2020·3 min readCurated summary

How we wrote a Python profiler

Datadog built a Python continuous profiler because Python lacked Java-style, always-on production profiling tools. The post argues that deterministic profilers such as `cProfile` impose too much overhead for continuous use, while statistical profiling can provide representative performance data with minimal disruption. Datadog’s profiler addresses this through modular collectors, recording, scheduling, and data export. ## Profiling Versus Tracing - **Profiling** measures resource consumption such as CPU time and memory allocation to reveal performance problems. - **Tracing** records individual operations—such as SQL queries or HTTP requests—within a request timeline. - Tracing explains request latency, but profiling provides deeper insight into code-level execution and operating-system resource usage. ## Limitations of Deterministic Python Profilers - Python’s `cProfile`, available since CPython 2.5, records every function call and the time spent in each call. - It can provide a complete execution flow, but its usefulness depends heavily on code structure: - A program built around a few large functions produces little actionable detail. - A program containing thousands of functions can incur two- or three-times runtime overhead. - This overhead makes deterministic profiling unsuitable for always-on production environments. ## Why Profile in Production? - Optimizing without profiling is essentially guessing; real workloads often differ from development environments. - Production systems vary from developer machines in hardware, concurrency, input data, and workload behavior. - Continuous profiling captures how an application actually consumes resources under authentic conditions. - These requirements lead to statistical rather than deterministic profiling. ## Statistical Profiling in Python - Statistical profilers sample program activity periodically instead of recording every function call. - Individual short-lived calls may be missed, but repeated sampling over hours produces a reliable picture of resource consumption. - Lower overhead allows the application to run closer to its normal, unprofiled behavior. - Datadog evaluated numerous open-source Python profilers but found limitations involving platform support, collected data, or presentation-focused designs. - The team therefore developed its own statistical profiler, incorporating ideas from the tools it studied. ## Datadog Python Profiler Design The profiler was designed around three constraints: - Keep runtime overhead as low as possible. - Make deployment simple. - Support common operating systems and environments. Its architecture, inspired by the JDK Flight Recorder, consists of: - **Collectors:** Gather data such as CPU usage and memory allocation. - **Recorder:** Stores events produced by collectors. - **Exporter:** Sends profiling data outside the application. - **Scheduler:** Invokes components at appropriate intervals, such as exporting data every 60 seconds. - **Profiler:** Provides the high-level interface used by applications. ## Stack Collection - The stack collector is the primary built-in collector. - It wakes 100 times per second and captures the execution stack of every Python thread. - For each thread, it gathers information including: - The currently executing function - CPU time consumed - Exceptions being handled - The collector monitors the time required to inspect the application so it can control and limit its own CPU overhead. A statistical profiler with low overhead is the appropriate foundation for continuous production profiling, giving teams evidence about real application behavior without substantially changing that behavior.

Read original(opens in new tab)

PythonDatadogJavaProfiling+4

datadog·Jun 3, 2019·3 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised. ## Independent Integration Publishing - Agent integrations were traditionally bundled into full Agent releases. - This delayed important integration updates and prevented users from trying new integrations immediately. - Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself. ## End-to-End Verification with in-toto - TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure. - in-toto defines the software supply chain as a fixed sequence of signed steps. - Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work. - The integration pipeline includes: - Developers signing Python and YAML source files. - CI/CD packaging the source into Python wheels without modifying existing wheels. - A signing step applying TUF signatures to the wheels. - The Datadog Agent verifying that the downloaded wheel matches the developer-signed source. ## Secure Distribution with TUF - in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys. - TUF supplies signed, compromise-resilient metadata for: - The root of trust for wheels and supply-chain metadata. - The in-toto-defined workflow. - Public keys used to verify the workflow. - TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata. - Offline trust bootstrapping and protected developer keys are essential to the overall security model. ## Hardware-Protected Developer Signing - Developers use Yubikeys to generate and store GPG signing keys. - Private keys cannot be exported from the device, assuming correct firmware. - Signing requires both a secret PIN and physical interaction with the Yubikey. - A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk. ## Transparent Verification for Users - The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations. - Users need no workflow changes under normal conditions. - If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure. Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.

Read original(opens in new tab)

PythonDatabase DesignDatadogCI/CD+4

datadog·Apr 16, 2018·2 min readCurated summary

Cgo and Python

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

Read original(opens in new tab)

PythonDatabase DesignGoDatadog Agent+4