Performance Profiling

discordJun 30, 20262 min readCurated summary

Cost Attribution in Discord’s API

Discord’s API runs from a shared Python codebase with more than 1,700 endpoints and 700 background tasks across hundreds of Kubernetes deployments. While existing observability tracks performance and reliability, Discord lacked a way to understand hosting costs by product feature or endpoint. Because deployments share code and workers handle multiple features concurrently, the solution was to extend application profiling to allocate deployment costs according to the time spent serving each feature. ## A Large, Continuously Deployed API - Discord operates a unified Python codebase containing: - Over 1,700 API endpoints - Around 700 background tasks - Engineers deploy changes daily to several hundred Kubernetes deployments. - Phased rollouts and instrumentation help monitor: - Latency - Throughput - Error rates - These metrics make it possible to detect regressions affecting users or infrastructure. ## The Missing Cost Dimension - Discord wanted to determine how hosting costs were distributed across product features. - Example questions included: - How much does it cost to send and receive messages? - What does it cost to start a stream or send a Nitro gift? - How do feature costs change over time? - Did a recent code change materially affect a team’s hosting spend? - The goal was to measure costs at both: - Individual endpoint level - Broader feature level, such as chat ## Why Kubernetes Deployment Costs Were Insufficient - Cloud providers can generally report costs by Kubernetes deployment. - However, Discord’s deployments do not map cleanly to product features: - The same codebase runs across all deployments. - Each deployment handles a particular subset of HTTP traffic or background tasks. - Splitting deployments further would make the system impractical to operate. - Discord therefore needed cost attribution without changing its deployment topology. ## Allocating Costs Through Profiling - API worker processes handle multiple tasks concurrently. - A single worker may simultaneously perform work for many different features. - Existing traffic isolation was not detailed enough for feature-level cost analysis. - Discord’s approach was to allocate a deployment’s cost based on the amount of time spent executing code associated with each feature. - By extending its application profiling tools, Discord could track this execution time and use it to estimate feature and endpoint hosting costs. In practice, the profiling-based approach provides a way to analyze infrastructure spending within shared deployments, without requiring separate services or Kubernetes environments for every product feature.

Read original

Performance Profiling Kubernetes Observability Python+2

metaNov 21, 2025Original article

Zoomer: Powering AI Performance at Meta's Scale Through Intelligent Debugging and Optimization (opens in new tab)

Zoomer is Meta’s centralized, automated platform designed to solve performance bottlenecks and GPU underutilization across its massive AI training and inference infrastructure. By integrating deep analytics with scalable data collection, the tool has become the internal standard for optimizing workloads ranging from Llama 3 training to large-scale ads recommendation engines. Ultimately, Zoomer enables significant energy savings and hardware efficiency gains, allowing Meta to accelerate model iteration and increase throughput across its global fleet of GPUs. ### The Three-Layered Architecture * **Infrastructure and Platform Layer:** This foundation utilizes Meta’s Manifold blob storage for trace data and employs fault-tolerant processing pipelines to manage massive trace files across thousands of hosts. * **Analytics and Insights Engine:** This layer performs deep analysis using specialized tools such as Kineto for GPU traces, NVIDIA DCGM for hardware metrics, and StrobeLight for CPU profiling. It automatically detects performance anti-patterns and provides actionable optimization recommendations. * **Visualization and User Interface Layer:** The presentation layer transforms complex data into interactive timelines and heat maps. It integrates with Perfetto for kernel-level inspection and provides drill-down dashboards that highlight outliers across distributed GPU deployments. ### Automated Profiling and Data Capture * **Trigger Mechanisms:** To ensure data accuracy, Zoomer automatically triggers profiling for training workloads during stable states (typically around iteration 550) to avoid startup noise, while inference workloads use on-demand or benchmark-integrated triggers. * **Comprehensive Metrics:** The platform simultaneously collects GPU SM utilization, Tensor Core usage, memory bandwidth, and power consumption via DCGM. * **System-Level Telemetry:** Beyond the GPU, Zoomer captures host-level data including CPU utilization, storage access patterns, and network I/O through dyno telemetry. * **Distributed Communication:** For large-scale training, the tool analyzes NCCL collective operations and inter-node communication patterns to identify stragglers and network bottlenecks. ### Inference and Training Optimization * **Inference Performance:** Zoomer tracks request/response latency, GPU memory allocation patterns, and Thrift request-level profiling to identify bottlenecks in serving user requests at scale. * **Workflow Acceleration:** By correlating application-level annotations—such as forward/backward passes and optimizer steps—with hardware performance, developers can pinpoint exactly which part of a model's execution is inefficient. * **Operational Impact:** These insights have led to significant improvements in Queries Per Second (QPS) for recommendation models and reduced training times for generative AI features by eliminating resource waste. For organizations managing large-scale AI clusters, the Zoomer model suggests that the key to efficiency is moving away from manual, reactive debugging toward an "always-on" automated profiling system. Correlating high-level software phases with low-level hardware telemetry is essential for maximizing the return on investment for expensive GPU resources and maintaining rapid iteration cycles.

Performance Profiling Ai Infrastructure Distributed Training Gpu Optimization+4