Observability | Techlist.io

naverNov 17, 2025Original article

Naver TV (opens in new tab)

NAVER is transitioning its internal search monitoring platform, SEER, to an architecture built on OpenTelemetry and open-source standards to achieve a more scalable and flexible observability environment. By adopting a vendor-agnostic approach, the engineering team aims to unify the collection of metrics, logs, and traces while contributing back to the global OpenTelemetry ecosystem. This shift underscores the importance of standardized telemetry protocols in managing complex, large-scale service infrastructures. ### Standardizing Observability with OTLP * The transition focuses on the OpenTelemetry Protocol (OTLP) as the primary standard for transmitting telemetry data across the platform. * Moving away from proprietary formats allows for a unified data model that encompasses metrics, traces, and logs, ensuring consistency across different services. * A standardized protocol simplifies the integration of various open-source backends, reducing the engineering overhead associated with supporting multiple telemetry formats. ### The OpenTelemetry Collector Pipeline * The Collector acts as a critical intermediary, decoupling the application layer from the storage backend to provide greater architectural flexibility. * **Receivers** are used to ingest data from diverse sources, supporting both OTLP-native applications and legacy systems. * **Processors** enable data transformation, filtering, and metadata enrichment (such as adding resource attributes) before the data reaches its destination. * **Exporters** manage the delivery of processed telemetry to specific backends like Prometheus for metrics or Jaeger for tracing, allowing for easy swaps of infrastructure components. ### Automated Management via OpenTelemetry Operator * The OpenTelemetry Operator is utilized within Kubernetes environments to automate the deployment and lifecycle management of the Collector. * It facilitates auto-instrumentation, allowing developers to collect telemetry from applications without manual code changes for every service. * The Operator ensures that the observability stack scales dynamically alongside the production workloads it monitors. ### Open-Source Contribution and Community * Beyond mere adoption, the NAVER engineering team actively participates in the OpenTelemetry community by sharing bug fixes and feature enhancements discovered during the SEER migration. * This collaborative approach ensures that the specific requirements of high-traffic enterprise environments are reflected in the evolution of the OpenTelemetry project. Adopting OpenTelemetry is a strategic move for organizations looking to avoid vendor lock-in and build a future-proof monitoring stack. For a successful implementation, teams should focus on mastering the Collector's pipeline configuration to balance data granularity with processing performance across distributed systems.

Observability Monitoring Opentelemetry Sre+3

tossNov 13, 2025Original article

Frontend Code That Lasts 1 (opens in new tab)

Toss Payments evolved its Payment SDK to solve the inherent complexities of integrating payment systems, where developers must navigate UI implementation, security flows, and exception handling. By transitioning from V1 to V2, the team moved beyond simply providing a library to building a robust, architecture-driven system that ensures stability and scalability across diverse merchant environments. The core conclusion is that a successful SDK must be treated as a critical infrastructure layer, relying on modular design and deep observability to handle the unpredictable nature of third-party runtimes. ## The Unique Challenges of SDK Development * SDK code lives within the merchant's runtime environment, meaning it shares the same lifecycle and performance constraints as the merchant’s own code. * Internal logging can inadvertently create bottlenecks; for instance, adding network logs to a frequently called method can lead to "self-DDoS" scenarios that crash the merchant's payment page. * Type safety is a major hurdle, as merchants may pass unexpected data types (e.g., a number instead of a string), causing fatal runtime errors like `startsWith is not a function`. * The SDK acts as a bridge for technical communication, requiring it to function as both an API consumer for internal systems and an API provider for external developers. ## Ensuring Stability through Observability * To manage the unpredictable ways merchants use the SDK, Toss implemented over 300 unit tests and 500 E2E integration tests based on real-world use cases. * The team utilizes a "Global Trace ID" to track a single payment journey across both the frontend and backend, allowing for seamless debugging across the entire system. * A custom Monitoring CLI was developed to compare payment success rates before and after deployments, categorized by merchant and runtime environment (e.g., PC Chrome vs. Android WebView). * This observability infrastructure enables the team to quickly identify edge-case failures—such as a specific merchant's checkout failing only on mobile WebViews—which are often missed by standard QA processes. ## Scaling with Modular Architecture * To avoid "if-statement hell" caused by merchant-specific requirements (e.g., fixing installment months or custom validation for a specific store), Toss moved to a "Lego-block" architecture. * The SDK is organized into three distinct layers based on the "reason for change" principle: * **Public Interface Layer:** Manages the contract with the merchant, validating inputs and translating them into internal domain models. * **Domain Layer:** Encapsulates core business logic and payment policies, keeping them isolated from external changes. * **External Service Layer:** Handles dependencies like Server APIs and Web APIs, ensuring technical shifts don't leak into the business logic. * This separation allows the team to implement custom merchant logic by swapping specific blocks without modifying the core codebase, reducing the risk of regressions and lowering maintenance costs. For developers building SDKs or integration tools, the shift from monolithic logic to a layered, observable architecture is essential. Prioritizing the separation of domain logic from public interfaces and investing in environment-specific monitoring allows for a highly flexible product that remains stable even as the client-side environment grows increasingly complex.

Observability TypeScript Software Architecture Elasticsearch+3

datadogNov 4, 20251 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog

The supplied content does not include the blog post’s article text. It contains Datadog’s navigation menu and a promotional link to its Gartner recognition, while the URL suggests the post concerns CDC replication and search. ## Available Information - Datadog was named a **Leader in the 2026 Gartner Magic Quadrant for Observability Platforms**. - The page promotes Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and digital experience - Software delivery and service management - AI-powered observability - The referenced article URL is `engineering/cdc-replication-search`, indicating a likely focus on **change data capture (CDC), data replication, and search systems**. ## Missing Article Details - No sections, technical explanations, architecture diagrams, implementation details, or conclusions from the blog post are present in the supplied text. - A reliable summary of the CDC replication approach cannot be produced without the article body. Please provide the full post text or its main sections for a complete summary.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+2

datadogOct 21, 20252 min readCurated summary

Detecting malicious pull requests at scale with LLMs | Datadog

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+4

datadogOct 1, 20251 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, but no substantive discussion of the linked “Husky Query Architecture” article. ## Available Content ### Datadog’s Observability Platform - Datadog promotes products covering: - Infrastructure and container monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI and agent observability - The navigation emphasizes Datadog’s broad, integrated platform approach. ### Gartner Recognition - The page links to Datadog’s announcement that it was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The supplied text does not include the evaluation criteria, cited strengths, limitations, or Gartner’s comparative analysis. No reliable summary of the Husky query architecture can be produced without the article’s body text.

Read original

Observability Kubernetes Application Performance Monitoring Log Management+4

datadogSep 18, 20251 min readCurated summary

From hand-tuned Go to self-optimizing code: Building BitsEvolve | Datadog

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional link announcing its 2026 Gartner Magic Quadrant recognition. As a result, there is not enough article content to produce a reliable technical summary. ### Available Information - Datadog is promoted as a “Leader” in the Gartner Magic Quadrant for Observability Platforms. - The page links to Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Incident and service management - AI and automation - The referenced blog URL appears to be titled **“Self-Optimizing System,”** but its article text is not included. Please provide the blog post’s main content or a complete page extract for an accurate summary.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+3

datadogSep 18, 20253 min readCurated summary

From hand-tuned Go to self-optimizing code: Building BitsEvolve

Datadog found that small Go-level optimizations can produce substantial infrastructure savings when applied to heavily used, autoscaled services. Manual work—such as removing bounds checks and prioritizing common input paths—delivered improvements ranging from 25% to over 90% in targeted functions. These successes also revealed the need to automate expert optimization techniques through systems like Datadog’s internal BitsEvolve. ## Finding Hotspots That Matter - Micro-optimizations are worthwhile when: - Functions run millions or billions of times. - Services are aggressively autoscaled, allowing CPU savings to reduce machine counts. - Resource usage drops measurably. - Datadog focused on high-throughput services processing timeseries tags and values. - Individual hotspots sometimes represented only 0.5% of compute, but repeated savings could add up to tens of thousands of dollars annually. - The broader goal was a 5–10% reduction in CPU usage across many improvements. ## Removing Bounds Checks from `NormalizeTag` - `NormalizeTag` called `isNormalizedASCIITag`, a frequently executed validator for ASCII tag strings. - AI coding tools suggested changes that were correct but produced no measurable performance gains. - Examining Go assembly with Compiler Explorer revealed two `runtime.panicBounds` calls per loop iteration. - Restructuring the loop eliminated unnecessary bounds checks and enabled further tuning. - The function became 25% faster, reducing service CPU usage by 0.75% and producing projected annual savings of tens of thousands of dollars. ## Using Observability to Optimize for Real Inputs - `NormalizeTagArbTagValue` handled arbitrary input, including invalid UTF-8 and binary data, and consumed 4.5% of CPU in its processing service. - Production data showed: - Nearly all inputs were ASCII. - UTF-8 appeared in fewer than 3% of cases. - Invalid UTF-8 represented less than 0.01% of inputs. - A fast path optimized for common ASCII data made the function more than 90% faster without reducing correctness or safety. - The change generated projected annual savings of hundreds of thousands of dollars. - The result demonstrated that observability is essential: optimization decisions should reflect actual workloads rather than hypothetical edge cases. ## From Manual Optimization to Automation - Deep performance tuning requires specialized knowledge of profiling, compiler behavior, assembly, and workload analysis. - Although the results can be valuable, the process is time-consuming and difficult to scale across a large organization. - Datadog wanted to move beyond isolated “heroic” optimizations toward a repeatable and automated process. - The manual techniques used by performance engineers became the foundation for heuristics in BitsEvolve, an internal agentic system intended to optimize code systematically. Datadog’s experience suggests that organizations should combine production observability with compiler-level analysis, prioritize high-impact hot paths, and automate proven optimization patterns so performance gains do not depend solely on a small group of experts.

Read original

Observability Database Design Go Ai Assisted Coding+4

datadogAug 12, 20251 min readCurated summary

Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, so there is insufficient technical material to summarize the article. ### Content Included - A link to Datadog’s Gartner announcement. - Navigation categories covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article may concern scaling process or pipeline efficiency, but its body is not present. Please provide the full blog post text for a substantive summary.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+4

datadogAug 4, 20251 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale | Datadog

The provided content does not include the tech blog post itself. It contains Datadog’s navigation menu and a link to an engineering article titled “Rust Timeseries Engine,” but no article text to summarize. Please provide the blog post content or its URL, and I can summarize it in the requested format.

Read original

Observability Rust Time Series Database Database Engine

datadogJul 17, 20251 min readCurated summary

How we tracked down a Go 1.24 memory regression across hundreds of pods | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link to an article titled `go-memory-regression`, but no article text to summarize. Please provide the post body or a complete page extract, and I can summarize it in the requested format.

Read original

Observability Go Garbage Collection Memory Management+1

datadogJun 24, 20251 min readCurated summary

How we built a real-time, client-side noise suppression library without server dependencies | Datadog

Datadog’s page announces that the company was named a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms. However, the provided content contains only the page header, navigation links, and product categories—not the blog post itself—so its technical argument and supporting details cannot be reliably summarized. ## Available Content - Announcement: - Datadog was named a Leader in the Gartner® Magic Quadrant™ for Observability Platforms. - The page links to a Gartner-related resource. - Product areas listed: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and cloud security - Digital experience monitoring - CI/CD and software delivery - Service management - AI and observability tools - The URL references a “noise suppression library,” but no corresponding article text was included. Please provide the full blog post content for a substantive summary.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+3

datadogJun 23, 20252 min readCurated summary

How we built reliable log delivery to thousands of unpredictable endpoints | Datadog

Datadog’s “Reliable Log Delivery” post explains how log-collection systems can avoid losing data when networks, destinations, or agents fail. Its central recommendation is to combine acknowledgments, buffering, retries, and controlled backpressure to provide at-least-once delivery without allowing outages to overwhelm the collector. ## Why Reliable Delivery Matters - Logs are often needed during incidents, precisely when infrastructure and networks may be unstable. - Temporary destination failures can cause data loss if collectors only keep logs in memory. - Retrying without limits can create duplicate logs, unbounded memory usage, or cascading failures. ## Buffering and Persistence - Collectors should buffer logs while downstream services are unavailable. - In-memory buffers provide speed but cannot survive process crashes or host restarts. - Disk-backed queues improve durability by preserving unsent logs across transient failures. - Storage limits are necessary so a prolonged outage does not fill the host’s disk. ## Acknowledgments and Retries - A log should be removed from the queue only after the destination confirms successful receipt. - Failed or unacknowledged deliveries are retried, allowing temporary network and service failures to recover automatically. - At-least-once delivery is the practical reliability target, meaning duplicates may occur and downstream systems should handle them safely. - Retry policies should use delays and backoff rather than continuously retrying at full speed. ## Backpressure and Operational Trade-offs - When downstream systems slow down, collectors must apply backpressure instead of accepting unlimited data. - Backpressure can limit memory consumption and protect the rest of the host. - Teams must define what happens when buffers reach capacity, such as dropping the oldest data, rejecting new logs, or prioritizing important streams. - Reliability also requires monitoring queue size, delivery latency, retry rates, and dropped records. A dependable logging pipeline is not built from retries alone. It requires durable buffering, explicit delivery acknowledgments, bounded resources, and clear failure behavior; organizations should choose retention and overflow policies according to the operational value of their logs.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+4

datadogJun 17, 20251 min readCurated summary

How we scaled fast, reliable configuration distribution to thousands of workload containers | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a link whose URL suggests an article about scaling configuration delivery to containers, but no article text or technical sections are available to summarize. Please provide the post’s body or a readable URL extract, and I can summarize it in the requested format.

Read original

Observability Kubernetes Datadog Application Performance Monitoring+3

datadogJun 17, 20251 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale | Datadog

The provided text does not contain the blog post itself. It mainly includes Datadog’s navigation menu and a promotional link announcing its Gartner Magic Quadrant recognition, so the article’s argument and technical conclusions cannot be reliably summarized. ## Content Present in the Extract - A promotional banner links to Datadog’s recognition as a **Leader in the Gartner Magic Quadrant for Observability Platforms**. - The page navigation lists Datadog offerings across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article is **“Unwinding a Shared Database”**, but its body text is missing. ## Practical Conclusion Please provide the article’s actual text or a complete page extract for a meaningful technical summary.

Read original

Observability Database Design Datadog Microservices

datadogJun 17, 20253 min readCurated summary

How we scaled fast, reliable configuration distribution to thousands of workload containers

Datadog’s seemingly simple tenant-configuration CRUD system must propagate updates rapidly and reliably to thousands of containers processing millions of logs per second. Loading configuration on every log is too expensive, while periodic caching introduces stale data and delayed updates. Datadog initially used database-backed caches invalidated through Kafka, but growing scale exposed reliability and resilience problems tied to repeated workload access to the central database. ## The Challenge of Propagating Context Data - Datadog calls tenant-specific settings—such as log parsing rules, Sensitive Data Scanner settings, and storage quotas—“context data.” - Configuration changes are expected to take effect almost immediately, including in Live Tail. - The same context data may be consumed by thousands of containers handling traffic for many tenants. - Because configuration directly affects customer-data processing, propagation must be both low-latency and highly reliable. - The system must assume that failures can occur anywhere in a large distributed environment. ## Why On-Demand Fetching and Simple Caching Fail - Fetching configuration from a database for every incoming log would create an impractical read load. - Large tenants can generate hundreds of thousands of logs per second. - Each processing instance could require thousands of database reads per second. - Multiplying this across many instances would require extensive, highly performant database replicas. - Caching configuration in each workload container reduces reads but does not eliminate the scaling problem. - Many workload instances still cache data for a high number of tenants. - Increasing the cache interval reduces database load but delays configuration updates. - With periodic invalidation, the average propagation delay is roughly half the cache interval. ## Context Loading v1: Database-Backed Caches and Kafka Datadog’s first successful architecture kept tenant configuration in a central durable database while allowing workload containers to cache entries indefinitely. - A user changes a log-processing configuration. - The central context database stores the update. - Kafka publishes an invalidation message after the database write. - Every workload container receives the notification. - Each container reloads the affected tenant’s configuration from the database. - This minimized routine database reads while preserving low-latency updates. ## Why the Initial Architecture Needed Reconsideration - The design required every workload instance to reach the central context database whenever a configuration changed. - As Datadog added more workloads and containers, update-related database traffic grew substantially. - Internal game days and production incidents showed that problems affecting the context database could spread to downstream processing workloads. - Database failures could prevent configuration updates from propagating and potentially make it impossible for new workload containers to initialize their context. - These reliability concerns demonstrated that Kafka-based invalidation alone did not sufficiently isolate workload processing from context-database failures. Datadog’s experience shows that configuration propagation at large scale requires more than a durable database and cache invalidation. The system must also reduce dependency on the central database during updates and startup, while continuing to provide near-immediate, reliable propagation.

Read original

Observability Database Design Datadog Log Management+4