Kubernetes | Techlist.io

naverNov 17, 2025Original article

Replacing a DB CDC Replication Tool Handling Tens (opens in new tab)

Naver Pay successfully transitioned its core database replication system from a legacy tool to "ergate," a high-performance CDC (Change Data Capture) solution built on Apache Flink and Spring. This strategic overhaul was designed to improve maintainability for backend developers while resolving rigid schema dependencies that previously caused operational bottlenecks. By leveraging a modern stream-processing architecture, the system now manages massive transaction volumes with sub-second latency and enhanced reliability. ### Limitations of the Legacy System * **Maintenance Barriers:** The previous tool, mig-data, was written in pure Java by database core specialists, making it difficult for standard backend developers to maintain or extend. * **Strict Schema Dependency:** Developers were forced to follow a rigid DDL execution order (Target DB before Source DB) to avoid replication halts, complicating database operations. * **Blocking Failures:** Because the legacy system prioritized bi-directional data integrity, a single failed record could stall the entire replication pipeline for a specific shard. * **Operational Risk:** Recovery procedures were manual and restricted to a small group of specialized personnel, increasing the time-to-recovery during outages. ### Technical Architecture and Stack * **Apache Flink (LTS 2.0.0):** Selected for its high-availability, low-latency, and native Kafka integration, allowing the team to focus on replication logic rather than infrastructure. * **Kubernetes Session Mode:** Used to manage 12 concurrent jobs (6 replication, 6 verification) through a single Job Manager endpoint for streamlined monitoring and deployment. * **Hybrid Framework Approach:** The team isolated high-speed replication logic within Flink while using Spring (Kotlin) for complex recovery modules to leverage developer familiarity. * **Data Pipeline:** The system captures MySQL binlogs via `nbase-cdc`, publishes them to Kafka, and uses Flink `jdbc-sink` jobs to apply changes to Target DBs (nBase-T and Oracle). ### Three-Tier Operational Model: Replication, Verification, and Recovery * **Real-time Replication:** Processes incoming Kafka records and appends custom metadata columns (`ergate_yn`, `rpc_time`) to track the replication source and original commit time. * **Delayed Verification:** A dedicated "verifier" Flink job consumes the same Kafka topic with a 2-minute delay to check Target DB consistency against the source record. * **Secondary Logic:** To prevent false positives from rapid updates, the verifier performs a live re-query of the Source DB if a mismatch is initially detected. * **Multi-Stage Recovery:** * **Automatic Short-term:** Retries transient failures after 5 minutes. * **Automatic Long-term:** Uses batch processes to resolve persistent discrepancies. * **Manual:** Provides an admin interface for developers to trigger targeted reconciliations via API. ### Improvements in Schema Management and Performance * **DDL Independence:** By implementing query and schema caching, ergate allows Source and Target tables to be updated in any order without halting the pipeline. * **Performance Scaling:** The new system is designed to handle 10x the current peak QPS, ensuring stability even during high-traffic events like major sales or promotions. * **Metadata Tracking:** The inclusion of specific replication identifiers allows for clear distinction between automated replication and manual force-sync actions during troubleshooting. The ergate project demonstrates that a hybrid architecture—combining the high-throughput processing of Apache Flink with the robust logic handling of Spring—is highly effective for mission-critical financial systems. Organizations managing large-scale data replication should consider decoupling complex recovery logic from the main processing stream to ensure both performance and developer productivity.

Kubernetes Apache Kafka Kotlin Mysql+4

datadogNov 4, 20251 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog

The supplied content does not include the blog post’s article text. It contains Datadog’s navigation menu and a promotional link to its Gartner recognition, while the URL suggests the post concerns CDC replication and search. ## Available Information - Datadog was named a **Leader in the 2026 Gartner Magic Quadrant for Observability Platforms**. - The page promotes Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and digital experience - Software delivery and service management - AI-powered observability - The referenced article URL is `engineering/cdc-replication-search`, indicating a likely focus on **change data capture (CDC), data replication, and search systems**. ## Missing Article Details - No sections, technical explanations, architecture diagrams, implementation details, or conclusions from the blog post are present in the supplied text. - A reliable summary of the CDC replication approach cannot be produced without the article body. Please provide the full post text or its main sections for a complete summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+2

slackOct 23, 20253 min readCurated summary

Advancing Our Chef Infrastructure: Safety Without Disruption

Slack chose to improve its existing Chef and EC2 infrastructure rather than migrate to Policyfiles, avoiding disruptive cookbook and role changes. The central strategy is to divide production into six Availability Zone–based Chef environments, limiting deployment blast radius while preserving existing workflows. A canary environment and staggered release train provide earlier detection of configuration problems and safer fleet-wide rollouts. ## Why Slack Avoided Policyfiles - Policyfiles could have improved long-term safety by replacing roles and environments. - Migrating dozens of teams and their cookbooks would have required substantial effort. - Slack concluded that the short-term disruption and migration risk outweighed the benefits. - Instead, the team extended its existing EC2 framework without requiring cookbook or role changes. ## Splitting Production Chef Environments - Previously, all production nodes used one shared Chef environment. - Cron-triggered Chef runs were staggered across Availability Zones to prevent simultaneous fleet-wide changes. - This reduced the impact of bad changes on existing nodes, but newly provisioned instances immediately consumed the latest version from the shared environment. - During large scale-out events, a broken configuration could therefore spread rapidly to many new nodes. - Slack split production into six environments: `prod-1` through `prod-6`. - Service teams still launch instances as `prod`; internally, nodes are assigned to a numbered environment based on their Availability Zone. - Updates to one environment now affect only the nodes mapped to that environment. ## Extending Poptart Bootstrap - Slack’s base AMIs include `Poptart Bootstrap`, which runs through `cloud-init` during instance startup. - It creates the node’s Chef object, configures DNS, and posts success or failure notifications to Slack. - Slack extended it to inspect the node’s AZ ID and select the appropriate numbered production environment. - This automatically distributes new nodes across isolated Chef environments without requiring service teams to change their provisioning process. ## Canary Deployments and the Release Train - Cookbook changes are promoted: - To sandbox at the top of the hour - To development environments through a Kubernetes cron job - To production beginning at 30 minutes past the hour - `prod-1` acts as the canary production environment. - It receives the latest changes hourly when new cookbook artifacts exist. - This tests changes in real production conditions soon after they are created. - `prod-2` through `prod-6` follow a release train. - A version advances gradually through the production environments. - The next rollout begins only after the previous version has reached `prod-6`. - This sequencing limits the number of affected nodes and makes regressions easier to identify. ## Why `prod-1` Updates Frequently - If the canary waited until a version had passed through every production environment, it would test artifacts containing larger batches of accumulated changes. - Updating `prod-1` frequently keeps the feedback loop close to the originating change. - The remaining production environments provide progressively broader validation after the canary stage. - For example, a new artifact can move from sandbox and dev to `prod-1`, then advance through `prod-2` to `prod-6` while newer artifacts continue entering the canary path. Slack’s approach preserves its existing Chef ecosystem while adding isolation, automated environment assignment, and staged promotion. The result is a safer deployment process that reduces blast radius and catches production issues earlier without forcing widespread application changes.

Read original

Kubernetes Aws Ec2 Chef Canary Deployment+3

datadogOct 21, 20252 min readCurated summary

Detecting malicious pull requests at scale with LLMs | Datadog

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

datadogOct 15, 2025Original article

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog | Datadog (opens in new tab)

Following a major 2023 incident that caused a near-total platform outage despite partial infrastructure availability, Datadog shifted its engineering philosophy from "never-fail" architectures to a model of graceful degradation. The company identified that prioritizing absolute data correctness during systemic stress created "square-wave" failures, where the entire platform appeared down if even a portion of data was missing. By moving toward a "fail better" mindset, Datadog now focuses on maintaining core functionality and data persistence even when underlying infrastructure is compromised. ## Limitations of the Never-Fail Approach * Classical root-cause analysis focused on a legacy, unsupervised global update mechanism that disconnected 50–60% of production Kubernetes nodes. * While the "precipitating event" was easily identified and disabled, the engineering team realized that fixing the trigger did not address the systemic fragility that caused a binary (up/down) failure pattern. * Prioritizing absolute accuracy meant that systems would wait for all data tags to process before displaying results; under stress, this caused the UI to show no data at all rather than "almost correct" data. * Sequential queuing, aggressive retry logic, and node-specific processing requirements exacerbated the bottleneck, preventing real-time recovery. ## Prioritizing Graceful Degradation * The incident prompted a shift away from relying solely on redundancy to prevent outages, acknowledging that some level of failure is eventually inevitable at scale. * Engineering priorities were redefined to ensure that data is never lost (even if delayed) and that real-time data is processed before stale backlogs. * The platform now aims to serve partial-but-accurate results to customers during an incident, providing visibility rather than a complete blackout. * Implementation is handled as a company-wide program where individual product teams adapt these principles to their specific architectural needs. ## Strengthening Data Persistence at Intake * Analysis revealed that data was lost during the outage because it was stored in memory or on local disks before being replicated to persistent stores. * The original design favored low-latency responses by acknowledging receipt of data before it was fully replicated, making that data unrecoverable if the node failed. * Downstream failures caused intake nodes to overflow their local buffers, leading to data loss even on nodes that remained online. * New architectural changes focus on implementing disk-based persistence at the very beginning of the processing pipeline to ensure data survives node restarts and downstream congestion. To build truly resilient systems, engineering teams must move beyond trying to prevent every possible failure trigger. Instead, focus on designing services that can survive partial infrastructure loss by prioritizing data persistence and allowing for degraded states that still provide value to the end user.

Kubernetes System Design Graceful Degradation Incident Analysis+4

datadogOct 15, 20253 min readCurated summary

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

Read original

Kubernetes Database Design Data Loss Prevention Graceful Degradation+4

airbnbOct 9, 20253 min readCurated summary

From Static Rate Limiting to Adaptive Traffic Management in Airbnb’s Key-Value Store

Airbnb evolved Mussel’s QoS system from static, per-client QPS limits into adaptive traffic management designed to maximize goodput. The newer approach accounts for the actual cost of requests, prioritizes critical workloads under stress, and detects hot keys or attack traffic before they overwhelm storage. Together, resource-aware quotas and real-time load shedding provide stronger protection against traffic spikes, uneven workloads, and DDoS-like bursts. ## Why Static QPS Limits Fell Short - Mussel is a multi-tenant key-value store serving millions of point and range reads across Airbnb. - Its original Redis-backed limiter assigned each client a fixed requests-per-second quota. - Requests exceeding the quota received HTTP 429 responses. - This model worked when backend effort roughly matched request count. - As usage grew, it could not account for: - The difference between a cheap one-row lookup and a 100,000-row scan. - Hot keys accessed by many clients simultaneously. - Localized storage-shard overload that affected unrelated traffic. - Sudden events such as bot floods, DDoS attacks, or large uploads. ## Resource-Aware Rate Control - Mussel replaced raw request counting with request units (RU), which represent estimated backend work. - RU calculations incorporate: - Fixed per-request overhead. - Rows and payload bytes processed. - Request latency, which distinguishes cached operations from disk-heavy ones. - The system uses calibrated linear formulas for reads and writes, with weights based on compute, network, and disk-I/O measurements. - Dispatchers debit a local token bucket according to each request’s RU cost rather than charging every request equally. - Periodic RU refills preserve simple, static quotas while making them more proportional to actual resource consumption. - Requests are rejected with HTTP 419 when the RU bucket is exhausted. - Load shedding remains separate, allowing latency-based protection to react dynamically without changing the underlying quota-refill mechanism. ## Load Shedding Under Sudden Stress - RU rate limiting smooths normal traffic but may react too slowly to rapidly changing workloads. - Mussel adds a load-shedding layer based on: - Traffic criticality. - A real-time latency ratio. - A CoDel-inspired queue-management policy. - Each dispatcher compares long-term p95 latency with short-term p95 latency. - A ratio near 1.0 indicates stable performance; a drop toward 0.3 signals rapidly increasing latency. - When stress crosses the threshold: - The system raises the effective RU cost for a designated lower-priority client class. - That class’s token bucket drains faster, causing its traffic to back off. - If conditions worsen, the penalty expands to additional classes. - Critical workloads, such as customer support and trust-and-safety traffic, can remain responsive while less important traffic is reduced. - The latency estimate uses the constant-memory P² algorithm, avoiding raw sample storage and cross-node coordination. ## Hot-Key Detection and DDoS Protection - Client-level quotas cannot prevent overload when many clients request the same popular key. - Mussel therefore detects skewed access patterns in real time. - When duplicate requests target a hot key, the system can protect storage by: - Serving responses from cache. - Coalescing identical requests before they reach the backend. - This approach protects the underlying shard whether the traffic comes from legitimate popularity, automation, or a DDoS burst. Mussel’s experience suggests that mature multi-tenant services should move beyond fixed QPS limits. Combining resource-based accounting, priority-aware load shedding, and hot-key mitigation provides a more effective way to preserve reliability while maximizing useful work during unpredictable traffic conditions.

Read original

Kubernetes Distributed Systems Redis Rate Limiting+4

discordOct 8, 20253 min readCurated summary

From Single-Node to Multi-GPU Clusters: How Discord Made Distributed Compute Easy for ML Engineers

Discord argues that distributed machine learning becomes practical when developer experience is treated as a first-class engineering problem. Ray provided the distributed-computing foundation, while Discord built a platform around it with a CLI, Dagster and KubeRay orchestration, and the X-Ray observability interface. This transformed GPU-intensive ML from manual experimentation into reproducible production pipelines, enabling Ads Ranking to move to multi-GPU neural networks and produce major business gains. ## Scaling Beyond Single-Node ML - Discord’s ML systems grew from simple classifiers to complex models serving hundreds of millions of users. - Teams needed: - Multiple GPUs for training - Datasets larger than a single machine - More compute than existing infrastructure could provide - Ray addressed the distributed-computing challenge, but Discord still needed a standardized internal platform to make it easy to use. ## Problems with Ad-Hoc Ray Clusters - Early ML engineers manually created Ray clusters using open-source documentation. - This led to: - Inconsistent cluster configurations - Uneven resource management - No centralized scheduling - Limited monitoring - Multiple teams independently rebuilding infrastructure solutions - Discord concluded that Ray needed an internal platform layer rather than direct, manual use. ## A Parameterized CLI for Cluster Creation - Discord replaced numerous GPU-specific YAML templates with one parameterized template. - Engineers specify requirements such as: - GPU type - Worker count - Memory - The CLI generates Kubernetes configuration, security settings, and hardware-specific resource requests. - It manages the full cluster lifecycle, including creation and deletion. - This made multi-GPU environments available through a single command and standardized deployments across teams. ## Automated Orchestration with Dagster, KubeRay, and Ray - Discord combined three systems: - **Dagster** defines workflows, dependencies, schedules, and validated configuration. - **KubeRay** dynamically provisions Ray clusters on Kubernetes with the appropriate namespace, service account, and GPU node pool. - **Ray** executes distributed training, evaluation, and batch inference. - The workflow is: 1. An engineer launches or schedules a Dagster pipeline. 2. Dagster submits the job specification. 3. KubeRay creates the required Ray cluster. 4. Ray distributes the workload across GPUs. 5. Logs and metrics flow back to Dagster and monitoring systems. - The approach provides predictable, reproducible jobs with centralized visibility. - Discord’s ad relevance model now trains daily without engineers manually editing cluster configurations. ## Centralized Observability with X-Ray - Discord built X-Ray as a web UI for monitoring Ray infrastructure. - It displays: - Active clusters - Cluster ownership - Machine types - Current status - Engineers can inspect dashboards and launch interactive notebooks for experimentation from one place. ## Ads Ranking as a Production Test - Ads Ranking determines which Quest advertisements are most relevant to individual users. - Before Ray, the system relied on XGBoost and lacked: - Model sharding - Multi-GPU support - Scalable, frequent retraining - Ray enabled sharded neural networks trained on multi-GPU clusters. - Reported results included: - Twice as many players joining Quests - Ad coverage increasing from roughly 40% to nearly 100% - A production pipeline that retrains daily and continuously delivers new model versions Discord’s experience suggests that distributed ML succeeds when powerful infrastructure is paired with simple interfaces, automated orchestration, and strong observability. Organizations adopting Ray should build comparable platform tooling around it rather than expecting ML engineers to manage clusters, scheduling, and monitoring themselves.

Read original

Kubernetes Xgboost Ray Gpu Computing+3

slackOct 7, 20254 min readCurated summary

Deploy Safety: Reducing customer impact from change

Slack’s Deploy Safety Program reduced customer-impact hours by 90% from its peak by focusing on safer change across all deployment systems, rather than optimizing individual services in isolation. The program combines measurable reliability goals, automated detection and rollback, blast-radius reduction, and cultural change. Its core lesson is to invest broadly, measure results, and expand approaches that demonstrably reduce customer impact without slowing development. ## Defining the Reliability Problem - Slack became increasingly mission-critical, raising customer expectations for reliability. - In analysis of customer-facing incidents, 73% were triggered by Slack-induced change, especially code deployments. - Incidents occurred across hundreds of services and multiple deployment systems, producing inconsistent levels of customer impact. - Customers reported that interruptions became significantly more disruptive after roughly 10 minutes. - Earlier reliability efforts often focused on individual deployment systems or services, leading to manual processes that slowed innovation and reduced engineering morale. ## North Star Goals and the Deploy Safety Manifesto The initial program goals applied to Slack’s highest-importance services: - Detect and automatically remediate deployment problems within 10 minutes. - Detect and manually remediate problems within 20 minutes. - Identify problematic deployments before they reach 10% of the fleet. - Preserve Slack’s engineering and development velocity. These goals later evolved into a Deploy Safety Manifesto covering all deployment systems and processes, including: - Automated safety improvements. - Deployment guardrails. - Changes to engineering practices and safety culture. ## Measuring Customer Impact Slack defined its primary program metric as: - **Hours of customer impact from high-severity and selected medium-severity change-triggered incidents.** The metric is an imperfect proxy for customer sentiment because: - Incident severity reflects current or anticipated impact, not always the final customer experience. - Medium-severity incidents require additional filtering to determine whether their actual impact is relevant. - It can be difficult to connect an individual engineering project directly to changes in customer sentiment. Slack evaluates the metric using four principles: - Measure outcomes rather than activity. - Distinguish real measurements from proxy metrics. - Apply subjective criteria consistently. - Regularly validate the metric against feedback from leaders who speak directly with customers. ## Choosing Where to Invest At the beginning of the program, Slack did not know which projects would produce the greatest benefit or when results would appear. Incident data is inherently delayed, while customers are experiencing reliability problems immediately. The investment strategy therefore emphasized: - Broad initial investment and a bias toward action. - Addressing known customer pain first. - Expanding successful projects and repeatable patterns. - Reducing investment in areas with limited impact. - Maintaining a flexible roadmap that could change as results emerged. Projects were prioritized according to whether they could: - Detect deployment problems earlier. - Improve automatic remediation time. - Improve manual rollback and remediation time. - Reduce severity by limiting deployment blast radius. ## Improving Webapp Backend Deployments Slack identified Webapp backend deployments as the largest source of change-triggered incidents and iteratively improved their safety: - Built automated metric monitoring. - Added automatic alerts and manual rollback procedures to validate alignment with customer impact. - Introduced automatic deployments and rollback. - Demonstrated that repeated automatic rollbacks could keep customer impact below 10 minutes. - Expanded monitoring to additional metrics. - Optimized manual rollback processes. - Added manual rollback capability for the frontend. - Began consolidating deployment practices through a centralized orchestration system inspired by ReleaseBot and AWS Pipelines. - Extended metrics-based deployment and automatic remediation beyond Bedrock and Kubernetes. These improvements made Webapp backend, frontend, and some infrastructure deployments significantly safer, with continued quarter-over-quarter improvement. ## Iterative Expansion Slack applied the same pattern across other areas: - Try an intervention. - Measure whether customer impact improves. - Invest further when the approach succeeds. - Reuse successful patterns in other systems. - Reduce or redirect investment when results are limited. The article notes that some efforts, such as faster mobile-app issue detection, were successful, while others produced less noticeable improvements. Slack’s experience suggests that deployment safety works best as an ongoing program: establish measurable customer-focused goals, automate detection and recovery, control blast radius, and continuously replicate proven practices without sacrificing delivery speed.

Read original

Kubernetes Incident Management Aws Pipelines Deployment Orchestration+2

datadogOct 1, 20251 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in Gartner’s Magic Quadrant for Observability Platforms, but no substantive discussion of the linked “Husky Query Architecture” article. ## Available Content ### Datadog’s Observability Platform - Datadog promotes products covering: - Infrastructure and container monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI and agent observability - The navigation emphasizes Datadog’s broad, integrated platform approach. ### Gartner Recognition - The page links to Datadog’s announcement that it was named a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. - The supplied text does not include the evaluation criteria, cited strengths, limitations, or Gartner’s comparative analysis. No reliable summary of the Husky query architecture can be produced without the article’s body text.

Read original

Kubernetes Observability Application Performance Monitoring Log Management+4

airbnbSep 24, 20253 min readCurated summary

Building a Next-Generation Key-Value Store at Airbnb

Airbnb rebuilt Mussel, its key-value store for derived data, from a complex EC2-based system into a cloud-native NewSQL platform. Mussel v2 combines bulk ingestion, streaming writes, low-latency reads, flexible consistency, and automated operations while supporting more than 100 existing use cases. A gradual, reversible blue/green migration moved production workloads without data loss or customer-visible downtime. ## Why Airbnb Rebuilt Mussel - New use cases—including real-time fraud detection, personalization, and dynamic pricing—required both streaming updates and large-scale bulk ingestion. - Mussel v1 had become difficult to operate and scale: - Node changes required multi-step Chef scripts on EC2. - Static hash partitioning created hotspots and latency spikes. - Consistency options were limited. - Resource consumption and costs were difficult to track. - Mussel v2 provides Kubernetes-based automation, dynamic range sharding, configurable consistency, namespace tenancy, quotas, and usage dashboards. ## Mussel v2 Architecture ### Stateless Dispatcher - A horizontally scalable Kubernetes service translates client requests into backend queries and mutations. - It supports: - Dual writes and shadow reads during migration - Retries, rate limiting, and dynamic throttling - Service-mesh security and discovery - Point lookups, range queries, prefix queries, and low-latency stale reads - Each dataname maps to a logical table, simplifying access patterns. ### Kafka-Based Write Pipeline - Writes are first persisted to Kafka for durability. - The Replayer and Write Dispatcher apply them to the backend in order. - Kafka absorbs traffic bursts and supports consistency, migrations, bootstrapping, and upgrades. - Airbnb plans to eventually rely more directly on the distributed database for ingestion and replication to reduce latency and operational complexity. ### Bulk Loading - Mussel retains support for both: - **Merge** jobs, which add data to existing tables - **Replace** jobs, which swap in a new dataset - Existing Airflow onboarding workflows transform warehouse data into a standard format and upload it to S3. - A stateless controller coordinates ingestion, while Kubernetes StatefulSet workers load data in parallel. - Deduplication, delta merges, and insert-on-duplicate-key-ignore improve throughput and reduce unnecessary writes. ## Scalable Data Expiration - Mussel v1 depended on storage-engine compaction for TTL expiration, which became inefficient at scale. - V2 uses a topology-aware expiration service: - Namespaces are divided into range-based subtasks. - Multiple workers scan and delete expired records concurrently. - Scheduling limits interference with live queries. - Max-version enforcement and targeted deletes help manage write-heavy tables. - The result is faster, more visible, and more scalable retention management. ## Blue/Green Migration - The migration had to handle massive datasets, thousands of tables, and mission-critical traffic with zero data loss and no availability impact. - Because v1 lacked table-level snapshots and CDC, Airbnb built a custom migration pipeline. - Tables were selected and migrated individually according to usage and risk. ### Migration Stages - **Blue:** All production traffic continued serving from v1. - **Shadowing:** Bootstrapped v2 tables processed parallel reads and writes, but v1 still served responses. - **Reverse:** V2 served live traffic while v1 remained available as a fallback. - **Cutover:** After validation, traffic was permanently moved to v2 one dataname at a time. - Automatic circuit breakers and fallback logic enabled rapid rollback if v2 showed errors or replication lag. - Kafka’s replication stream maintained eventual consistency between the two systems throughout the transition. ## Practical Takeaway Mussel v2 demonstrates that large datastore rearchitectures can be made safe through incremental migration, durable event logs, shadow traffic, and reversible per-table cutovers. The key recommendation is to combine a more scalable backend with strong operational automation and migration tooling, rather than attempting a single disruptive replacement.

Read original

Kubernetes Apache Kafka Database Migration Apache Airflow+4

datadogSep 18, 20251 min readCurated summary

From hand-tuned Go to self-optimizing code: Building BitsEvolve | Datadog

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional link announcing its 2026 Gartner Magic Quadrant recognition. As a result, there is not enough article content to produce a reliable technical summary. ### Available Information - Datadog is promoted as a “Leader” in the Gartner Magic Quadrant for Observability Platforms. - The page links to Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Incident and service management - AI and automation - The referenced blog URL appears to be titled **“Self-Optimizing System,”** but its article text is not included. Please provide the blog post’s main content or a complete page extract for an accurate summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogAug 12, 20251 min readCurated summary

Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, so there is insufficient technical material to summarize the article. ### Content Included - A link to Datadog’s Gartner announcement. - Navigation categories covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article may concern scaling process or pipeline efficiency, but its body is not present. Please provide the full blog post text for a substantive summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+4

lineAug 7, 2025Original article

Flexible Multi-site Architecture Designed with N (opens in new tab)

LINE NEXT optimized its web server infrastructure by transitioning from fragmented, manual Nginx setups to a centralized native Nginx multi-site architecture. By integrating global configurations and automating the deployment pipeline with Ansible, the team successfully reduced service launch lead times by over 80% while regaining the ability to use advanced features like GeoIP and real client IP tracking. This evolution ensures that the infrastructure can scale to support over 100 subdomains across diverse global services with high reliability and minimal manual overhead. ## Evolution of Nginx Infrastructure * **PMC-based Structure**: The initial phase relied on a Project Management Console using `rsync` via SSH; this created security risks and led to fragmented, siloed configurations that were difficult to maintain. * **Ingress Nginx Structure**: To improve speed, the team moved to Kubernetes-based Ingress using Helm charts, which automated domain and certificate settings but limited the use of native Nginx modules and complicated the retrieval of real client IP addresses. * **Native Nginx Multi-site Structure**: The current hybrid approach utilizes native Nginx managed by Ansible, combining the speed of configuration-driven setups with the flexibility to use advanced modules like GeoIP and Loki for log collection. ## Configuration Integration and Multi-site Management * **Master Configuration Extraction**: Common directives such as `timeouts`, `keep-alive` settings, and `log formats` were extracted into a master Nginx configuration file to eliminate redundancy across services. * **Hierarchical Directory Structure**: Inspired by Apache, the team adopted a `sites-available` structure where individual `server` blocks for different services (alpha, beta, production) are managed in separate files. * **Operational Efficiency**: This integrated structure allows a single Nginx instance to serve multiple sites simultaneously, significantly reducing the time required to add and deploy new service domains. ## Automated Deployment with Ansible * **Standardized Workflow**: The team replaced manual processes with Ansible playbooks that handle everything from cloning the latest configuration from Git to extracting environment-specific files. * **Safety and Validation**: The automated pipeline includes mandatory Nginx syntax verification (`nginx -t`) and process status checks to ensure stability before a deployment is finalized. * **Rolling Deployments**: To minimize service impact, updates are pushed sequentially across servers; the process automatically halts if an error is detected at any stage of the rollout. To effectively manage a rapidly expanding portfolio of global services, infrastructure teams should move toward a "configuration-as-code" model that separates common master settings from service-specific logic. Leveraging automation tools like Ansible alongside a native Nginx multi-site structure provides the necessary balance between rapid deployment and the granular control required for complex logging and security requirements.

Kubernetes Nginx Ansible Infrastructure Automation+4

datadogJul 17, 20253 min readCurated summary

How we tracked down a Go 1.24 memory regression across hundreds of pods

Go 1.24 initially caused an unexpected ~20% increase in memory usage across several services, despite its Swiss Tables implementation being expected to reduce memory consumption. The increase appeared in system-level RSS metrics but not in Go’s runtime metrics or heap profiles. Investigation showed that a runtime allocator refactor likely caused more of the Go heap’s virtual memory to be committed to physical RAM. ## The Unexpected Go 1.24 Memory Increase - The issue emerged during an internal rollout of Go 1.24. - Multiple environments showed approximately 20% higher memory usage. - A staging bisect directly linked the increase to the Go 1.24 upgrade. - The behavior was surprising because Go 1.24’s headline Swiss Tables feature promised lower CPU and memory overhead. ## Ruling Out Swiss Tables and Mutex Changes - Swiss Tables were disabled with: ```bash GOEXPERIMENT=noswissmap ``` - Memory usage did not improve, ruling out the new map implementation as the cause. - The new spin-bit mutex implementation was disabled with: ```bash GOEXPERIMENT=nospinbitmutex ``` - The memory increase remained, eliminating this runtime change as the likely culprit. ## System Metrics vs. Go Runtime Metrics - Go runtime metrics showed almost no change after the upgrade. - System metrics reported a significant increase in resident set size (RSS). - RSS measures physical memory currently used in RAM, while Go’s runtime accounting primarily reflects allocated virtual memory. - This discrepancy matters operationally because systems such as Kubernetes and the Linux OOM Killer rely on physical-memory metrics. ## Examining the Go Heap with `/proc/[pid]/smaps` - Linux’s `/proc/[pid]/smaps` exposed memory usage for individual mappings. - In Go 1.24, the main Go heap mapping had roughly: - 1.28 GiB of virtual memory allocated - 1.26 GiB resident in physical RAM - In Go 1.23, a similarly sized heap mapping had about 300 MiB less RSS than its virtual size. - Other memory regions were not significantly affected, indicating that the increased RSS was isolated to the Go heap. - Upstream changes to label Go-allocated memory regions should make future `maps` and `smaps` investigations easier. ## The Suspected Allocator Regression - The evidence suggested Go 1.24 was not requesting substantially more virtual memory. - Instead, previously uncommitted virtual memory was being committed to physical RAM, increasing RSS without changing Go’s internal memory totals. - A major refactoring of the runtime’s `mallocgc` function stood out in the Go 1.24 changelog. - The investigation therefore focused on this allocator change as the likely source of the regression. Go 1.24’s memory increase was caused not by Swiss Tables or mutex changes, but likely by altered heap allocation behavior in the runtime. Comparing RSS with Go’s runtime metrics—and inspecting `/proc/[pid]/smaps`—was essential for identifying the allocator-related discrepancy.

Read original

Kubernetes Database Design Go Linux+4