line·Nov 21, 2025Original article

Connecting thousands of LY Corporation services (opens in new tab)

LY Corporation developed a centralized control plane using Central Dogma to manage service-to-service communication across its vast, heterogeneous infrastructure of physical machines, virtual machines, and Kubernetes clusters. By adopting the industry-standard xDS protocol, the new system resolves the interoperability and scaling limitations of their legacy platform while providing a robust GitOps-based workflow. This architecture enables the company to connect thousands of services with high reliability and sophisticated traffic control capabilities. ## Limitations of the Legacy System The previous control plane environment faced several architectural bottlenecks that hindered developer productivity and system flexibility: * **Tight Coupling:** The system was heavily dependent on a specific internal project management tool (PMC), making it difficult to support modern containerized environments like Kubernetes. * **Proprietary Schemas:** Communication relied on custom message schemas, which created interoperability issues between different clients and versions. * **Lack of Dynamic Registration:** The legacy setup could not handle dynamic endpoint registration effectively, functioning more as a static registry than a functional service mesh control plane. * **Limited Traffic Control:** It lacked the ability to perform complex routing tasks, such as canary releases or advanced client-side load balancing, across diverse infrastructures. ## Central Dogma as a Control Plane To solve these issues, the team leveraged Central Dogma, a Git-based repository service for textual configuration, to act as the foundation for a new control plane: * **xDS Protocol Integration:** The new control plane implements the industry-standard xDS protocol, ensuring seamless compatibility with Envoy and other modern data plane proxies. * **GitOps Workflow:** By utilizing Central Dogma’s mirroring features, developers can manage service configurations and traffic policies safely through Pull Requests in external Git repositories. * **High Reliability:** The system inherits Central Dogma’s native strengths, including multi-datacenter replication, high availability, and a robust authorization system. * **Schema Evolution:** The control plane automatically transforms legacy metadata into standard xDS resources, allowing for a smooth transition from old infrastructure to the new service mesh. ## Dynamic Service Discovery and Registration The architecture provides automated ways to manage service endpoints across different environments: * **Kubernetes Endpoint Plugin:** A dedicated plugin watches for changes in Kubernetes services and automatically updates the xDS resource tree in Central Dogma. * **Automated API Registration:** The system provides gRPC and HTTP APIs (e.g., `RegisterLocalityLbEndpoint`) that allow services to register themselves dynamically during the startup process. * **Advanced Traffic Features:** The new control plane supports sophisticated features like zone-aware routing, circuit breakers, automatic retries, and "slow start" mechanisms for new endpoints. ## Evolution Toward Sidecar-less Service Mesh A major focus of the project is improving the developer experience by reducing the operational overhead of the data plane: * **Sidecar-less Options:** The team is working toward providing service mesh benefits without requiring a sidecar proxy for every pod, which reduces resource consumption and simplifies debugging. * **Unified Control:** Central Dogma acts as a single source of truth for both proxy-based and proxyless service mesh configurations, ensuring consistent policy enforcement across the entire organization. For organizations managing large-scale, heterogeneous infrastructure, transitioning to an xDS-compliant control plane backed by a reliable Git-based configuration store is highly recommended. This approach balances the need for high-speed dynamic updates with the safety and auditability of GitOps, ultimately allowing for a more scalable and developer-friendly service mesh.

EnvoyKubernetesGitopsService Mesh+4

figma·Aug 7, 2024·3 min readCurated summary

How We Migrated onto K8s in Less Than 12 months | Figma Blog

Figma migrated most of its core services from AWS ECS to Kubernetes in under 12 months because ECS was increasingly limiting its platform ambitions. Kubernetes offered better support for stateful workloads, Helm-based software, autoscaling, service networking, and the broader CNCF ecosystem. The migration was considered worthwhile because Figma had relatively few core services and had already containerized its workloads, making the transition more manageable. ## Figma’s Existing Compute Platform - By early 2023, Figma was already running all services in containers on Amazon ECS. - ECS had enabled rapid adoption of containerized workloads, but Figma’s growing infrastructure team began evaluating a more capable long-term platform. - Figma is not organized around thousands of microservices: - A small set of powerful core services provides modularization and traffic isolation. - New product capabilities are usually added to existing services rather than creating new ones. - This limited service count made a Kubernetes migration more practical. ## Limitations of ECS - ECS lacked Kubernetes primitives needed for complex workloads. - Running `etcd` on ECS required fragile custom startup code to manage cluster membership because ECS does not provide StatefulSets or persistent pod identity. - Kubernetes StatefulSets provide stable identities and stateful networking for systems such as `etcd`. - ECS did not natively support deploying groups of services packaged as Helm charts. - Open-source tools such as Temporal would require manual conversion into Terraform configurations. - This increased installation and maintenance effort. - ECS also made routine infrastructure operations more cumbersome. - For example, safely removing a malfunctioning EC2 instance was difficult. - EKS can cordon a node and move its pods elsewhere while respecting graceful shutdown behavior. ## Access to the CNCF Ecosystem - Kubernetes would give Figma access to a larger ecosystem of open-source cloud-native tools. - Autoscaling was a major motivation: - Figma was provisioning services for peak demand, wasting resources during lower-traffic periods. - Kubernetes tooling such as KEDA supports scaling based on CPU, SQS queue length, and custom Datadog metrics. - Figma expected to adopt a service mesh eventually. - Existing AWS load balancer routing created operational drawbacks: - Network Load Balancers could take several minutes to register or remove targets. - This slowed emergency deployments and increased incident remediation time. - Envoy offered more customization than AWS load balancers, including custom filters for shedding load during incidents. - Figma had already deployed standalone Envoy machines for a major service and saw Kubernetes ecosystems such as Istio as a path toward fleet-wide service-mesh adoption. Figma’s experience suggests that Kubernetes was justified not simply as a replacement for ECS, but as a foundation for more capable operations and broader platform tooling. Organizations considering a similar move should first assess their workload complexity, existing container maturity, and whether Kubernetes capabilities will materially reduce infrastructure work.

Read original(opens in new tab)

EnvoyKubernetesDocker & ContainersTerraform+4

datadog·May 26, 2023·3 min readCurated summary

Not just another network latency issue: How we unraveled a series of hidden bottlenecks

Repeated high-startup-latency pages in Datadog’s usage estimation service were caused by several independent bottlenecks rather than application changes. The investigation eventually identified four issues: CPU-throttled Envoy sidecars, a Linux kernel bug affecting ENA transmit queues, insufficient EC2 network bandwidth, and requests routed to terminating cache pods. Fixing each layer progressively reduced remote-cache p99 latency from roughly one second to its normal level of about 100 ms. ## Service Architecture and the Original Symptoms - The service consists of router, counter, and aggregator applications. - At startup, `counter` loads data from a remote cache into a local cache. - While the local cache is populating, request processing is slower and backlog grows. - Normal p99 remote-cache latency was approximately 100 ms, but it exceeded one second during deployments. - Scaling the remote cache did not help, indicating that the cache itself was not underprovisioned. ## CPU-Throttled Envoy Sidecars - Requests to the remote cache passed through an Envoy sidecar that batched queries into packets. - When `counter` restarted, Envoy reached its two-core CPU limit and was throttled. - Delayed request and response processing caused retries, TCP retransmits, and increased remote-cache latency. - Increasing Envoy’s CPU allocation eliminated the issue in staging and reduced production latency, but did not fully resolve rollout spikes. ## Linux Kernel and ENA Transmit-Queue Bug - Investigation of system and network metrics revealed a Linux kernel bug affecting AWS Elastic Network Adapter traffic. - The kernel mapped all outbound traffic to the first transmit queue instead of distributing it across eight queues. - This limited throughput and caused retransmits during high-traffic periods such as deployments. - A hotfix distributed traffic across all eight queues. - The change removed non-rollout latency spikes but rollout latency still fluctuated between 200 and 600 ms. ## EC2 Network Bandwidth Limits - ENA metrics showed that instances exceeded AWS inbound and outbound bandwidth allowances. - AWS dropped packets at the hypervisor when those limits were exceeded, causing retransmissions and slower cache requests. - Migrating to network-optimized EC2 instance types with higher bandwidth allowances largely restored p99 latency to around 100 ms. - Occasional one-second spikes continued despite the improvement. ## Requests Sent to Terminating Cache Pods - Remaining spikes correlated with remote-cache pods that were shutting down. - Clients continued sending requests to terminating pods, leading to one-second timeouts and retries. - The cache’s graceful-shutdown behavior did not adequately wait for Envoy clients’ in-flight requests. - The team added a `preStop` hook that sets an `XXX_MAINTENANCE_MODE` key to notify clients before termination and began coordinating shutdown with outstanding requests. The incident demonstrates the importance of tracing latency across the entire request path, from application startup through proxies, kernel networking, hardware interfaces, cloud bandwidth limits, and pod lifecycle behavior. Layered system metrics and component-level investigation were necessary to eliminate alert fatigue and restore reliable deployment behavior.

Read original(opens in new tab)

EnvoyDatabase DesignLinux KernelAws Ec2+4